Microtasking for crowdsourcing and anonimization of data in Python : An idea to handle sensible data to external parties for pre-processing without revealing the true identity of the individual. Case of address and name in Colombian data-bases
I have often crossed with this situation. There are some raw data in really bad shape, in the power of an institution that has de duty to protect the identity of individuals in the data base. Just to know that someone is on the data might itself be problematic (think for instance if it’s a database about default on loans or on repiting an entrance test. People usually want this not to be publicly known). On the other hand, an external institution needs the data, and could do wonders with it, because it has technical staff and access to techniques that are out of the scope of the institution that owns the data. Or simply, the institution that owns it would be overflown by requests of data that would imply a lot of pre-processing. If there was no problem of disclosure, the institution would want to give free access to the data so that each user could transform to tailor its own needs. But that is just out of the question.
A possibility would be that the owning institution could do some anonimization of the data to give the non-identity related variables to the user. However, sometimes it is the sensible information the one that needs to be somehow processed. For example, think about when you want to merge several data bases from several sources thorugh fuzzy match by names of the people on them. This, as the document ID is missing or has problems. In Colombia, matching the ID of underage people with their ID’s when they are adults is quite imposible, and the matching has to be done by name, which has a lot of typos or appears writen in different ways on the different sources of information one wants to merge. Another case is when you have to convert addresses into longitude and latitude coordinates for georeferencing. In such cases, many times the address has a lot of typos, or the taxonomy is writen with abbreviations or in very heterogeneous ways. Again, to reveal the entire address is not an option, but you would like to give the data in some shape, so that the homogeneization of addresses can be done elsewhere.
Nowadays, the division of such tasks in microtasks given to a crowd of volunteers or people recieving micropayments (the practice of crowdsourcing of which I have talked about elsewhere in this blog) could handle such a task, if we find a way to facilitate the data to them. Here, I propose a way to do so. First, lets consider the next column of addresses, in the taxonomy used in Colombia.
As you see, sometimes the address is incomplete. Other times, the same kind of term is writen in different ways (like Calle and Cl.). Now imagine a column like this, but with thousands of entries. Using regular expressions, or simple search and replace protocols, would imply to correct this through many lines of code (the regex inferno as they call it) and would require a lot of work. Now, imagine we want others to do it for us, so we do this instead: First, we split the different parts of the address into several columns, splitting every time a white space appears. Like this:
The good thing about python is that it authomatically creates the right number of columns, which was unknown by us when we used the command (the larger address had 4 components, while the shortest had None, as it was a missing value). After assessing it was a limited number of columns, we can do something like this:
Now, lets create a new data set that only has the split addresses.
Now, imagine we could randomly sort each column, giving something like this:
Look that this data base has all the information of the original addresses, but sorted in a way that no single original address can be tracked. The identity and location of the person is protected, and yet, all the inputs needed for a code that will do the necessary pre-processing are there. Therefore, we could safely give this to an external party to do the rest of the job, so he can use as much regular expressions (or other methodology, like fuzzy matching through phonetic algorithms) as needed to give shape to the addresses as the GIS requires. Something like this:
Once finished, this very same code, if applied not to the randomized data base, but to the original, will correct the problem. Therefore, once the external party has finished the code, they can handle it to you, so you can run it in your server, and then do the georeferencing. To use crowdsourcing, the only additional step needed would be to give to each person only a sample of rows of the big data frame, so that each can work on a subset of the problems needed. You could even give several of them duplicated rows, to do cross validation among them.
titsay
janainaart
germansingergirl
ellievsbear
discoholicmusic
dailybridgerton
gummygunk
blakekathryn
2087