#makebigtwitterlist

[rhesearch]

Using Mechanical Turk (Eventually) to Follow Researchers Part 2

In a previous post, I compiled a list of authors from CHI'13 to ask mechanical turk workers to find the researcher's twitter usernames.  I tested with a batch of 20 researchers and decided that it would be better to start with an algorithmic list of potential candidates (I will refer to the list as Researchers Twitter Results) that turkers vet rather than find by themselves.

In this post, I share the code I used to compile a list of search results for "<authors name> twitter" (RTR) and discuss the problems with detecting whether the twitter name belongs to a CHI author.

I built two scripts for compiling RTR.  (1) A PhantomJS script that runs a server and returns google search result urls.  (2) A python script that asks the server for search results and puts them in a big python dict.

I've been using PhantomJS for side projects.  PhantomJS is a headless scriptable Webkit that lets one use Javascript.  The script runs a server that accepts a google search url (e.g. http://www.google.com/search?q=banana) in a get param and returns a json list of resulting urls.  This is pretty useful as Google's custom search API's free is pretty stingy at 100 queries per day.

The python script queries the PhantomJS server for all researcher names and compiles them into a big json dict.  I didn't get banned from google, I wait an average of 3 seconds between queries.  You can see full results here in this json file.  Here are a couple examples:

The first result is the right twitter page:

 "Alan Gruner": [

    "https://twitter.com/AlGruner",

    "https://www.facebook.com/public/Peter-Alan-Gruner-Jr",

    "http://www.discogs.com/artist/Alan%2BGruner",

    "http://www.imdb.com/name/nm0344412/",

    "http://www.intelius.com/people/Peter-Gruner/08pxz2t9kvv",

    "http://en.wikipedia.org/wiki/Billy_Kidman",

    "http://www.manta.com/c/mm3tplc/allen-k-gruner-attorney",

    "http://www.yasni.fr/alan%2Bgruner/recherche%2Bpersonne",

    "https://myspace.com/billykidman12345",

    "http://www.lawyers.com/louisville/kentucky/Allen-K-Gruner-4567896-f/",

    "https://twitter.com/AlGruner"

  ],

If all results were like this, it would be easy to find all of the usernames I'm looking for.

This example has 10 twitter usernames...

  "Alex Jansen": [

    "https://twitter.com/TallChineesGuy",

    "https://twitter.com/AlexLovesChachi",

    "https://twitter.com/Alexjensen90",

    "https://twitter.com/AlexGaareJansen",

    "https://twitter.com/awpjansen",

    "https://twitter.com/AlexJansenBirch",

    "https://twitter.com/phaoust",

    "https://twitter.com/TheeAlexJansen",

    "https://twitter.com/LanginStrife",

    "https://twitter.com/alexculinair",

    "https://twitter.com/TallChineesGuy"

  ],

This example has none.  It's not likely they have a twitter account.

"Vicki L. Hanson": [

    "http://researcher.ibm.com/view.php%3Fperson%3Dus-vlh",

    "http://hanson.massachusettsbox.com/c-82090.htm",

    "http://womeninplanetaryscience.wordpress.com/2010/12/30/vicki-hansen-celebrating-research-and-inquiry-and-respect-for-different-ideas/",

    "http://www.intelius.com/people/Vicki-Redoutey/061zkb02n8y",

    "http://jslhr.highwire.org/cgi/content/abstract/32/1/2",

    "http://mail.free-knowledge.org/references/authors/vicki_l__hanson.html",

    "http://www.slideshare.net/mikecrabb/aiding-data-gathering-in-web-usability-studies",

    "http://www.youinweb.com/profiles/02341/vicki-l-lyall_344727063.htm",

    "http://www.peekyou.com/vicki_hanson",

    "http://www.directority.com/MA/Hanson/Mental-Health-Services/Mental-Health/384379-Lyall-Vicki-L.htm",

    "http://researcher.ibm.com/view.php%3Fperson%3Dus-vlh"

  ],

The distribution of the number of results is as follows:

0:218 1:94 2:312 3:153 4:105 5:116 6:40 7:40 8:48 9:41 10:48 11:62 12:1

So, there may be more to vet than I originally anticipated.  I also thought about problems in vetting as I tried to do it myself.

Researchers may not talk about research on twitter.

If tweets are private so you can't read them to help understand if it's research.

[rhesearch]

Using Mechanical Turk to Follow Researchers

Mechanical Turk is a platform from Amazon that makes it easy to make crowdsource processes.  The ACM portal contains a list of people I want to keep an eye on so that I can stay up to date with research.  If I'm reading a paper and I like it, I search for the authors name to see which university they are from.  If they say they use twitter, I follow them.  I want to follow whoever has a twitter account who published in CHI'13.

I want:

A list of authors from CHI'13.

The twitter accounts associated with them.

Getting the author list

Getting (1) is easy enough.  I go to the acm page that list papers published at CHI'13, open up the Chrome console to get a list of authors.  I will use this list in a CSV for a Mechanical Turk HIT.  I could use xpaths from the Chrome console on this page, but I'm going to use jQuery. After injecting jQuery into the console, I type into the console.

var authors = $('a[href*="author"]'); for(var i=0; i<authors.length; i+=1) console.log(authors[i].innerHTML);

This prints out each author for CHI'13 (a list of 1534).  I suppose I should remove duplicates.

var authors_d = {}; for(var i=0; i<authors.length; i+=1) authors_d[authors[i].innerHTML]=1; for(var i in authors_d) console.log(i);

Now, I have a list of 1278.

The Mechanical Turk HIT

I'm going to sign in as a request at reqester.mturk.com, I already have an account set up.  One of the things that makes HITs more effective is to make them easy for a worker to use.  I'm going to go with a bare and simple design that also provides me with some verification.  I want workers to give me three or pieces of data:

Twitter username of the author

Description from the twitter username

Certainty that the user is indeed a technology related researcher

I start with a template from amazon:

And I modify it to look like:

One thing you may notice is that I have changed the worker restrictions to be less stringent than the default.  Using 95% approval and from the USA has worked well for me in the past.  I then uploaded a csv file with one column "author".  This is all I needed to test a batch of 19 researchers.  Here's the code from the modified template.  I'll see what I have in the morning and update this post.

Update:

This morning, only one of the 19 hits are turned in.  This may be due in part the low reward price or the time of day (12:30 am) it was launched. At least the single submitted HIT looks correct:

@xuan_zhao does seem to be a CHI'13 author.

After launching this, I decided my approach is suboptimal.  It would be better to do google searches for every researcher automatically then extract the twitter descriptions for each, using mechanical turk workers to vote for the correct twitter username and estimate certainty.  I might try this soon.