In 2001, The Washington Post broke a big story. Dozens of children in the District of Columbia’s foster care system had died in cases where government agencies and workers were at fault, either through failing to take preventative action or by placing the children in unsafe homes. The story, “The District’s Lost Children,” won a Pulizer Prize. More importantly, it drew necessary attention to a flaw in the way D.C. handled foster care cases.
Sarah Cohen was on the Washington Post team that spent a year investigating and sifting through the records of 180 children who died after somehow coming to the attention of the foster care system. Cohen recalls the massive amount of time spent deciphering documents. It would have been helpful to have a computer read the information, but that simply wasn’t possible. The documents were scanned PDFs of forms filled out by hand. The handwriting was at times hard to read. In other instances, the writing would extend sideways up the margin of the paper or the response wouldn’t logically make sense.
Now, thirteen years later, technology has advanced. Optical character recognition (OCR) enables computers to transcribe and record many documents, some of them even handwritten. However, OCR technologies still fail on the types of documents Cohen’s team used in the Lost Children investigation. With investigative bureaus shrinking across media organizations, reporters have less time to spend looking through documents.
We first learned about the Lost Children article while pursuing our master degrees at Columbia Graduate School of Journalism. A professor, Susan McGregor, recounted Cohen’s dilemma with the documents and posed to us a challenge: create a platform that would help investigative journalists unlock the data trapped in these difficult PDF documents. We saw crowdsourcing as a potential solution. By leveraging the time of others who were not journalists, but were invested in the stories waiting to be told, we could help investigative journalists decipher that data. At the same time, we could increase engagement between citizens and professional journalists.
Thanks to the Freedom of Information Act (FOIA), investigative journalists are able to access documents from a wide variety of government agencies and sources. However, these documents are often provided in inconvenient formats.There are stories waiting to be told across the globe that would benefit from easier, quicker access to the data in PDFs. In real life investigative stories, documents can include handwriting, poorly scanned areas and redactions. Any of these quirks can make it impossible for OCR technologies to extract textual data. What this means is that reporters have to painstakingly go through each document one by one. The overall format of the documents that make up a set can differ as well, or be unidentical but similar enough to confuse a computer. For example, one journalist shared the horror story of receiving documents in the form of spreadsheets that were printed out then scanned. This effectively transformed the easiest form a machine can read (csv, xls) into a mess.
* There are a number of past and ongoing projects that pull out text from image-based documents:
DocHive by Charles and Edward Duncan of Raleigh Public Record extracts structured textual information from documents with a consistent format across pages, specifically PDFs of forms that were digitally produced, printed and scanned. The application processes page images through ImageMagick, then uses OCR to automatically read in content from user-designated areas of each page. Currently it does not handle handwriting.
Zooniverse is an online citizen science project portal that invites the public to annotate, filter, rank, or transcribe scientific records. Among their transcription projects are Old Weather, which digitized weather observations from ship logs dating from the mid-19th century, and more recently Notes from Nature, which transcribed biodiversity data from natural history museum records. They have an active blog that explains, among others, the technological background of their projects, and they have also released their Scribe transcription framework on GitHub.
Similarly, The New York Public Library Labs carries out digital library projects that frequently enlist members of the public to transcribe or verify information in library collections. Notable examples include What's on the Menu and Ensemble, which transcribes historical restaurant menu collections and performing arts programs respectively. These projects are great examples of crowdsourcing transcriptions for documents with very loose formats that not necessarily tabular.
While projects by Zooniverse and NYPL Labs are not journalistic in the conventional sense, Free the Files is a prime example of how crowdsourcing document transcriptions can be used in investigative journalism. Records of spending in the 2012 presidential and congressional elections by outside groups like super PACs and nonprofits with secret donors were filed at TV stations across the country; ProPublica turned to their readers to extract and structure their content. Many features of Free the Files were specific to this particular project, including its attention to geolocation data. ProPublica later open-sourced the core functionality as Transcribable.
The Reporters' Lab also coaches journalists on how to leverage crowdsourcing services such as Amazon's Mechanical Turk and FromThePage to transcribe contents of documents and audio recordings.
Building on the wisdom of these projects, InfoScribe seeks to be a crowdsourcing transcription platform specifically for journalists but general enough to handle different projects. Even from a purely functional standpoint, it is necessary to have human eyeballs on pages to account for handwritten information and possible format variation. But, as was the case for Free the Files, perhaps the greater benefit of crowdsourcing is the community that forms around the content of the documents. We want a single web service where users can engage in a continuous dialog about investigative stories.
InfoScribe's journalistic bent also necessitates certain features: Many existing non-journalistic projects do not appear to perform automatic validation, which is crucial for time-constrained journalistic investigations. Also important is an interface to monitor the transcription's progress, as stories can arise even as the documents are being transcribed.
** We are a team of two graduate students with complementary skills. As we build InfoScribe, we have assigned our roles accordingly:
Madeline's role is to develop a crowdsourcing strategy that builds a community around InfoScribe, creating an experience that benefits both the media organizations and the transcribers. What makes a crowdsourcing platform “sticky”? How can we make the process seamless and fun for our transcribers? These are the questions Madeline seeks to answer, through case studies and interviews of other crowdsourcing platforms that have been successful. Additionally, Madeline is leading the user studies and collecting data on user experience that informs the design of the platform.
Aram's role is to build the basic uploading and transcribing interface as well as the back-end functionality of assigning documents and validating transcriptions. In particular, assigning a document to a user must take into account both user and document information, selecting only the documents the user hasn't seen yet and are under-transcribed. What decides which documents are under-transcribed? This is where automatic validation comes in, which not only reduces work for the uploader but also decides which documents have entries that need more transcribing.
***
As mentioned, InfoScribe is a project that stands on the shoulders of those who have come before us. Crowdsourcing has been harnessed effectively to help with everything from digitizing library collections to mapping disaster zones to funding new and innovative projects. While designing the platform from InfoScribe, we are learning from established crowdsourcing platforms, like the aforementioned NYPL projects, Kickstarter and others. In addition, we are conducting extensive user studies to figure out how to make InfoScribe a satisfying encounter for both journalists and the transcribers who are helping them. We are continually incorporating these reactions into our design. Through cheap paper prototyping and revision, we are able to save future time and energy by avoiding costly code revisions.
We made some unexpected discoveries through this process: We had originally thought crediting transcribers would serve as a great motivation for them to participate, but interviews uncovered that in similar past projects, some power users did not want their names published. Now we will be sure to communicate that users don't have to be credited if they don't want to.
Our next step is to finalize the structure of our user and document information within the confines of Google App Engine's Datastore. The main consideration is to find the best structure that allows non-costly document assignment and transcription validation. For the time being, the front-end interface will focus more on function than form, incorporating knowledge from user studies of the prototype. These user studies are ongoing and will continue to help inform our design.
We are also continuing to work on case studies of other crowdsourcing platforms, such as Zooniverse and NYPL. We seek to publish these case studies so that others can benefit from the best practices and information we have gathered. As we gear up for a pilot run at The New York World, we will begin to work with the document set that they need transcribed and gather an initial community of Columbia graduate students to test run the platform as transcribers.