#infoscribe

[infoscribe]

Querying the Datastore (Part II)

Part II was going to be about how to write Datastore queries and set up indexes, but that was becoming a rather massive and repetitive post no one really needs; the documentation may be murky on what queries are allowed but reasonably accessible on how. Instead this is going to be a post about the queries needed for each component of the application, which determine the properties of each Datastore kind.

These are the screens that make up the application (from our paper prototype):

Homepage *Retrieve the 10 highest count of documents transcribed by each Scriber [to display as the leader board].

Scriber's Dashboard Screen *For the current user (a Scriber), retrieve his/her score [to display as badges/warnings]. *For the current user (a Scriber), retrieve Projects that he/she worked on and are finished [to display on "trophies"]. *For the current user (a Scriber), retrieve Projects that he/she worked on and are ongoing. *For the current user (a Scriber), retrieve Projects that he/she would be interested in and are newly added. *For the current user (a Scriber), retrieve Projects that he/she would be interested in and are under-transcribed. *Retrieve all other Projects, under-transcribed to not.

Scriber's Transcribing Screen *For the selected Project, retrieve 20 documents that have fields that haven't been verified. *(If the Scriber opts for a random project, select the Project that is most under-transcribed.) *Create Transcription.

Scriber's Congrats Screen *Perform validation. Update Project's progress. Update Scriber's score.

Journalist's Dashboard Screen *For the current user (a Journalist), retrieve Project he/she set up, ongoing to finished, new to old [searchable].

Journalist's Project Monitor Screen *For the current user (a Journalist) and the selected Project, retrieve validated Transcriptions *If the Journalist decides to shut down the Project, set finished=True & delete all related Transcriptions.

Journalist's Upload Screen Journalist's Descriptions Screen Journalist's Field Selection Screen Journalist's Congrats Screen *Create Project.

[infoscribe]

Querying the Datastore (Part I)

If you're used to traditional relational databases, using Google App Engine's datastore is a bit like being Alice in Wonderland: The usual rules don't apply, which isn't to say it doesn't follow (somewhat counterintuitive) rules of its own. This is a summary post for later reference, enumerating all possible queries and, in Part II, how to write them and what to expect in return. Note that all possible queries can be enumerated; the datastore supports only a limited number of simple queries for the sake of web speed.

Note 1: Okay, so these aren't ALL POSSIBLE queries. I left out the ones I can't imagine why anyone would ever want to do, like queries asking for DESC key order. Note 2: As far as I can tell, all queries in google.appengine.ext.db work also for google.appengine.ext.ndb with a slightly different syntax.

I. kindless entity query

1. filter on entity key   a. 1 equality(=) filter* ---Possible, but use get() instead   b. 1-or-more inequality(>,>=,<,<=) filter(s) [in "key order," by default]*

II. entity query (on 1 kind)

0. no filter or sort order   a. all entities of a kind [in "key order," by default]*

1. filter on entity key   a. 1 equality(=) filter* ---Possible, but use get() instead   b. 1-or-more inequality(>,>=,<,<=) filter(s) [in "key order," by default]*

2. filter and/or sort on 1 single-valued property   a. all entities of a kind in 1 ASC/DESC property value order*   b. 1 equality(=) filter [in "key order," by default]*   c. 1-or-more inequality(>,>=,<,<=) filter(s) [in ASC property value order, by default]*   d. 1-or-more inequality(>,>=,<,<=) filter(s) in DESC property value order*

3. filter and/or sort on 1 multiple-valued property   a. all entities of a kind in 1 ASC/DESC property value order   b. 1-or-more equality(=) filter(s) [in "key order," by default]   c. 1-or-more inequality(>,>=,<,<=) filter(s) [in ASC property value order, by default]   d. 1-or-more inequality(>,>=,<,<=) filter(s) in DESC property value order

4. filter and/or sort on 2-or-more single-valued properties   a. all entities of a kind in ASC/DESC property value orders on 2-or-more properties**   b. equality(=) filter(s) on 2-or-more properties [in "key order," by default]*   c. equality(=) filter(s) on 1-or-more property(ies) & inequality(>,>=,<,<=) filter(s) on 1 property [in ASC property value order of the property with the inequality, by default]**   d. inequality(>,>=,<,<=) filter(s) on 1 property in ASC/DESC sort order(s) on 1-or-more property(ies)**

5. filter and/or sort on 2-or-more single-valued/multiple-valued properties properties   ---Need to check how these work

III. keys-only query (on 1 kind)   ---Similar to II, but specify you only want __key__

IV. projection query (on 1 kind)   ---Similar to II, but specify the names of the properties you want**

*You don't have to configure index.yaml for these queries; the datastore creates the index for you automatically. **You have to configure index.yaml for these.

(...I'm going to read this over to check for mistakes, but after a break.)

[infoscribe]

I feel like I'm always scrambling to learn something new for InfoScribe, so here's a short post to keep track.

First it was Ruby on Rails. Zooniverse's Scribe and ProPublica's Free the Files, notable transcription crowdsourcing applications that we're expanding on for general investigative journalism projects, are both written in Rails. It seemed only natural to learn it if only to understand these systems, if not to actually develop InfoScribe using the same framework. I was learning the language (Ruby) and the framework (Rails) at the same time, and it actually took me a while to get a hang of it. While a lot of people seem to cite scaffolding as a good way to break into Rails, for me it only started to make sense only after I learned to create the individual components separately. I practiced by rewriting some of my old projects in Rails.

Then came AngularJS. Not only had I used it for a final team project in my user interface design class and found it very reasonable, but our application for Google's Computational Journalism grant had also gone through. At a stage when I wasn't sure what changes that would bring, a JavaScript framework seemed like a safe choice that would be supported on any platform. I was adamant that I should have a thorough grasp of AngularJS before I started applying it. Back when my team was working on our final project, our philosophy, if we can even call it that, was to try something and if it works, move on. Very soon, we were going around in circles fixing one bug and watching everything else break. We managed to patch our way through the project, but the lesson was learned.

The latest is Google App Engine and its Datastore. It turned out Google Cloud Service credits are indeed part of our grant, and App Engine, although not exactly intuitive, comes with enough support and flexibility that makes it enjoyable to play around with. On the other hand, I've been somewhat reluctant to pick up Datastore, which I knew to be somewhat different from traditional databases. I was briefly excited about the graph database Neo4j, which boasts an intuitive node and edge structure and easy migration, but despite scattered success stories online, I found it more trouble than it's worth to make it work with App Engine. I resigned myself to learning Datastore, to find that its completely indexed queries might actually be very fast.

[infoscribe]

User Testing

If there’s anything we learned, it’s that it’s really hard to get people to user test our stuff.

Madeline and I met a little early to add finishing touches to our paper prototypes for InfoScribe. We also had our use scenarios ready. In a user testing session, each tester is assigned a specific use scenario describing their relationship to the application and the task to be performed. Obviously with InfoScribe, you could be one of three things:

You’re a journalist, with documents in need of transcribing.

You’re an InfoScriber, ready to transcribe documents for the betterment of society.

You’re a random web surfer, who’s stumbled upon the URL and wants to explore.

Your familiarity with the app would also vary:

You already have an account.

You know vaguely what it’s about, although you’ve never used it.

You have no idea what it is.

Excluding the one impossible scenario—someone who’s just stumbled onto InfoScribe would not already have an account—we had eight to choose from.

Everything was going well at first. We posted on the Journalism School’s Class of ’14 Facebook group. We successfully ordered pizza after a brief struggle with an online ordering service (and vowed InfoScribe’s sign up process would be less annoying). We put up a sign that read “User Testing: Get Pizza & Good Karma.” I might add this was also when we decided on the term “InfoScribers.” (It seems so obvious now, but for a long time, we were at a loss for a friendly term to call our transcribers. “Transcribers” sounds strangely functional and detached; “citizens” invokes “citizen journalism” but has odd legal connotations: “Are you a citizen?” “No, I’m a legal alien.”)

We quickly realized we should have recruited at least some people beforehand. Where we were, the Stabile Center on the Journalism School’s ground floor, was hardly empty, but everyone was there for something else. It didn’t help that we both took our reporting classes with the class of ’13 (we’re both dual degree students), so neither of us are very familiar with the current batch of students. The rare acquaintances we did try to grab were sadly rushing off to other engagements.

In the end, we managed to get hold of Jonathan Stray, Tow Fellow and computational journalism professor, as he was leaving. After a brief rundown, he went through two use scenarios for us that we decided were the most likely cases for our initial users: as a journalist and as an InfoScriber, who have each heard of InfoScribe but never used it before. We were lucky to get Jonathan, as he’s no stranger to journalism app UIs, and he provided ample feedback about where the interface was unclear or misleading or what he thought was good.

The two major points I got out of his testing session was that it should be made clearer that the comments provided by our InfoScribers will be read directly by journalists, and that we should, in the long run, expect to handle a far greater variety of PDF page formats than we had anticipated, sometimes within a single file. One of the monstrous PDF examples he showed us over pizza was the Department of Defense’s budget submission file, a single PDF of more than 600 pages with no consistent page format.

It turned out to be a fruitful session in the end. We look forward to trapping more unsuspecting souls into user testing for us.

[infoscribe]

Paper Prototyping

During the semester, Madeline and I got together to make paper prototypes for InfoScribe. 

Hard at work (Sort of. I’m taking the picture with my other hand.)

Paper prototyping is exactly what it says on the tin: It’s a user interface design technique of using paper mockups of the expected final product to test user behavior. Transitions in the interface are performed by hand, sliding bits of paper in and out (or, when that’s physically difficult to do, explained verbally to the tester). It's quick, low-cost and easy to modify, and prevents the awkward situation where hours of coding go to waste or become subject to patchy revisions because the designers failed to predict certain user behavior.

I was awakened to the wisdom of this process through Prof. Steve Feiner's User Interface Design class, although I must say it’s much more fun doing paper prototyping for a real project than for class. For class, your user base comprises of the professor and the TAs, as opposed to, you know, real people.

Our prototypes were based on the virtual sticky notes we had previously put up on RealTimeBoard.com, describing the functionality we need for each scenario. The overarching objective is, of course, to build an online community interested enough in investigative journalism to help transcribe data trapped in documents.

Prototyping the interface raised some interesting questions we hadn't yet worked out: For example, what’s the right way to display InfoScribers’ achievements? We want users to have a sense of pride at how much they contributed to investigative journalism, but not so much that they’re distracted from transcribing.

Also, how do we deal with “field sets,” related fields of data that may occur multiple, even unspecified times on a page? A specific case we considered was when the document provided to a journalist comprises of spreadsheets that were printed out and scanned in as an image-based PDF (yes, sometimes that’s what you get after a FOIA request). We don’t want to tire out our InfoScribers with a single page. Do we let the journalist cut up their pages?

Some of our questions were resolved while making the prototypes, while others were left for further consideration and, of course, the user testing itself.