When User-Generated Data Allows you to Overcome one Centuries-Old Problem only to Create Another
This is part two of a multi-part series on the design, development, implementation, and impact of DaZiBao, a web application supporting Chinese literacy learning (Part 1, Part 2, ...).
As I describe in part one, learning to read and write in Chinese is a labor-intensive, time-consuming task that, for many individuals, is one of the less enjoyable aspects of learning Chinese. We created a working alpha version of DaZiBao to help learners take advantage of their listening and speaking vocabularies as an gateway to written Chinese production. In short, users spell out the sounds of the Chinese words they want to write and then use oral, visual, and translation clues to help select the corresponding character.
Images are a key component of DaZiBao as they uniquely and non-linguistically support learners in identifying which Chinese character is the one that corresponds to the meaning-pronunciation pair they already know. Learners who spelled out 'ma' in image 2 below as the pronunciation for the word they wanted to write (e.g. 'horse' or 'mother' or 'code' or 'hemp') would be able to use the images to discern the corresponding character. But in order to do this in an open-ended way we need access to multiple relevant images for any given combination of Chinese characters, with less than 10 second's notice, for free.
Image 2: screenshot of DaZiBao showing spelling-character-audio-image connections for 'ma'.
Before 2005 this would have been either impossible or costly in the extreme. Since 2005 however the amount of accessible user-created data has skyrocketed. Hundreds of millions of people tweet and their billions of tweets become part of a massive publicly-accessible repository, or corpus, of tweets. People add photos to photo-sharing sites and these too, unless otherwise designated, become publicly-accessible. Accessible both through the search bars on Twitter.com or Flickr.com as well as through a backdoor called an Application Programming Interface or API. APIs allow designers and developers to request, analyze, arrange, and display all the tweets with the #MeToo or #MarchForOurLives or #SecondLanguageAcquisition hashtag or do the same for the top ten images that correspond with 'horse' or 'mother' or 'code'. These requests can be triggered by user input or interaction and can be combined with other API requests to offer unique, sometimes educational, ways of interacting with repurposed data.
In the case of DaZiBao, the Flickr API makes it trivial for us to request images for any and all Chinese characters that are pronounced similarly ('ma': 马, 妈, 码, 麻, 吗, 嘛, 骂). Users type in the spelling of a Chinese word, we use one API to return all the possible matching characters, the device's operating system to create audio pronunciations of each returned character, and then send each character to Flickr so we can display a column of ten or so relevant images below each one (see the project code repository for details). In many cases conducting a search for the same spelling five minutes apart will yield several different images due to the dynamic, constantly updating nature of a repository maintained by 51 million users. Before we dig into the age-old problem of inappropriate content, a paragraph about the legality of using public images for our purposes.
We use public images under the the US Fair Use Guidelines, specifically section 107 of the US Copyright Act. DaZiBao fits under each of the four factors of fair use: 1) a: the software is free for all to use and was designed at the University of Nebraska Lincoln--a US non-profit educational institution, b: we are using the images in ways that are unique from their original purpose; 2) we seek only images with relevant image-word matches not entire photographic series by a single artist; 3) we use relevant images based on image-word matches across Flickr's 51 million user base--making it unlikely that more than one image would be used from the same user's collection; 4) each image we use, in and of itself, is non-focal, in other words images are always showing up as a dynamically-created collage of images--each at a smaller size than the original work. For recent supporting case law see (Philpot v. Media Research Center Inc. 2018).
So there we were, sitting around the conference room table, admiring our working prototype. Thanks to Shawn Hellwege it had a responsive, clean design. Thanks to Xianquan Liu it was aligned with teacher and student needs. And thanks to the Flickr API we had all the images we needed, the thing was, despite setting the safe_search value to 'safe' in the API call, we were also getting some images we really didn't need. In going through some use cases, the process of using the system to write 'I want to eat hamburgers' in Chinese characters resulted in plenty of helpful images but also included four low-cut bikinis, two breasts, one erect penis, two partial penises, and two otherwise sexually explicit images.
Image 3: screenshot of DaZiBao showing results of 'xia' with too-racy-for-US-K12-schools images blocked out in red.
In an adult language class in the US, or perhaps in any middle school in Sweden, this would be a non-issue but as we aimed to use DaZiBao to support K-12 public school children in Nebraska as they learned to write Chinese, the bikinis, breasts, penises, and other explicit images would have to go. Our options seemed to be 1) move to Sweden; 2) create a database of around 6.5 million images and find a way to check each one for appropriateness; or 3) dynamically filter each of the 70 images displayed per query for appropriateness within the 8 seconds between when images for a query were returned from the Flickr API and when they were displayed (continued in part three).
Xianquan Liu is a PhD candidate in the department of Teaching, Learning and Teacher Education in the College of Human Sciences at the University of Nebraska Lincoln.
Shawn Hellwege, earned his Master's of Education degree in the department of Teaching, Learning and Teacher Education in the College of Human Sciences at the University of Nebraska Lincoln.
Katie Bieber is a Master's student in the department of Teaching, Learning and Teacher Education in the College of Human Sciences at the University of Nebraska Lincoln.
Dr. Christopher Heselton, is Associate Director of the Confucius Institute at the University of Nebraska Lincoln.
Asha Srivastava, is a visiting scholar in the department of Teaching, Learning and Teacher Education in the College of Human Sciences at the University of Nebraska Lincoln.
Juana Paramo Reyes, is a participant in the Undergraduate Creative Activities and Research Experience (UCARE) program at the University of Nebraska Lincoln.
Dr. Justin Olmanson is an Assistant Professor of Instructional Technology specializing in the use of multimodality, artificial intelligence, and data in the design of learning experiences. His current and former education technology design projects include: DaZiBao, MICa, FunWritr, InfoWriter, and Distributed Biography. He teaches graduate courses about the Design of Learning Experiences [TEAC 859] and the Use of Data and Artificial Intelligence in the Design of Learning [TEAC 882D] and in the undergraduate teacher education program he teaches Technology Integration in Classroom and Speech Language Pathology settings [TEAC 259]. Justin is in the department of Teaching, Learning and Teacher Education in the College of Human Sciences at the University of Nebraska Lincoln and leads the work on DaZiBao/Chinese Character Helper.
Originally published on Tumblr. Cross-posted on Medium