NM3213 Final Project - Tasty Cartography (Post)
Download the KMZ files to simulate my prototype in Google Earth here: https://drive.google.com/open?id=0BypBpSlUszqAVVg0c2N6ek50OXM
--------------------------------------------------------------------------------------------------
Addressing Two Key Digital Humanity Terms
According to Wallack and Srinivasan, (2009) “Meta-ontology” and “Local knowledge” are always at odds with each other – they represent a struggle of the state versus the layman: both attempting to negotiate a middle-ground, but rarely succeeding to do so. To be clear, 'ontology' refers to the construction of reality, or how the world is perceived.
Meta-ontology refers to the reality of information constructed by power-holders – like governments – which is used to run nations or organizations. It essentially manifests as a state's data system: annual reports, financial earnings, crime rates, and more (Wallack & Srinivasan, 2009, p. 2). The concern is that such power-holders only prioritize specific "development" markers like health and education – which do not necessarily equate to a greater experience of well-being for citizens. (ibid., p. 3) As a result, the local knowledge – or ontology of ground-level citizens – is often ignored by authorities.
Local knowledge is important because it underscores the inner workings of smaller communities, and essentially defines their unique cultures. A positive example would be the rural communities in Vietnam, which store drinking water in tanks – when government officials planned to release microscopic copepods into such tanks to cull mosquito breeding, they consulted the villagers to understand who would be most capable of managing these tanks as project collaborators. (Vu, Nguyen, Kay, Marten, & Reid, 1998) Conversely, when meta-ontology and local knowledge clash, we get America's social climate: where less than half of eligible voters actually cast their electoral votes – showcasing a lack of confidence in a government which has ignored their needs. (Miller & Shanks, 1996; Wallace & Yoon, 2016)
Emotional analysis (a subset of sentiment analysis), however, presents a possible marriage of both meta-ontology and local knowledge. Emotional analysis classifies emotional responses into categories such as anger or happiness – and can be understood as a field interested in utilizing local knowledge to better interpret people's feelings.
Suggested Digital Humanities Term
How can analyzing emotional reactions bridge the gap between local knowledge and meta-ontology? According to Westbrook and Oliver (1991, p. 85), an individual's positive emotional experience with a product is somewhat dependent on their preconceived perceptions about it, and whether those expectations are affirmed or betrayed after they use that product. Simply put, emotional analysis can reveal an individual's orientation – and by extension, the state of their 'local knowledge'.
Additionally, local culture plays a part in shaping emotional responses – in Japan, positive emotions (e.g. being calm) are typically associated with experiences of interdependence with others. (Kitayama, Markus, & Kurokawa, 2000) As a result, Japanese service standards are associated with attending to a customers' needs first and foremost. (Winsted, 1997, p. 353) In short, emotional analysis also has to factor in the baseline of a culture – the shared orientation of a group.
To that end, I would suggest the term 'Crowd-sourced ontology' – emotional analysis represents a different breed of meta-ontology which has to account for local culture and personal orientations when interpreting facial expression. Positive and negative facial responses vary between races and genders – and because most facial analysis programs mathematically examine facial dimensions to interpret emotion, good facial analysis tools must have a large sample of facial references to pinpoint and seek out common ethnic traits (for example, Asians having comparably smaller eyes than Americans).
In this sense, the meta-ontological parameters of emotional analysis are only effective when its foundations are firmly modeled on data sourced from a local culture and population. Notably, a lack of ethnic diversity in facial references can restrict the scope of usefulness for such tools – which will be discussed further in the "Limitations" section.
Background/Context of Sentiment and Emotional Analysis
Textual analysis used to be at the forefront of sentiment analysis – the technology has been particularly useful for identifying themes in historical texts (Lincoln Logarithms, n.d.), but has additionally been co-opted to weigh the tone of online tweets (Barbosa & Feng, 2010) and reviews. (Pang & Lee, 2008) This is typically done by breaking textual data up into individual words and aggregating their positivity and negativity.
London Feels is a sentiment-mapping visualization that highlights hotspots of tweets from London in real-time (see http://london.feels.website/). Tweets are interpreted for their sentiment level before being placed on the map with a color representing positivity or negativity – ranging from blues and purples to represent "great" to "not so great" sentiment. Each hotspot can also be clicked on to reveal the full tweet, and the visualization aggregates an overall sentiment rating for these tweets originating from within London.
Textual analysis, however, has become less effective with the increasing prevalence of social media, where viral memes are birthing colloquialisms that machines cannot interpret. Textual analysis tools also face language barriers – most are in English, and cannot read foreign languages like German or Mandarin.
Enter facial recognition technology, which is mostly featured-based – scanning faces for geometric coordinates (of the eyes and mouth) and comparing them to an existing library of mathematically mapped emotional expressions. (Zhang, 1999) One may view this as digital mediation – "folding" real world images into computer language. (Berry, 2011, p. 1)
In general, facial analysis can be an accurate indicator of human sentiment – a study of more than 5000 recordings of facial reactions to funny commercials by McDuff, El Kaliouby, and Picard (2015) found that facial analysis results could predict participants' self-reported levels of enjoyment. Essentially, such tools are capable of capturing positive or negative emotional expressions – which are more uniform across different individuals, and do not suffer as heavily from the complications of context-sensitive textual analysis.
Face++ (see http://www.faceplusplus.com/demo-detect/) was chosen as a tool for my project because It outputs a full suite of facial measurements that factored into a 'smile' rating, denoting how likely someone is smiling. Skybiometry (see https://skybiometry.com/) was also utilized as it is capable of classifying facial expressions according to multiple emotional categories (Happy, Sad, Angry, Disgust, and Surprise).
The data produced by these two tools would be considered nominally classified, as each image is assigned a single emotional classification. (Sperberg-McQueen, 2004) However, it has the added benefit of quantifying these emotional characteristics with scalar ratings – instead of being lumped into broad emotional categories, each face retains some information about its degree of emotional intensity. (ibid.)
Intraface (see http://www.humansensing.cs.cmu.edu/intraface/) is an iPhone application which gives real-time emotional analysis of video feeds from an iPhone's front camera. This data is displayed as six bars denoting: Anger, Disgust, Fear, Happy, Neutral, Sadness, and Surprise. Unfortunately, the app does not have recording features, making its live-analysis interface difficult to obtain data from. However, using such mobile technology was considered, and will be discussed further in the "Possible Future Developments" section.
Building An Alternative Singapore Map
It is evident that sentiment analysis mapping is not new to the data visualization landscape. However, such visualizations are merely overlays that do not necessarily disrupt the geographical landscape that they observe. Their intent is merely to show where positive and negative sentiment originate from – and little else. Similarly limited is emotional analysis, which has mostly only been used to gauge consumers' feelings toward advertisements – its usefulness has not yet extended beyond the commercial realm.
In contrast, my visualization prototype aims to break away from such conventions and utilize facial analysis to represent local citizens' feelings about food – of which some dishes are "endowed with iconic status." (Henderson, 2014, p. 904) To achieve this, my prototype mapping visualization will take the form of a three-dimensional chloropeth (Cartogeek, 2016) – with regions of Singapore lifted up at varying altitudes to represent citizens' emotions about local food. A reference that inspired my project is the 2011 LIVE! Singapore exhibition, which visualized Singaporeans' intensity of mobile data usage on a daily basis (see http://senseable.mit.edu/livesingapore/visualizations.html) by using a three-dimensional overlay as well.
However, the map is not primarily meant to be an objective interpretation of reality, and can never be one – Drucker (2011, p. 12) convincingly argues that all 'data' is fundamentally biased because researchers have to interpret it. No 'accurate' sentiment map regarding food can exist, because it would inevitably leave out the sentiments of citizens outside my sample. As such, my disclaimer is that this project is less of a measurement of Singapore's population, and more of an experiment to structure emotional sentiment in visual forms.
Instead, my project's aim is to kick-start a platform for multiple reinterpretations of Singapore through its culinary scene, so as to counter the mainstream cartography of roadmaps. Those viewing the project may reflect on whether eateries can become new intersections in their daily commute, as opposed to urban infrastructures (like train stations and skyscrapers) that render the local citizen "insignificant." (Sim, 2011, p. 361)
Description of Project
My proposed project is a prototype digital map based on Google Earth, programmed with emotional ratings toward eateries in Singapore – each location appears as a vertical three-dimensional polygon (like a tower). These polygons will be uploaded and available as KMZ (Compressed Keyhole Markup Language) files with the submission of this report – and can be imported into any installation of Google Earth to simulate the prototype.
Two data types are present in the prototype: emotional data (a scalar rating associated with one of five categories: 'happy', 'sad', 'angry', 'disgust', and 'surprise'), as well as a 'smile' rating (also a scalar rating). Two of the tools discussed earlier were utilized – the emotional rating was extracted with a facial analysis tool demo from Skybiometry, while the 'smile' rating was obtained with a similar demo from Face++.
The polygons' heights are proportional to each other based on their emotional rating multiplied by a constant, which in this case is 25. Meanwhile, each polygon is color-coded according to the emotion detected: yellow for 'happy', blue for 'sad', red for 'angry', pink for 'surprise', and green for 'disgust'. For example, a detection of 'happy' emotion at 56 percent is represented by a yellow polygon that is 1400 meters (i.e. 56 times 25 meters) tall.
Accompanying each emotional rating is a 'smile' rating – this measures the percentage certainty that an analyzed face is depicting a smiling expression, and acts as a secondary indicator in identifying if a subject has a positive response. In general, having a common rating like this across all datasets would allow for different emotional responses (for example, 'angry' faces and 'happy' faces) to be compared on a similar scale.
To sum up: one unit of data in this visualization comprises of a photo of some facial response taken when someone was eating food – accompanied by their respective emotional rating and 'smile' rating. Due to the limited resources currently available for this prototype, the images were extracted from food video logs (or "vlogs") found on YouTube, which still feature some involuntary reactions on-screen. In a full-scale project, however, a dedicated sampling of such images from real-life would be preferable.
Proposed Interaction Experience of User/Viewer/Audience
The below flowchart is a rough outline of a viewer's interaction experience with a hypothetical, fully-fleshed out version of the map. Due to the limitations of Google Earth, the prototype does not have most of these processes (like the live generation of polygons).
When booted up, the interactive map zooms into Singapore immediately for convenience. Users should be able to freely zoom in or rotate the map to view it at different angles or scale, and they will be prompted to search for a local dish of their choice.
When users search for a local dish (e.g. Laksa) in the fully-featured map, they will see three-dimensional polygons rising up from the ground to mark out eateries that sell that local dish. As mentioned earlier, users can rotate the map to get a better perspective of how an eatery compares to others based on the height of their polygons. Alternatively, users can search 'all' to see every dataset currently available at the same time.
When users click on a polygon, a pop-up displays the emotional rating and 'smile' rating alongside the image analyzed to produce those ratings. Indicated as well is the dish being consumed, and the general location that sells that dish. Essentially, these pop-ups summarize subjects' emotional responses in regards to the food being eaten.
There should also be an option to toggle a visual style where each polygon is represented by stacks of the dish that they represent, with their emotion rating on top. This might make it easier to interpret from afar what the data is representing, instead of a static polygon that has to be clicked on for clarification. In the example below, a 'sad' emotional rating is associated with McDonalds' hamburgers.
In the current prototype, attempting to compare static polygons that are far apart from each other does not yield any meaningful analysis – however, the fully-featured map should also allow for users to select two or more polygons to compare side-to-side. By dragging two polygons to a side-menu, food enthusiasts and researchers alike can get a direct comparison between the various aspects of two datasets.
Limitations
Unfortunately, facial analysis is not flawless. As mentioned earlier, a caveat of emotional analysis tools is that they require a global sample of faces to account for different ethnicities' unique facial dimensions, as well as the differences in emotional intensity between cultures. As this task is costly and tedious, it is not uncommon for facial analysis tools to be restricted to a smaller sample of facial references.
This was evident in Skybiometry's facial demo – a practice run of the tool revealed that it sometimes categorized Asian faces erroneously as giving 'disgusted' expressions. It is likely that Skybiometry's facial recognition algorithm is mostly based on Caucasian facial data, rendering it less accurate for a project that aims to represent Singaporeans – whom are predominantly Chinese, Malay, and Indian in ethnicity.
Designating each photo to a discrete emotional 'category' is itself also problematic, as Drucker (2011, p. 11) argued – doing so might erroneously imply that only happiness, sadness, anger, disgust, and surprise exist. Reducing the complexity of such data makes it more palatable – however, we would risk ignoring other expressions like boredom or uncertainty. Drucker suggests a "sliding scale" to represent data as a continuity between binary labels such as male and female (ibid.) – but this might not be feasible for the multiple scalar categories used in this project.
Another issue lies in the photo data – in particular, the sources used were mostly video logs of foreign tourists eating, and not of Singaporean citizens. This was probably because the novelty of eating local Singapore dishes was an experience worth capturing – conversely, however, Singaporean citizens would likely not go to the trouble of recording themselves eating dishes that they are already familiar with.
By extension, this meant that the polygons mostly reflected locations that these tourists visited – a number of them being renowned eateries recommended by online reviews, instead of coffee shops in smaller neighborhoods. This is unfortunate, given that the food of Singapore's heartlands is precisely what constitutes 'local knowledge' and culture – which was the focus of this project. The use of mobile technology to circumvent this overarching issue will be discussed in the next section.
Possible Future Developments
Given the involuntary nature of emotions, it might be possible for emotional analysis tools to evaluate the faces of those who consume spicy food – which can cause cheeks and foreheads to flush red. (Ka, Kim, Kim, Kim, & Cho, 2014) Having a facial analysis tool detect the intensity of redness at those areas of the face (in addition to an emotional rating) might yield data that is useful for determining a meal's spiciness, and how enjoyable that spiciness is. Given the prevalence of spiciness in Singapore's dishes (Huat & Rajah, 2001), a visualization built on such data could yield other reorganizations of Singapore's geography.
As mentioned earlier, a particular limitation of my prototype was the difficulty of gathering large amounts of image samples from Singaporean citizens. To obtain a representative visualization of locals' feelings, the means to perform facial analysis should be widely accessible – the ideal would be a mobile phone application that boasts a facial analysis API. Users would be able to use the tool to upload their own emotional ratings and tag them to their geographical location, thereby crowd-sourcing the map visualization. As noted earlier, existing facial analysis apps like Intraface could be useful if they were further adapted for this purpose.
Concluding Remarks
With this project, my goal was to explore how to reconcile 'Local Knowledge' and 'Meta-ontology' through emotional analysis – with the express aim of populating Singapore's map with visual manifestations of Singaporean citizens' feelings about local dishes. The biggest limiting factors that prevented further development beyond the prototype stage were (1) a lack of photos from local citizens, and (2) the restricted scope of some facial recognition tools. Nevertheless, it is my hope that this project acts as a breeding ground for future visualizations which elevate the importance of local citizens' emotions.
In particular, food as a topic was chosen because of its cultural diversity in Singapore, which makes it a popular subject of small-talk amongst citizens. Other national issues such as education or housing can also elicit emotional responses which simply go unheard. However, with the ideas presented here, such local sentiments need not be lost in the rapid flow of life – instead, they might one day be manifested as the building blocks of a larger, more colorful world.
(Word count: 2989)
References
Barbosa, L., & Feng, J. (2010, August). Robust sentiment detection on twitter from biased and noisy data. In Proceedings of the 23rd International Conference on Computational Linguistics. Association for Computational Linguistics. 36-44.
Berry, D. M. (2011). The computational turn: Thinking about the digital humanities. Culture Machine, 12(0), 1-22.
Cartogeek. (2016). 2012 US Presidential Election Results by County: Democrat & Republican Share of Vote. Retrieved from https://www.arcgis.com
Drucker, J. (2011). Humanities approaches to graphical display. Digital Humanities Quarterly, 5(1), 1-21.
Henderson, C. J. (2014). Food and Culture: in Search of A Singapore Cuisine. British Food Journal, 116(6), 904-917.
Huat, C. B., & Rajah, A. (2001). Hybridity, Ethnicity and Food in Singapore. Changing Chinese Foodways in Asia, 161-98.
Ka, M. K., Kim, M. H., Kim, B. H., Kim, H. D., & Cho, D. U. (2014). Effect of Spicy Food on Face Image Color According to Sasang Constitution. Journal of the Korea Academia-Industrial Cooperation Society, 15(5), 2671-2677.
Kitayama, S., Markus, H. R., & Kurokawa, M. (2000). Culture, Emotion, and Well-being: Good Feelings in Japan and the United States. Cognition & Emotion, 14(1), 93-124.
Lincoln Logarithms: Finding Meaning in Sermons. (n.d.). Retrieved from http://disc.library.emory.edu/lincoln/
McDuff, D., El Kaliouby, R., & Picard, R. W. (2015, September). Crowdsourcing facial responses to online videos. In Affective Computing and Intelligent Interaction (ACII), 2015 International Conference. IEEE. 512-518.
Miller, W. E., & Shanks, J. M. (1996). The new American voter. Cambridge, MA: Harvard University Press.
Pang, B., & Lee, L. (2008). Opinion mining and sentiment analysis. Foundations and trends in information retrieval, 2(1-2), 1-135.
Sim, G. (2011). Historicizing Singapore cinema: questions of colonial influence and spatiality. Inter-Asia Cultural Studies, 12(3), 358-370.
Sperberg-McQueen, C. M. (2004). Classification and its Structures. In S. Schreibman, R. Siemens, & J. Unsworth. (eds.), A Companion to Digital Humanities (Chapter 14). Oxford: Blackwell.
Vu, S. N., Nguyen, T. Y., Kay, B. H., Marten, G. G., & Reid, J. W. (1998). Eradication of Aedes Aegypti from a Village in Vietnam, Using Copepods and Community Participation. The American journal of tropical medicine and hygiene, 59(4), 657-660.
Wallace, G., & Yoon, R. (2016, November 12). Voter turnout at 20-year low in 2016. Cable News Network. Retreived from http://edition.cnn.com/2016/11/11/politics/popular-vote-turnout-2016/
Wallack, J. S., & Srinivasan, R. (2009, January). Local-global: Reconciling mismatched ontologies in development information systems. In System Sciences, 2009. HICSS'09. 42nd Hawaii International Conference. IEEE. 1-10.
Westbrook, R. A., & Oliver, R. L. (1991). The dimensionality of consumption emotion patterns and consumer satisfaction. Journal of consumer research, 18(1), 84-91.
Winsted, K. F. (1997). The service experience in two cultures: A behavioral perspective. Journal of Retailing, 73(3), 337-360.
Zhang, Z. (1999). Feature-based facial expression recognition: Sensitivity analysis and experiments with a multilayer perceptron. International journal of pattern recognition and Artificial Intelligence, 13(06), 893-911.