a place for some notes

[selfie-in-pixelated-mirror]

Recap on 2019, onto 2020

I did create more. I took film photos. 

I did read more, too. Read Permutation City. And Greg Egan’s Diaspora, too. 

(And Lords of Finance, On Earth We Are Briefly Gorgeous, Trick Mirror...)

Did not write more, but maybe that can be this year’s goals. To write more in response to readings, but also to return to this practice of writing down my thoughts and feelings.

[selfie-in-pixelated-mirror]

Have started CNF49 at Stanford. Creative Non-Fiction. And I’ve realized I have forgotten how to write. I feel somewhat numb, indifferent to many details I would have savored before and hurried to write down. Is this a function of the past year? Or just general rustiness? Must practice more expression through writing. Will share some of my writing here--

[selfie-in-pixelated-mirror]

Notes on Videos as Space-Time Region Graphs

A summary of this paper: https://arxiv.org/abs/1806.01810

Attempts to recognize human actions by using “space-time region graphs,” with the nodes’ edges defined by 1) “similarity relations” (for corresponding objects over time) and 2) spatio-temporal relations amongst one another.

This was tested on Charades and Something-Something. Charades is a dataset that involves actions performed indoors with a lot in the background. Something-Something consists of objects and hands interacting with it against a clean background.

They consider the action of opening a book. What happens when a book is opened? Well, first, we have the object, the book, and it’s visual appearance. The book’s appearance changes over time (as the state of the book goes from open to closed). But the book does not do this on it’s own, it’s state change is due to the human-object interaction of a person opening it. They argue that object’s states change through object-object or human-object interactions.

Interpreting the scene as this means that they formulate the problem to examine and somehow evaluate the book’s features over time. They also consider the region around the book. Rather than evaluating each scene and the visual features in that scene as a whole, they create some structure to their approach.

Their aim to “perform long-range temporal modeling of human-object and object-object relationships via a graph-based reasoning framework.”

Related Works-

a. Using spatio-temporal relations is popular, but many features are often hand-designed. Alternative is deep representations from data

This paper focuses on objects, rather than examine features for the entire frames

b. Using pairwise relationships (human-object or object-object) has been found helpful although the approach has been used for static images rather than video

c. Graph Convolutional Networks used to update node representation, which can then be used for classification

Take data, which is video. Represent it as a graph of objects and recognize actions from the graph.

Input - video clips of length > 5 seconds

Network - 3D ConvNet

Output - TxHxWxd, T temporal, HxW spatial, d channel number

RPN for N proposals, ROI Align to get the corresponding features for each bounding box. The features have d dimensions, and the vector is now of size Nxd.

The graph constructed has N nodes with the N objects tracked over T frames. (Nxd*T)

They break the graph into two sub-graphs for the different type of relations. Apply the Graph Convolution Networks to output vectors of size Nxd. Apply average pooling over the nodes for one vector of d dimensions.

Average pooling over whole video (TxHxW) for another vector of length d. Concatenate and use for video-level classification.

Similarity edges are described as the dot product of the two nodes transformed by separate weights:

The edges for each node will be normalized with softmax so that the sum of edge values will e 1.

The spatio-temporal edges are defined as the intersection of unions between object I and j. The edges for a node are also normalized so that the sum is 1. We create two adjacency matrices, effectively creating two sets of edges, one for pointing forward and the other back. “richer structure and enlarge the number of propagation neighborhoods during graph operations”

Conducted experiments on Charades and Something-Something.

32 frames, 6fps

Generate 50 and 10 RPN boxes for Charades and SS respectively. Variation in number of proposals does not change the mAP to go below the baseline (31.8%). 36.2% with 50, 35.9% with 25, 36.1% with 100 . They show that their model performs better than the baselines used (including non-local networks) in situations where actions are classified as a sequence of events rather than a continuous event (shooting a goal v. running), when the pose changes greatly over the frames, and when the actions involve interactions with other objects.

[selfie-in-pixelated-mirror]

beach by Pescadero, Dec. 2018

[selfie-in-pixelated-mirror]

checking in ...

2019, and the goal of the year is to produce more content, whatever that may be. Will start with film photos and writing in response to art and technical concepts. A sure way to test my understanding of a concept is to be able to summarize it or draw some conclusion from it.

Last year’s goal was to read more, and slowly but surely that was achieved. Will continue reading (am currently on Americanah and Permutation City).

[selfie-in-pixelated-mirror]

Went to the SF MOMA for the second time (last visit was in November). There was a special exhibition on Magritte, which I thought would be overrated but was pleasantly surprised by. 

Magritte was a Belgian surrealist artist, most famous for his painting of the man in the chapeau melon with the apple obscuring his face, also famous for the image of the pipe with the sentence, "Ceci n'est pas une pipe." 

His work makes the most sense when viewed as sets. It was interesting to see the elements reappear through out the exhibit, and having one idea played out in many different paintings really gave it more heft. (There’s something to be said to consuming art/content as a body of work rather than singular pieces.)

I never thought I would like surrealism because of the way it looked (strange surrealism, everything is too perfect, almost flat colors at times) but reading the ideas driving him to create these images made me realize how engaging the pieces are. He played with contrasting ideas such as lightness and heaviness with rocks and birds. And he was always pointing out the subjective nature of the things we see. In one room, he obscured or filled in the shapes of people and things, to emphasize the way we perceive things as one way because of what is seen rather than what is hidden and truly there. 

My favorite room was where he played with illusions of having images within images, either framed or through a window. By altering the reflections and transparency of the surfaces, he forced the viewer to question what they were seeing. He played with the idea that we view things through a window or frame, so the things we see may either be reflections or through some other medium rather than what is truly there.

Also really enjoyed The Dominion of Light, a series of paintings with bright skies and darker streets softly illuminated by streetlamps. It gave the appearance of existing both at night and during the day. But it also gave that feeling one gets towards the end of the day, when you can feel it slipping from your grasp and the sky is alight and you are so happy to be alive. Before this, I usually thought pictures did an adept job at what the original works resembled but now I see... these will do no justice.

Last, but most beautiful was a blurry painting of the sea at sunset in motion. Seestück by Gerhard Richter. Gave that same emotion as the previous set by Magritte. This too, needs to be seen in person, so the magnitude of the painting and the full richness of the colors used can be seen. It was like a photograph. 

[selfie-in-pixelated-mirror]

one year later

An update on life -- I moved to California. Living so far from home (and working full time) made me grow more than I expected to after college. I got caught up in enjoying my free time with new activities, but I miss reading and writing, so this will be a good way to go back to that practice. And it will force me to progress through my reading list (keeping a reading list, a habit I realized quite a few of my friends have, too...)

[selfie-in-pixelated-mirror]

response to HRI readings

This week’s readings for Humanoid Robotics involved the following four papers on Human Robot Interaction (HRI):

A Survey of Autonomous Human Affect Detection Methods for Social Robots Engaged in Natural HRI

ExpressionBot: An Emotive Lifelike Robotic Face for Face-to-Face Communication

Robot gains Social Intelligence through Multimodal Deep Reinforcement Learning

Recognition and Expression of Emotions by a Symbiotic Android Head

Mostly dealt with robots attempting to identify human beings, and how to empathetically respond to them. Oddly enough, the question of whether anyone else found this strangely disturbing (to have machines look and behave like us...is it just me?) came up only once and was brushed aside rather quickly.

Below, I’ll identify the main points of each paper and/or what stood out to me most and questions that arose after reading.

1. A rather long and comprehensive paper (as survey papers tend to be) that covers different types of robots in the field by breaking them down into three types of HRI: 1) collaborative HRI (working together), 2) assistive HRI (teaching or helping), 3) mimicry HRI (imitating). Went over the various models for Affect Categorization, which is just another way of reading the way someone is feeling (described as “a complex combination of emotions, moods, interpersonal stances, attitudes, and personality traits that influence his/her own behavior”). This is primarily done through extracting features from images of the human being’s face, although they also go into other traits that can factor in to a robot’s ability to perceive affect (such as body language- jerkiness or smoothness of motions). They mention several robots for each type of HRI, I liked iCat best (a cat designed to teach children how to play chess!)

https://www.researchgate.net/figure/220888708_fig1_Fig-1-Two-users-playing-chess-with-the-iCat-observing-the-game

2. This paper was concerned with making the robotic head more realistic, the two main parts that went into creating the head was the hardware and the facial animation projected onto the head. Their goal was to cover three aspects 1) facial expressions, 2) eye movement/gaze, and 3) accurate visual speech.

Interesting ideas they introduced was the “Uncanny Valley,” in which a robot’s appearance affects the way we interact with it and the Mona Lisa effect, which describes the phenomena that when image is on a 2D surface, the eyes seem to follow the viewer regardless of positioning (whereas projecting the face onto a 3D head would give a more realistic effect).

For visual speech, they grouped visually similar phonemes (termed visemes) with the corresponding face that would have to be made. However they had to come up with a formula to combine the face that had to be made to pronounce the phoneme with the face that would be made to express a certain emotion (weighing expression model and combining with current expression model). Side Note- there was an interesting phrasing they used “to blend the expressions with the lip movement”. Personally, I think this is the most interesting part of this paper, the eye movement conclusion does not seem as significant, they concluded that the animated face projected onto the physical form was better received than a 2D animation. I would say that if done well, I would prefer a physical 3D form rather than a flat screen, too. Although this does seem cool, because the other paper we read did not project a face onto a form, but rather had the face built in.

3. Using Multimodal Deep Q-Network (MDQN), the robot learned how to interact with human beings over the course of 14 days. They broke the process into data generation and training parts. This paper was the most technically challenging to read, and I have to go back and do a  more careful, in-depth analysis of it and also learn what a deep Q-network is.The gist of it was that they were trying to identify when to shake hands with a user by taking in a grayscale image stream and also depth data and feeding 8 channels into a sort of network. Conclusion— revisit this and do a  better reading of it!!!

4. The most eerily realistic robotic head was presented in this paper. Her name is the FACE android, and her skin is made out of Frubber (!!) It’s built by David Hanson, but that shouldn’t be a surprise since all the super-realistic robots seem to come from there. The description of how they put servos and wires behind the skin to act as tendons was very cool too! The goal of this project was to have an attention-based gaze that followed the person who seemed to demand the most attention (approaching human being in empty room, or new person joining an on-going conversation) and then interact with the human being appropriately. It is interesting to note that they modeled the Cognitive Architecture after Damasio’s idea that inputs turn into knowledge structure for reasoning, taken from his book “Descartes’ Error” and formalized by Bosse (something to look up, since I am curious what this is about!) Conclusion best summarized by this sentence, “…results demonstrated the promising social capabilities of the robot to perceive and convey emotions to humans which is essential starting point for the development of a real social agents.” Good starting point, more nuanced and sophisticated techniques still needed both for perceiving using more senses and reacting to different scenarios and with additional responses.

[selfie-in-pixelated-mirror]

ADI General Body Meeting- Civic Technology

Will attempt to quickly summarize the conversation we had on how we could use technology to help with the way people interact with our government. We started the discussion off by introducing the two questions motivating this conversation.* 

We then jumped in with how do we interact with our government today? Taxes, voting, food stamps, education, Amber alerts, filing bankruptcy...

One good example was a Twilio app that had been created, where users could text a phone number and would be guided through a series of questions to figure out if they were eligible for food stamps. If they were, they would then be walked through the process, effectively speeding up a three week process to a matter of a few days. 

Someone asked whether the problem may be the systematic nature of government, that it is slow and bureaucratic whereas the tech industry attempts to innovate and move and break things as fast as possible. If this is the case, perhaps civic tech projects is not the right approach, as people are not the ones meant to be responsible for solving these issues. While a few of these problems are due to outdated processes and some slow paperwork/forms to be filled out, are the bigger problems ones that can only be addressed systematically?

A. brought up a good counter-argument that this is exactly why we need these movements, government takes a while to pass legislation, but there are other ways that tech enables us (the question remains, who us should be - government organizations or the public at large) to help others, with a fast turn-around. In the case of the food stamps, a simple app that sent out text messages sped up a process that takes way too long, and even longer to change the program itself.  He conceded that the issue was that we often end up relying on larger companies (Google and Facebook explaining to their users how to vote during election season) or individuals (a friend from Harvard taking a semester off to work on a civic tech project) to volunteer to work on these problems, and they often do this as a one-off project. With whom does the responsibility lay? Does it fall to the government or to people and companies to step up and fix issues? 

What we need is for someone to be responsible for these initiatives, to see that they are continuously being supported and to follow-up with them. We can’t rely on arbitrary initiatives to solve all of our problems. Then the problem is that not many software engineers will or are motivated to work for the government, given the opportunity to work at other companies w/ larger salaries and environments in which they can grow and receive more mentorship. 

Side Note: I think, that means that we may need to change our attitude towards helping the general good. Maybe our generation has to have the mentality of helping the public, and working on problems that will improve the collective communities experiences. But I acknowledge that to do this would be harder than it sounds. I’m also hopeful, given the times we’re in, that my peers and I will be more aware and willing to step up to help fix problems in our society.

This was followed up with the question of are there enough opportunities for us to be involved with civic tech initiatives around the city. It was pointed out that NY high schools will be required to have their students learn computer science at some point in the future (p. vague about the details on this) but the bottom line was that we should begin with helping out the students in local high schools with programming. J. interjected and said that he didn’t think it was a matter of whether there are opportunities to get involved (because there are many, and he had brought up NY Open Data  earlier in the night), but rather, should we, at this stage of our lives, go into civic tech. He had apparently asked someone in the field this question, of whether college students should look for internships in this area and had been surprised when the person had advised against it, on the grounds that it was not an area where one would receive a lot of mentorship. Y. jumped in and said she had heard similar things, and her own plan was to learn as much as possible before going in and working on tech + gov. issues. 

[selfie-in-pixelated-mirror]

from 02/25

Had dinner with a friend. She talked about her knowledge of theater, and introduced me to a few things that I’ve been thinking of since

- Sunspring, a movie written using an AI. Watch here. (Will read this in the future: https://arstechnica.com/the-multiverse/2016/06/an-ai-wrote-this-movie-and-its-strangely-moving/)

- The idea of removing the work from the context of when it was created and the intent the author had. Roland Barthes’ introduced this in his essay The Death of an Author. 

- Note to Self: read Sarah Kane’s 4.48 Psychosis. N. noted how it was difficult in this instance to remove the author from her work, since Kane committed suicide shortly after writing this piece.

[selfie-in-pixelated-mirror]

Selfie in a Pixelated Mirror

1. I’m back. What happened? 

I went to an engineering dinner, where we sat at a table with an alumni whose advice to us was to take in as much information as possible and create a blog/newsletter where we attempt to explain the world as we see it to our peers but also the particular problems we work on day-to-day to a more general audience. Okay, I know every other millennial is doing this (who doesn't have a Medium?) Nonetheless, I think this is a good exercise, and so I’ve decided to try my hand at it, once more.

2. I decided to rename this blog. Tumblr blogs do sometimes have longer names, and I chose this based on the closest (physically atm) reading material to me. It’s the reading packet that was last week’s homework for the poetry workshop I’m taking. John Beer’s Selfie in a Pixelated Mirror. Read it here. We’ll take a closer look at it in the future.