human computer interaction — modeling a parasocial relationship to my palestinian family
listening machines final
my father is a palestinian refugee from gaza. growing up in the united states, my relationship to that history has always been mediated. mediated through stories, mediated through distance, mediated through documents, borders, and bureaucracies. in some ways, my relationship to palestine is parasocial. i know it through narrative, through inherited memory, through fragments.
for this final project, i wanted to model that mediated relationship using a listening machine.
instead of the computer understanding meaning, emotion, or context, it listens only through the narrow lens of acoustic features and transcription. speech becomes data. words become triggers. meaning collapses into pattern recognition.
the result is an interactive ambient composition titled “tough to talk.”
in the piece, i speak into a microphone. the system listens for specific phrases:
these phrases come from marina and the diamonds’ song “starring role.” in the project, fragments of the song appear as sonic responses when the machine detects those phrases.
the system does not understand the emotional weight of what is being said. it simply listens for recognizable patterns and triggers audio events.
this creates a strange interaction. the computer appears to be listening, but it is not listening in the human sense. it is listening computationally. this gap between human meaning and machine interpretation is the overarching theme.
→ node bridge converts udp to websocket
→ browser p5 receives osc
→ visuals + transcript + trigger + audio clips
i used OpenAI's Codex and my friend Olivia Brown's computational expertise in the development of this sound art system. when the system runs, the computer continuously listens to the microphone. if speech is detected, whisper transcribes it and sends the result as osc messages. the browser receives these messages and updates the display. if a phrase such as tough to talk is detected, the system triggers an audio fragment from marina and the diamonds’ recording.
the user experiences a conversation with a listening machine. however, the conversation is asymmetrical. the machine cannot understand emotional context or intention. it simply maps speech to predefined triggers.
this project is influenced by readings from the course, particularly jonathan sterne’s work on machine listening. sterne argues that machine listening is not equivalent to human listening. machines do not hear meaning. they hear signals, features, and statistical patterns. the system in this project demonstrates that distinction directly. the computer hears my voice, but it only understands it as a sequence of tokens.
phrases that carry emotional significance in human communication become simple control messages in the system.
my relationship to palestine has often felt mediated through fragments: stories told by my father, news coverage, bureaucratic categories like passport and citizenship. the listening machine reflects that fragmentation. it captures pieces of speech and turns them into triggers, but the meaning of those words remains outside the system.
