Kinect Video to Audio (Call and Response)
For my project, I initially drafted a proposal for a reconfigurable lazer maze with servos. I decided to forsake that idea for a task which I was unsure whether or not I could accomplish, and so I decided to create a video to audio interface between the Microsoft Kinect and using the Arduino for support processing through the serial. I wasn't sure how much of this was possible but I was adamant to see how far I could get until I could achieve no further.
Bugs All The Way Down...
I had an initial plan. I would use a program to read the Kinect visual stream and monitor how many people had entered the space. I would capture this data stream and process it with a program I would write myself. I would then read values from the Arduino through the potentiometers to create a reconfigurable audio sequence which would be depicted by the values of three different pots. From here, depending on how many people were inside the space a server which I would also write would catch the response from the arduino and the Kinect to write to another program which would ultimately execute the sequence and then write the note sequence to the LEDS as they played each note. My idea was clear to me when I went in and then fell apart almost immediately after I started interfacing the Arduino and the Server.
I had selected a few tools for the job:
KinectA - KinectA is a fantastic application used for motion tracking through the Kinect. There were a few other alternatives for motion tracking, but they did not work nearly as well as the KinectA. The advantages of the KinectA over the others, was the GUI interface which allows me to reconfigure the parameters of the monitoring space without getting inside the code the rewrite it for every place I set up the KInect at. It was a no brainer. KinectA looks for a certain pattern from the camera and transmits the data across the network to "localhost:3333", which is port 3333 on the machine that runs it. It does this through a protocol called OSC (Open Sound Control). OSC as a protocol, required certain libraries to operate... Either that or I would of had to manually create each OSC packet and send it, which is far too tedius.
Python - Python proved to the be the language I would use to create the server that catches the OSC packets from KinectA. In this sense the Server and the KinectA were entangled in a constant call and response. I used an open source Python library called 'PythonOSC' to manipulate the thread used to receive the server and it worked very well. As part of its example code, it provided a segment of code that would handle the receipt of OSC packets and after modifying this code it would then run a function all the while it receives data through localhost:3333. This lead to some problems with the inner structure of the program... That is, the way I had used the library left me no choice but to put the entirety of my code inside the OSC packet handler. The disadvantage of this, was that when no OSC packets (no objects by the Kinect) were being sent across the network, there were no lines of code I could execute outside of that instance as the program was caught in a function that would act as the server forever. This meant that I could only run code when the Kinect was seeing objects in the field. It sounds like a powersaving decision, but it wasn't... But for the sake of extra marks, lets say that it is.
Supercollider - Supercollider is a very popular audio programming language used by legendary trance artist BT. It offers some very powerful benefits such as live coding, and the structure of the language is easyish to understand and well documented. It has two parts, one called Scsynth, which acts as the server which receives an interpretted language called Sclang. Scsynth doesn't know a lick of what Sclang is written in, but that's because it contains an interpretter that takes the language and converts them into OSC packets from which Scsynth receives. I began to learn supercollider quite intimately and it was great and good... until...
I tried to interface python with supercollider. This is where everything started to go horribly wrong. Interfacing one language with another is no easy feat and many have tried to hack together solutions for how to solve this problem. The Supercollider wiki had linked three methods of doing so: The first of which was written in 2011, and was outdated to the point the site didn't exist anymore. The second option, sc_0.31, seemed promising... But as I went to even import the library: Everything in the library had syntax errors. Something had gone out of date, and it totally rendered this interfacing library pointless. The third....
Supriya - Supriya seemed like the most modern library being written to interface python with Supercollider. The documentation was well written and the installation seemed easy enough. Then came the problem of installing the library. There was an issue installing the library as the author had left a lot of very important information about the installation out. One such example: The author required that ./scsynth as a command be executed from within any directory on the computer. It did not provide an explanation for how to do this. Hours of scavenging and trying to debug that, we broke through to the next problem: Each dependency had to be individually downloaded and installed. At this point, entered a lot of frustration due to me installing into the wrong version of python. Finally! The damn library was installed. Now I tried to import the library in the Python server and.... It couldn't find its own module... Queue 2 hours of trying to debug why it couldn't locate its own functions to discover that the author had not written the install correctly... and the only way to execute any program using the library was to execute the program from within the library directory.... So I moved the server into the directory and lo and behold! The import was successful. I wrote two lines of code... Suddenly all hell breaks loose as all the assertions within the library begin failing as I boot the server up.
Excusing my language and speeding up the story: There were bugs all the way down into the library and the only person working on it was this one guy... And so I reluctantly scrapped the idea of using Supercollider to play finely synthesised audio. Instead, I used...
Pygame - Pygame is a set of modules used for writing games in python! It's very well written and many projects have been built upon it. It _is_ a bit excessive for the purpose of playing audio, in the sense that it also supports making video viewports and general game processing... But I only needed it for its audio capabilities. After the disaster of trying to interface Supercollider with Python... This solved all my problems. It allowed me to read files from the current directory and play them. It allowed me to manipulate their volume on the fly... But most importantly it allowed me to loop the sound files. I obtained the soundfiles off a site called http://www.looperman.com which was a fantastic site that after making an account for, gave me access to entire libraries of royalty free loops. I would highly recommend it for anyone looking for free audio loops. Do NOT go to http://www.free-loops.com/ They false advertise and will force you to give a donation to download the entire library, despite listing all the files under creative commons.
Further Disasters and Breakthroughs
Throughout the project, I had assumed that the Arduino's role to the server would be the one component that couldn't fail me. And to a degree it wouldn't of.... However the physical limitations of the serial connector was something I didn't anticipate. Taking data from the Arduino was one thing, but sending data back... Especially using Python3 as the server was problematic. As part of the conditions of Python3, data sent through the serial connector must be sent in bytes. Because of how quickly the laptop is in comparison to the Arduino, this resulted in an overflow of the serial buffer and strange values being expressed within the Arduino serial. This inconsistency in speed made it almost impossible to read and write off the Arduino without overloading the microcontroller.
Improvements
Could use processing.












