This is sort of the "power user" section of the site. You can use Synapse to do plenty of great stuff in Ableton and Quartz Composer without ever having to touch even a bit Max programming. But, if you're still interested, read on...
Max/MSP is a wonderful visual programming language developed by the fine folks over at Cycling '74. You can get a 30-day trial here. There is also a free equivalent of Max called Pure Data, which I have not used, but I think you can do everything here in that as well.
I only started using Max a few weeks ago, and I am incredibly impressed by how effective it is. From this point on, I'm going to assume you have an okay working knowledge of Max. If you don't, it is pretty easy to learn, and there are a wealth of incredibly helpful Youtube tutorials out there.
Synapse sends OSC messages out on ports 12345 and 12347, and receives OSC messages on port 12346. The messages sent via 12345 and 12347 are identical; I do this so that multiple applications can receive data from Synapse simultaneously. So, to get data from Synapse, you want a "udpreceive 12345" object, and to send data to Synapse, you want a "udpsend localhost 12346" object.
Here are the messages that Synapse sends:
/tracking_skeleton <bool> - Sent with 1 when we start tracking a skeleton, and 0 when we lose it.
/<joint>_pos_world <float> <float> <float> - The x,y,z position of the joint in world space, in millimeters.
/<joint>_pos_body <float> <float> <float> - The x,y,z position of the joint relative to the torso, in millimeters.
/<joint>_pos_screen <float> <float> <float> - The x,y,z position of the joint as projected onto the screen, in pixels.
/<joint> [up,down,left,right,forward,back] - Sent when we detect a "hit" event, such as a punch forward, which would be "righthand forward".
/<joint>_requiredlength <float> - The current tuning value of how far you must move in a straight line (in mm) before a hit event is generated
/<joint>_pointhistorysize <int> - The current tuning value of how many points are being tracked for hit event detection.
And here are a list of valid joints:
Here is a list of valid messages to send to Synapse:
/<joint>_gettuninginfo - This causes the _requiredlength and _pointhistorysize messages to be sent from Synapse, so that you can see what the current values are.
/<joint>_requiredlength <float> - Use this to change how far you must move this joint in a direction (in mm) to trigger a hit event. Defaults to 150,
/<joint>_pointhistorysize <int> - Use this to change how many points are being tracked for hit event detection. This is essentially a control for how fast you must move your hand to cause a hit event, lower means you must do it faster. Defaults to 5.
/<joint>_trackjointpos <int> - This is the keepalive to cause joint positions to continue being spit out (more about this below). Valid values to pass are: 1, to get _pos_body positions; 2, to get _pos_world positions; and 3, to get _pos_screen positions.
/depth_mode <int> - This allows you to cut the background out of the depth image and only see the user. Valid values to pass are: 0 to see the whole depth buffer, 1 to see only the tracked user (or all person-shaped things if no one is tracked), 2 to see all person-shaped things even if a user is tracked.
I mentioned above that /<joint>_trackjointpos works as a keepalive to keep joint positions spitting out of the Synapse app. I do this because I want to avoid spewing out every joint position in every space every frame, because that could potentially cause packet loss. Instead, you pick and choose which joint positions you want to track. You must tell Synapse every 2-3 seconds that you want to track a joint position to keep it spewing joint positions out. For example, to get the right hand's position in body space, make a patch that looks like this:
This will send "/righthand_trackjointpos 1" to Synapse every 2 seconds, and cause messages like "/righthand_pos_body 150 200 400" to be sent out by Synapse every frame.
As I mentioned on the Quartz Composer page, you must use a Max patch as a passthrough to get this OSC data into Quartz. Using Max as a passthrough also affords you the opportunity to filter incoming data or do math operations in Max before sending them off to Quartz. Look at "quartz_passthrough_plus.maxpat", included in my Quartz Composer example project, to see an example of this.
I also created a Jitter patch (runs on Mac only, sorry) that allows you to get the depth buffer from Synapse into Jitter. Look for it on the Download page.
