The Quantified Artist - Biometric Augmented Reality
One of my favorite sessions was led by Robert Tuttle, exec tech director @frog. A classically trained clarinet player, he has created a new way to experience the symphony. In short, take Microsoft’s latest Kinect sensor to track the biomechanics, a vest that tracks your biometrics (heart rate, respiratory system, etc.), a microphone setup (for pitch accuracy), and custom code—all controlled by a tablet—and you have an awesome new musical performance. Believe me, no one will fall asleep while seeing all musicians’ actions being quantified in real time. It’s one of coolest things I saw this year, and the applications for teaching are endless. Bravo!
Hardware
Hexoskin (biometrics, Bluetooth)
iPhone (app + gateway, Bluetooth)
Microsoft Kinect for Windows v2 (biomechanics, acoustics)
AirTurn (control, Bluetooth)
Microsoft Surface Pro 2 (processing, rendering)
Software
.NET Framework / C# (logic, data binding)
WPF / XAML (UI layout, rendering)
OxyPlot, WriteableBitmapEx (high pert dataviz)
Background services threads
- Pitch detection (NAudio, PitchTracker)
- Finger motion (Kinect DepthFrame + BodyFrame)
- Biometrics (Hexoskin REST API)