The Challenges of Realtime Performer Tracking for Moving Lights
Tracking a performer on stage in realtime is not something new. There are, in fact, a couple of companies that sell technology bundles that do this already for sound (StageTrackerFX and TiMax Tracker) using ultra-wide band RFID tagging. Many hospitals and shipping companies use similar set ups to track patients and packages, though both StageTrackerFX and TiMax Tracker use a few more receivers/communicators and some special algorithms to more quickly accurately tracker a person in motion. At first I thought I might be able to utilize such a system for Wonder Dome to track a performer or object with light, but after some research I came across a few problems. First, the price tag on a system like this (one geared specifically for theatre or otherwise) minimum $5,000-$10,000, far beyond affordable for the Dome an most others interested in realtime reactive lighting. Second, after talking with multiple companies, I discovered that this type of tracking is typically accurate only to within 6", and often have a 1-2 second tracking delay to adjust for accuracy within the algorithms.
That sort of thing might work for sound tracking (balancing a speaker system so that it appears that a performers voice is coming from wherever they are on stage), but the human eye is far less forgiving when it comes to light. Visually, if a performer is standing even a few inches off their mark, it is noticeable to an audience member; we are a bit more forgiving if a "sound" is a foot or two off to one side of a performer.
With this in mind I turned back visual tracking. Using a camera has it's own set of issues, however. In order to pick out a specific individual or object, the camera or an image processing software must be capable of either blob tracking or skeleton tracking. In addition, in order to accurately track a performer or object within Wonder Dome specifically, a camera would need a viewing angle of at least 90°. Blob tracking programs and protocols tend to be fairly resource intensive on a computer's processor, but some cameras have this built in already. The camera would also need to be able to remember or pick out specific individuals so as not to track random audience member, just a performer. This can be done is via color recognition or using special infrared filters, LEDs and/or special IR reflective materials.
Thanks in no small part to the video game and robotics industries, several cameras specifically made for interaction already exist on the market, namely Microsoft's Kinect, Sony's Playstation Eye, Nintendo's Wii Remote, and Charmed Lab's Pixy. The Kinect utilizes onboard skeleton tracking and a tunable infrared camera which allows for the camera to calculate depth information. Skeletal tracking provides excellent data from a front view perspective, but a top down view (which we are planning to use in Wonder Dome) tends to confuse the Kinect skeletons. The Kinect also only has a camera angle of 57° horizontally and 43° vertically, which means I would need to matrix between 4 and 6 Kinect cameras to be able to track the entire Dome. As an added complication, each Kinect needs to be on separate USB bus to function properly; most desktop computers come equipped with at least two, so potentially 3 computers would need to be networked together for this method to work.
The Nintendo Wii Remote (Wiimote) has a tiny on board IR camera that does a couple of remarkable things; built in blob tracking, and data rate for 100Hz. The in board blob tracking is incredibly useful, as it frees up resources from processing computer, and the data speed makes the camera super responsive. There have been numerous hack-and-mod projects for them, so documentation is abundant on almost every aspect of the device. There are some extremely limiting factors to the Wiimote, however. In in it's most basic and unmodified form, the Wii Remote has a camera angle of 33° horizontal, 23° vertical requiring a matrix of 12 cameras to be effective in the Dome. Additionally, the remotes themselves are battery powered and communicate via Bluetooth to either computers or a Wii. This means a maximum of 4 Wiimotes can be connected to one computer before data clogging occurs, which means 3 computers for processing, and batteries need to be changed regularly. Osculator or Glovepie can both be used to read data from the Wii remote. As I mentioned previously, the Wiimote has been the subject of numerous modifications and hacks, and there is a lot of documentation on how to remove the camera itself from the Wiimote, and mount it as a standalone camera, or, in some cases, in a matrix for robotic vision. I have a separate post I am in the process of writing regarding the math and theory aimed specifically for the Dome and a matrix of 12 Wiimote cameras.
The Playstation Eye, all by itself, is more of a webcam than a motion tracking device. It has a frame rate of 60Hz, a maximum camera angle of 75°, and can only track objects in concert with a special Move controller. Peau Productions, a company specializing in custom and DIY touch surfaces, sells modified Eye cameras with swappable lenses, going up to a viewing angle of 122°. The most wide angle lenses are expensive (up to $440), but also are equipped with integrated distortion correction, making the image the sensor sees is not curved in a way that can effect tracking. These cameras modifications were developed to work along side a light weight, open source, blob tracking software called Creative Core Vision (CCV). CCV works on both Mac OSX and Windows, is incredibly customizable for blob tracking, works with many different cameras, and can send it's data with OSC protocols. This is likely the camera set up I will use with Wonder Dome, despite the price tag of around $550. That said, a matrix of 12 Wii remotes would cost $480 just for the remotes, not to mention more gear to get them all to work happily together and a development period for that as well, so $550, doesn't seem like such a bad price tag!
Last, but certainly not least is Pixy from Charmed Labs. Now, Pixy is my dream camera: and arduino based camera with a speed around 100Hz, swappable lenses, and built in blob tracking. Really smart blob tracking. Honestly, check out their Kickstarter page for all the details, but in a nutshell it can be trained to remember specific color signatures and track many objects simultaneously. The video is pretty killer. Price: $75 with a stock 75° lens. So what is my hold up? Well, unfortunately these incredible cameras won't start shipping until February for Kickstarter rewards, with an estimated arrival date of "well into March". Wonder Dome is presenting starting March 19th, so I'm hedging my bets that I wouldn't be able to work with a Pixy until the week before at best, which is way too late. In future iterations of the Dome I'll be very excited in seeing what a Pixy can do however!
I'll post some screen shots, videos and pictures of the Creative Core Vision and some camera/tracking work in the near future.