State of Mind
What a rollercoaster ride the last five months have been. Since my last post, there are several updates to address including the completion of my thesis research and presentation at SIGGRAPH 2013. This post serves to be a reflection and overview of the hard work that was accomplished over the past several months and the incredible opportunities that resulted from it.
Thesis Complete - "Virtual Production: Beyond Big Studio Production"
In the last blog update, I presented the key features of the (then called) MobileVCS in the form of a demonstration promo video. Since then, a lot of work has been done to the system to emphasize the accessibility and intelligence of the system, which has now been rebranded as the SmartVCS.
First and foremost, the system needed a polished redesign, one with an ergonomic control grip and slimmer profile. To accomplish this, I swapped out the copper tubing with lighter wooden dowels and proceeded to 3D print a custom fitted casing that would better grasp each of the Playstation controllers (no need for zip ties & duct tape). After modeling my new design, I was able to print the casings with the help of Drexel's Product Design major and the Makerbot 3D printer. Assembling the pieces was straightforward enough that I was able to create multiple systems for my booth at the SIGGRAPH conference.
The following images showcase the process from initial designs to the final, sleeker SmartVCS prototype.
Beyond just a new form factor, there was a major software and user interface overhaul that was refined upon over the past couple months. As of this writing, the following are the list of features developed for the SmartVCS:
Native iPad app running the Unity game engine connecting directly to the Playstation Move.Me server.
Freespace, realtime control of the Virtual Camera position and orientation via the Playstation 3, Move controller, and Eye camera.
Camera controls for tracking, dollying, panning, and booming via the Playstation Navigation controller's Analog sticks.
Custom Shotcode setup for saving recorded camera takes.
Cinema Prime Lens adjustments through quick-toggle face buttons on the Navigation controller. Zoom is accomplished using the controller's soft-touch triggers.
On-Screen menu for setting the Virtual Camera’s recording aspect ratio.
Bi-Planar Tracking Sensitivity Adjustment through the Navigation controller's face button. This would enable a user to translate a greater or lesser distance in the virtual world in relation to their physical motion in the capture volume.
Scene Modification through touch-dragging of on-screen objects in camera space.
Record & Playback a Virtual Camera shot using the aforementioned freespace and camera controls. Replay controls include standard playback functions as well as the ability to scrub across a time slider to pin-point a specific replay position. Capture playback includes camera and animation data.
Bake & Save camera takes directly on the iPad in a text-file format as a comma separated series of position and orientation values. Data can then be retrieved by connecting the iPad to iTunes and visiting to the app's File Sharing menu where the user will be able to download said text files. From there, a Python parser was developed for Maya to convert the data into a baked camera within the animation package. The resulting camera contains all the information that was captured using the SmartVCS.
Additional tests included:
Collaboration using multiple SmartVCS systems and an actor captured Live with the Microsoft Kinect. This "tele-capture" demonstration allowed several camera operators to interact in a live virtual set with a remote actor performing in real-time.
Match moving & 3D Integration of virtual objects over a live video backplate using the iPad's rear-facing camera. This test showcased the ability to accurately track, with low latency, 3D objects in a real world environment without the need for Augmented Reality markers.
The control setup overview described was designed to avoid on-screen clutter, and place dynamic menus on the screen while giving quick, tactile controls through the Navigation controller. Here is the control setup developed for the SmartVCS:
Overall, my thesis research was well received by the university at several public showcases including the Drexel University Research Day 2013 and was awarded the honor of Drexel University Masters Research Award 2013. Due to the innovative nature of the results, I have battled through the legal process with the help of the university's Law School and was able to protect my R&D work with a provisional patent.
As with any Masters degree completion, the thesis defense included an interactive presentation and showcase where students, faculty, and industry professionals gathered to evaluate the work presented. After two years of hard work and dozens of pages of research written, my thesis titled "Virtual Production: Beyond Big Studio Production" was officially defended.
I'd like to give sincere thank you to my advisor Dr. Paul Diefenbach who has been incredibly supportive in my research efforts. In addition, a thanks goes out to my co-advisors Jeremy Fernsler and David Schwartz for their input and industry perspective as well as Drexel University Digital Media program director Dr. Michael Wagner for his constant support. Each of you were instrumental in shaping the system to its working prototype state that it is today.
Interested in reading my paper and learning more about the system?
girishbalakrishnan.com/vault/GirishBalakrishnan_MastersThesis.pdf
SIGGRAPH 2013: Talk & Booth Experience Post-Mortem
Where to begin! I was offered the opportunity to present my research work at the international SIGGRAPH Computer Graphics Conference which took place in Anaheim, California. The SIGGRAPH Conference hosts students, researchers, and industry professionals in the field of computer graphics to share the latest and greatest technology and innovation in this growing field. My presentation was in the form of a 40-minute talk as well as a week-long interactive booth on the show floor.
This being my first conference presentation, which I was the sole contributor for, was daunting to say the least. A lot of preparation and coordination was done in order to ensure the best showcase of the SmartVCS system. This would be the first official public demonstration of the system that had to stand through 8 hours of rigorous user interaction every day for 5 days - not an easy task for a prototype system.
The booth consisted of two key interactive demonstration of the SmartVCS, the first displaying the mobile virtual camera system functionality I developed and the second demoing "tele-capture" with a SmartVCS & Microsoft Kinect.
Throughout the week, industry enthusiasts were given the opportunity to get some hands-on time with the system. This was honestly the most rewarding experience of the conference. Up until this point, much of my research and discussion with industry professionals was primarily theoretical, but with this booth I was able to see their reactions to using such a device Live - and boy were they impressed! The booth became a popular stop through word of mouth at the conference, attracting members from big studios, software companies, and small teams alike.
It was important to reiterate that the SmartVCS prototype is designed as a research project, not a commercial product. While the system does have commercial potential, in its current form, it is a glimpse of what is possible with such technology. No, you can't buy it just yet.
Mid-conference week, I was given the opportunity to give a talk on the research methodology of my system titled "SmartVCS: Shooting Avatar on your iPad?" and was requested to give a live demonstration on stage. As risky as a live demo, especially in a conference hall with conflicting wireless signals, the presentation went flawlessly with a strong crowd showing. I was humbled to have members from companies such as Walt Disney Research, WETA Digital, Lucasfilm/ILM, Dreamworks Interactive, Sony Computer Entertainment of America, Microsoft Studios, Autodesk, Unity Technologies, Tippet Studios, Framestore, SideFX, Giant Studios, Proof Inc, and Natural Point in attendance at the talk. The reaction was so encouraging that one member went so far as to say, "In my 15 years of coming to this conference, that was the most innovative idea I have seen in a long time." While I take such statements with a grain of salt, it is great to see people interested and excited about my work - something I was certainly not expecting!
This was the third time I have attended the conference, but first time as a contributor - what a difference that was. Unlike previous experiences where I spent more time perusing the conference exhibition floor and attending talks, this time I was focussed on showing off my hard work at the booth. I could not have asked for a better conference experience and look forward to following up with the industry connections I made.
I have put together an overview video of my talk presentation that features the latest SmartVCS design and the aforementioned new interface functionality. Let me know what you think: https://vimeo.com/girishb/smartvcs
Looking into the Future
Over my time working on this project, I have thoroughly researched cinematic principles, explored virtual camera capture techniques, and have become incredibly passionate about the previsualization pipeline and its importance to this industry. This area of preproduction, in my opinion, deserves the most attention, especially in a time where production expectations are skyrocketing and studio budgets are shrinking. We are at the crossroads of games and cinema where games are reaching new cinematic heights (ie. Bioshock Infinite & Last of Us anyone?) and film productions are beginning to leverage advanced real-time game engine technologies. By developing tools to lower this cost, we can ensure that every studio big or small can plan out their shots in a pressure-free interactive environment.
As I begin to evaluate my options for the future, I am excited to continue my work with the SmartVCS as a side project. I plan to work with a team at Drexel University to continue developing the system further and possibly releasing a version of the software within the coming year. The industry is hungry for new interface devices that would broaden the user-base of virtual camera capture and the SmartVCS is just one of those affordable & accessible tools that could be used on any film or game cinematic production. Intelligence is key to the system and I plan to collaborate with researchers working in the area of smart camera systems to layer cinematic intelligence to this capture system. Beyond this industry, the system at its core is an interface device that could be used as a visualization tool in other fields such as architecture and medicine.
One last bit of news, the SmartVCS was nominated for the category of "Best Student Project" at the Unity Awards taking place at the 2013 Unite Conference and is now considered an international finalist for the award! If plans work out, I will be attending the conference taking place in Vancouver, Canada at the end of August. If you will be in attendance, definitely let me know. Very excited for this opportunity and consideration.
Two years ago, when conceptualizing this idea of democratizing virtual cinematography, if you had told me that I would be given the opportunities I have been blessed with today, I would have called you crazy. The future looks bright, adventure is out there after all :)
