It has been fascinating to see all the unexpected applications for Kinect springing up as Microsoft’s sensor technology expands from the Xbox 360 to Windows. But I’ve never before seen anything quite this advanced.
In this piece I try to use magic to prototype or sandbox a possible future where 3D sensors and gestural sensing are intelligent and ubiquitous. Imagine the possibilities of a system like this for day to day communication or education. A digital storytelling tool for the future?
Technically I am combining a HD video camera with the depth camera of a Kinect. I myself become a virtual object on a virtual stage filmed by a virtual camera and projected behind me in real-time. All visuals are generated in real-time and the sleight of hand and augmentation are blended seamlessly to tell the story of storytelling and magic.
The main task was to bring together a very long list of demanding interactive real-time effects – such as hand-tracking, face-tracking, face substitution, particle systems, elements in 3d space – with the exact course of a six minutes show, timed by the given soundtrack. For the development onformative together with checksum5, the developers of vvvv. Likewise Tebjan Halm and Enrico Viola provided additional vvvv development and preliminary studies done in openFrameworks.
The aim in »Magic and Storytelling« was to find a magical and engaging way to enhance a live performance with interactive graphics. The live component could be anything from a lecture to a performance. The augmentation layer is not pre-rendered. It is generated by a data flow system in real time and in reaction to the live elements. I think that this type of magic can help us to map out what that part of the future might feel like, where 3D projection and gestural sensing are intelligent and ubiquitous. The collaboration with onformative and checksum5 created the perfect synergies to explore my vision. vvvv proofed to be the ideal tool to develop this graphics and computational intensive app. Magic and illusion are a good sandbox for creativity. Adding the element of illusion to an existing technology allows us to create things that are otherwise not possible. An illusion takes a technology beyond its natural boundaries. Magic is a unique tool for pre-visualization. - Marco Tempest
Hardware: One PC, Kinect, HD Camera, Two Monitors.
Software: openFrameworks for preliminary studies, vvvv for final.