Kinect Interactivity Demo

A simple vision and interactivity experiment with the amazing Microsoft Kinect

April 12, 2011
Reading Time
3 minutes

I’m sure by now everyone in the world has seen some ‘hack’ involving a Microsoft Kinect and some clever programming. Microsoft’s Kinect is a computer-vision dream come true; an infrared depth sensor with impressive resolution and a standard RGB video camera. Throw in some microphones for sound capture and you have a force to be reckoned with.

Released in 2010, the Kinect was developed for the Xbox 360 with the technology and legwork done by PrimeSense who wrote the libraries and on-board processing for the vision data. However, soon after it’s release, the Kinect’s USB protocol was hacked and libraries were written for pretty much every programming language under the sun and coders got to work.

With all of the crazy possibilities that the Microsoft Kinect offers, we got a Kinect for the Palo Alto office to play around with and learn about the possibilities it offers for storytelling, prototyping, and straight-up awesome.

I wanted to give a shot at a simple program to demonstrate the basics of the Kinect for a workshop. I thought it’d be fun to make the most bare-bones interaction possible to get peoples’ brains going on the possibilities with this rich of data. The program is a demonstration of the ease of dynamically filtering moving objects based on depth and using these objects to interact with non-physical objects of our choice. You can see the grayscale depth data in the silouhetted person + couch outline in addition to the noisy outline along with some smoothed outlines. The IDEO boxes have their own physical characteristics (weight, friction, etc) and use the person + couch outlines as boundaries for interaction.

All in all, it took a whopping 2-3 hours of coding, most of which was library and driver issues at the beginning, to code this quick demo, so you get a sense of how easy it is to work with in terms of developing.

Information on how it works and links to the source code are provided after the jump.

Behind the scenes, it is a basic application of rudimentary computer vision techniques. The premise is to use the depth field to outline the moving object(s) and others nearby, use that outline to create a virtual body that other virtual bodies will react with, and then superimpose the depth field over the outlined body in addition to drawing the additional bodies (IDEO letter boxes) for visual purposes.

The depth field is used to make a binary image of the moving person. A contour-finding function is applied to the binary image (really easy since it’s binary) in order to generate an outlined silouhette of the person and objects that are nearby in the depth field. This outline is then used to generate a body in the virtual physics world which the boxes can bounce off of, exactly like they would in a 2D world. You can add boxes which spawn with random sizes and physics properties using your mouse and clicking wherever you want them to appear from. The binary image is then replaced with a white background and the grayscale depth map of the objects within the chosen depth range. Lastly, the outline of the silhouette is displayed along with several iterations of smoothings in order to show their effects on reducing noise in your outlines. This helps not only aesthetically but also computationally to eliminate bodies from accidentally intersecting other bodies.

The coding was done in openFrameworks, a great open source C++ library which focuses on image processing, video, and graphics libraries. One of the best parts about openFrameworks is the inclusion of the OpenCV library which is the standard for image processing and computer vision applications which is perfect for this application.

The ofxKinect add-on was used to access the Kinect and get depth and RGB data. ofxBox2D was used for the physics behind it, which was ported over to openFrameworks by our very own Todd Vanderlin. The code for the project can be downloaded from the Google Code repository here. The code requires the following addons to the openFrameworks core library: ofxKinect, ofxBox2D, and ofxOpenCv (included in the openFrameworks FAT installation).