I’m sure by now everyone in the world has seen some ‘hack’ involving a Microsoft Kinect and some clever programming. Microsoft’s Kinect is a computer-vision dream come true; an infrared depth sensor with impressive resolution and a standard RGB video camera. Throw in some microphones for sound capture and you have a force to be reckoned with.
Released in 2010, the Kinect was developed for the Xbox 360 with the technology and legwork done by PrimeSense who wrote the libraries and on-board processing for the vision data. However, soon after it’s release, the Kinect’s USB protocol was hacked and libraries were written for pretty much every programming language under the sun and coders got to work.
With all of the crazy possibilities that the Microsoft Kinect offers, we got a Kinect for the Palo Alto office to play around with and learn about the possibilities it offers for storytelling, prototyping, and straight-up awesome.
I wanted to give a shot at a simple program to demonstrate the basics of the Kinect for a workshop. I thought it’d be fun to make the most bare-bones interaction possible to get peoples’ brains going on the possibilities with this rich of data. The program is a demonstration of the ease of dynamically filtering moving objects based on depth and using these objects to interact with non-physical objects of our choice. You can see the grayscale depth data in the silouhetted person + couch outline in addition to the noisy outline along with some smoothed outlines. The IDEO boxes have their own physical characteristics (weight, friction, etc) and use the person + couch outlines as boundaries for interaction.
All in all, it took a whopping 2-3 hours of coding, most of which was library and driver issues at the beginning, to code this quick demo, so you get a sense of how easy it is to work with in terms of developing.
Information on how it works and links to the source code are provided after the jump.
Behind the scenes, it is a basic application of rudimentary computer vision techniques. The premise is to use the depth field to outline the moving object(s) and others nearby, use that outline to create a virtual body that other virtual bodies will react with, and then superimpose the depth field over the outlined body in addition to drawing the additional bodies (IDEO letter boxes) for visual purposes.
The depth field is used to make a binary image of the moving person. A contour-finding function is applied to the binary image (really easy since it’s binary) in order to generate an outlined silouhette of the person and objects that are nearby in the depth field. This outline is then used to generate a body in the virtual physics world which the boxes can bounce off of, exactly like they would in a 2D world. You can add boxes which spawn with random sizes and physics properties using your mouse and clicking wherever you want them to appear from. The binary image is then replaced with a white background and the grayscale depth map of the objects within the chosen depth range. Lastly, the outline of the silhouette is displayed along with several iterations of smoothings in order to show their effects on reducing noise in your outlines. This helps not only aesthetically but also computationally to eliminate bodies from accidentally intersecting other bodies.
The coding was done in openFrameworks, a great open source C++ library which focuses on image processing, video, and graphics libraries. One of the best parts about openFrameworks is the inclusion of the OpenCV library which is the standard for image processing and computer vision applications which is perfect for this application.
The ofxKinect add-on was used to access the Kinect and get depth and RGB data. ofxBox2D was used for the physics behind it, which was ported over to openFrameworks by our very own Todd Vanderlin. The code for the project can be downloaded from the Google Code repository here. The code requires the following addons to the openFrameworks core library: ofxKinect, ofxBox2D, and ofxOpenCv (included in the openFrameworks FAT installation).
jswanson
13/04/2011 at 9:20 am // PermalinkAwesome proof of concept. My Kinect salivation level just jumped from moderate to excessive.
Evgeny
13/04/2011 at 9:46 am // PermalinkThis is awesome!
jerry
14/04/2011 at 10:48 pm // PermalinkCode archive does not include the required images (i d e o) .jpg or indicate their size, etc. What sizes are you using? Could you add the images to the code.google archive? thanks.
jerry
14/04/2011 at 11:40 pm // PermalinkProject also requires a couple more libs not included in the of FAT…
ofxTriangle, which I found at http://code.google.com/p/ofx-kikko/
and contourSimplify, which I found over at
http://joelgethinlewis.googlecode.com/svn/!svn/bc/18/trunk/cosmos/cosmosOF/src/computerVisionUtils/
Seriously? wow. thats a bit of extras to hunt down… not quite open source dependency hell yet (v2 of this requires v2.1 of that and v1.2 of the other, but the other requires v2.2 of that)
jerry
15/04/2011 at 12:28 am // PermalinkDrat! Another issue (btw, building on OSX using XCode 3.2.5, and of_preRelease_v0062_osxSL_FAT)…
testApp.cpp defines
vector rects;
but when compiled, we get ..
class ofxBox2dRect has no member named width
class ofxBox2dRect has no member named height
a whole bunch of times
It seems that ofxBox2dRect does not have width and height members, at least as provided in v0062_osxSL_FAT. I added them and set em in setup() and Eureka! it works!!
We bounced around some blank squares for a bit rejoicing.
Then went off to make four jpg images
and drink to our success.
Mahalo!
brad
15/04/2011 at 9:30 am // Permalinkawesome to hear jerry! forgot to add the jpgs, my bad, but I believe any square jpg image will work and the code should autoscale them to the size of the box2d rectangle (square). glad you got it working!
jerry
15/04/2011 at 2:36 pm // PermalinkBrad – you are welcome. Write-up on the adventure is in our blog about the Maui Makers meeting last night… http://www.mauimakers.com/blog/?p=670
Alex Porter
17/04/2011 at 5:16 pm // PermalinkThat’s awesome, Brad!
gabe
25/04/2011 at 1:10 pm // PermalinkHey guys:
I’m doing some color work and want to insert a split background on this one color on the left half another on the right. Then I want the squared to be children colors of the two backgrounds. I made a file in quartz composer that creates the backgrounds and squares with functionality that rotates each background through the full hue spectrum. Unfortunately, I’m an xcode and openFrameworks newbie and while I tried to get into the code, I really have no idea where to start.
With Jerry’s help, we replaced the falling squares with our own jpg files. To get them to be slowly changing colors based on the backgrounds might be too much, but it would be great to try.
The quartz composition is the last one uploaded in this chain: (sorry you might have to be in the kineme forum to read)
http://kineme.net/forum/Discussion/DevelopingCompositions/MoreColorMath
ross
23/06/2011 at 4:20 pm // PermalinkInteresting. I’m surprised you were able to get the fidelity required to hold an object between thumb and finger.
The official sdk doesn’t have a thumb joint.
tom
17/08/2011 at 10:42 am // PermalinkThis is great
suhit
19/02/2012 at 10:22 pm // PermalinkI am trying to run this tutorial but I am keep getting errors. can somebody please share the full working code. It would be great help for me.
Łukasz
02/03/2012 at 8:38 pm // Permalinksuhit read above comments, they have a fix for it.
Łukasz
02/03/2012 at 8:40 pm // Permalinkp.s. suhit I responded on your post in oF forum
gave some additional resources.
daniel
31/07/2012 at 3:11 pm // PermalinkI had many problems developing the exercise, especially in the location of the libraries, someone has the code that works, I have like 300 issues.
how do I link the libraries to a new xcode project to not be updating the routes all the time.?
thanks
Sanman
07/08/2012 at 3:10 am // PermalinkHi there,
The code is not present here anymore
http://code.google.com/p/ideokinectdemo/source/checkout
svn checkout http://ideokinectdemo.googlecode.com/svn/trunk/ ideokinectdemo-read-only
Has it been moved?
Thanks,
Sandeep
Dave Vondle
13/09/2012 at 6:22 am // PermalinkTry going to: http://code.google.com/p/ideokinectdemo/ then navigate from there.
NikJOsh
06/11/2012 at 1:04 pm // PermalinkI am running in to problems and looking with the code provided
I am getting a number of errors like:
1…\..\..\addons\ofxCosmos\cosmos\cosmosOF\src\computerVisionUtils\polylineSimplify.h|13|error: conflicting declaration ‘typedef struct Segment Segment’|
for PolylineSimplify.h
2…\..\..\addons\ofxBox2d\src\ofxBox2dPolygonUtils.h|32|error: ‘Segment’ has a previous declaration as ‘typedef struct Segment Segment’|
for ofxBox2dPolygonUtils.h
any one has any clue of these??
Brad Simpson
13/11/2012 at 10:17 pm // PermalinkIt looks to me from that debug error that there are multiple declarations of the Segment structure that you are defining in PolylineSimplify.h. You might be using a newer version of ofxBox2d given that there is now a conflict and there used to not be. Check in ofxBox2dPolygonUtils.h and see if the Segment structure is the same one that is defined in the PolylineSimplify.h. If so, comment one of those declarations out and you should resolve that conflict.