15 Aug '08

Home-brew Multi-touch

flash, multi-touch, open source

At IDEO we’re all about building to think. Learning from books and websites and product demos is cool, but we think the really good stuff comes when you get in there and start messing around for real.

In the case of multi-touch interfaces, that meant building a system we could start prototyping on. What we wanted was:

  • a multi-touch display large enough to facilitate use by several people at once
  • an API for flash that would let us quickly prototype multi-touch interfaces and applications

It took us about 5 weeks to get everything together. Kyle, one of our all-around gearheads, had already been building a drafting-table-style FTIR system in his garage which helped kick-start us.

We knew we wanted our system to be larger and vertical, though, so we went back to the drawing board on hardware. We had some old projection surfaces from a project we had done long ago in storage.

With that as our base we designed a system around a short-throw projector and a wide-angle IR cam. This allowed us to build a pretty large rear-projection display without the depth requirements of Han’s systems.

Meanwhile, Kyle and Nicholas (another all-around whiz) were hard at work building a multi-touch server and an API that could be used to quickly prototype interactions. They built the back-end in Processing. It’s a great platform for rapid prototypes, particularly where there’s hardware-software integration involved.

On the front end we chose to create an API for Flash because it lets us realize our ideas quickly and use a wide variety of media, from video to vector graphics. Nicholas’ Flash API is pretty cool: a system that allows anyone who knows ActionScript to quickly mock up multi-touch using reference to “fingers” (e.g. when two fingers are down on this image, scale it…). 

Using our server and API, Nicholas was also able to very quickly throw together a handful of demos. Take a look:

Want to jump into the multi-touch arena by creating a system of your own? We’ve just published the code from our own prototype system at Google Code: it should be all the software you need to make your own FTIR system: a back-end app to run a multi-touch server, and an API to create your multi-touch flash applications.

You can find the source code for the FTIR server and the Flash API at http://code.google.com/p/ideo-multitouch/. Everything is licensed under Creative Commons.

This stuff is definitely not polished to commercial application standards: It’s use-at-your-own-risk, unsupported code. We’ve all got day jobs and can’t offer much by way of technical support, but if you find it useful we’re happy to have you make use of it.

And by all means if you make something you’re excited about with it, send it our way!

35 Comments:

  1. rich rainbolt

    21/08/2008 at 11:49 am // Permalink

    exciting stuff cannot wait to start playing

  2. John Warno

    21/08/2008 at 8:45 pm // Permalink

    I saw some of the stuff at Flash Forward 2008 and thought it was really cool. It is definitely in our future. I can’t wait to experiment with your code.

  3. Dustin

    22/08/2008 at 9:20 am // Permalink

    Any chance you could hook us up with that extra file needed on Intel Macs? The site that is hosting it is down at the moment. It would be awesome if you could email it to me…

  4. Chris

    22/08/2008 at 9:30 am // Permalink

    Excellent work and very nice to push the industry forward with sharing your source. I’ve been working on similar experiments with webcam interaction rather than multitouch. Similar methodologies in technologies that will be the future of Flash.

  5. David A. Mellis

    22/08/2008 at 2:23 pm // Permalink

    Very cool! I like the games; it’s nice to see some new ideas for uses of a multi-touch system. Plus, it’s great that you’re releasing the libraries as open-source. The code looks pretty intense; I’m sure a lot of people will find it useful.

  6. justin powell

    25/08/2008 at 2:17 pm // Permalink

    thanks for sharing all this great info - process, code, and samples. what an excellent way to use flash and processing for rapid prototyping.

    i hope more examples start popping up once word gets out. keep up the great work!

  7. Pauric

    26/08/2008 at 5:44 am // Permalink

    Thanks for publishing this, building a ftir coffee table just move to the top of my projects list. /pauric

  8. Miquel

    27/08/2008 at 3:16 am // Permalink

    Why is the client in AS2?
    Will you develop the client in AS3? Do you need some help to do that?

    Thanks,
    Miquel

  9. Christian Moore

    29/08/2008 at 4:50 am // Permalink

    Really great stuff guys… Looking forward to trying it out.

    Ask Nicholas if he’d ever want to help work on a AS3 Touch API :)

    Thanks for your contributions, where about are you guys located? Palo Alto?

  10. Camilo Andrés Ayala Monje

    12/09/2008 at 3:41 pm // Permalink

    Hi…
    Thanks for published this codes…
    Congrats to make this multitouch sistem…
    The idea and develop of there is wonderful…

    Thanks to all

  11. James

    18/09/2008 at 5:33 pm // Permalink

    Hi, I found your blog on this new directory of WordPress Blogs at blackhatbootcamp.com/listofwordpressblogs. I dont know how your blog came up, must have been a typo, i duno. Anyways, I just clicked it and here I am. Your blog looks good. Have a nice day. James.

  12. Iron

    19/09/2008 at 3:20 pm // Permalink

    Hello,
    I’m trying to use.
    All ok with the server, I install and run. But I don’t know what happned with the client, just doesn’t work! How can I run de “swf” to connect correctly with the server?

    The other problem: When I open the “*.fla” and export, the swf stop work.

  13. David Sanchez

    22/09/2008 at 12:52 pm // Permalink

    Simply, Amazing, Thanks for inspiring us.

  14. Kirit Tanna

    26/09/2008 at 12:49 am // Permalink

    Hello,

    I am currently building a FTIR screen with help from the NUI group forums and I found your video very exciting more because your screen works in daylight. Can you share more details about the hardware and where it can be purchased… From what I’ve found -

    Wide angle IR cams -
    http://www.ir55.com/MikroScan.html

    Short throw projector -
    http://www.techgadgets.in/displays/2008/03/benq-unveils-mp771-short-throw-ratio-lens-enabled-projector-in-india/

    Can you suggest which one is suitable? Thanks!

  15. Angus

    30/09/2008 at 3:29 am // Permalink

    would be really love to see the Flash API in AS3. :)

  16. Gentry

    02/10/2008 at 11:14 am // Permalink

    @Kirit: Like all FTIR systems, ours really struggles in ambient daylight. The video you saw was shot in a room with only florescent lighting.

  17. Gentry

    02/10/2008 at 11:14 am // Permalink

    @Angus: Are you volunteering? ;)

  18. jwopitz

    03/10/2008 at 10:42 am // Permalink

    @Gentry

    I concur, I would like to see this in AS3. Seeing as Flex is becoming quite ubiquitous in the RIA market, this would be a valuable asset in kiosk/installation development. In fact, I have a personal non-OpenSource project that could make great use of this if it where in AS3.

    I could certainly volunteer some time to help you port it over if you like. Let me know the details.

  19. Kirit Tanna

    06/10/2008 at 3:20 am // Permalink

    Thanks Gentry,

    I still see this as being more feasible than a dark room.
    Any tips on how to setup the environment?

  20. scintilla

    20/10/2008 at 6:54 pm // Permalink

    This looks awesome! Did you just use a rear projection screen as your FTIR surface? Or did you have an additional surface on top of your projection screen?

  21. Diego Uribe

    26/10/2008 at 7:38 pm // Permalink

    Great stuff IDEO’ers
    By any chance, is the brainstorming/ideation multitouch softaware displayed in the video part of the open code? Is it available to the public?

    Salu2,

    Diego

  22. sparsh

    07/11/2008 at 2:47 pm // Permalink

    Hello Multi-touch Fans!

    We are Announcing Sparsh UI, an open source multi-touch API that works across devices and
    platforms.

    The Multi-Touch Lab at Iowa State University has released the open-source codebase Sparsh UI (v1.0b), which allows easy development of multi-touch
    applications that work across devices (e.g. FTIR touch table, Stantum, IR bezels, cardboard box touch devices) and across platforms (Java, C++; Mac,
    Windows, Linux).

    Here is what Sparsh UI offers!
    – Includes gestures (don’t need to program them),
    – Includes a Multi-touch Simulator so you can develop apps without the multi-touch device connected (Windows only for now)
    – Does not depend on hardware(An adapter for TouchLib enabled devices is provided alongwith)

    Coming soon:
    —Support for UserIDs for collaboration on the same multi-touch device [to identify different users on the same device]
    —Support for Static Gestures.

    Download and try it out at http://code.google.com/p/sparsh-ui/
    Find documentation on our wiki and sample applications on our home page, discuss it on our
    Google Group [ http://groups.google.com/group/sparsh-ui

    Feel free to contact us for any questions!

    –The Sparsh UI Team
    Virtual Reality Applications Center
    Iowa State University

Leave your comments