In January, Jerry O’Leary and I launched a Kickstarter campaign (currently <3 days to go) for the world’s thinnest watch, the CST-01. This is a watch we have worked on outside of project work while here at IDEO. On IDEO Labs, our aim is to share our process and give our readers an “under-the-hood” look at some [...]
Continuing our multi-touch research, we’ve been working on turning an off-the-shelf rear projection TV into a multi-touch display. This screen has the best width-to-depth ratio of any multitouch system (67″ diagonal viewing area and only 16″ deep). This is also the first example of hacking a multi-touch system into an off-the-shelf television. The system we settled on uses very few additional components and could potentially be applied to any rear-projection TV.
Setup and Theory
The system has four IR lasers mounted in the corners. Each laser has a line generator on it (line generators are put on laser-levels to create a line of laser light). The four lasers produce a plane of infrared light across the entire surface of the screen. When a finger intersects this plane of light, the light illuminates it. We have two cameras inside the TV that look at the interior of the screen from the inside. These cameras have filters on them that only allow infrared light through. The cameras see the “blobs” of infrared light and track these points.
Below, we’ve posted as much information as we have to enable you to build your own version!
tbeta for multi-cameras
We got in contact with the people at NUI group and offered to work on multi-camera support for their release of tbeta. (Tbeta is the software that accesses the cameras and turns the video feeds into touch events for controlling other programs (Flash, Processing, etc.)). At the time, tbeta was only compatible with one camera. They let us have a go at the source code which is written c++ with a fantastic set of libraries aimed at creative coding and native cross-platform operability called openFrameworks. We got a really rough version of the multiple-camera tbeta working which is a pre-release of this functionality and will eventually be rolled into a real tbeta release. We would like to invite anyone to test this out and let us know how it works!
NUI group will be releasing source and a new project management system for developing various NUI Group projects (including tbeta) within the next two weeks on their website.
Download (Mac only) : tbeta multi-cam pre-release test version
This will require you to patch the source and recompile the instructions for this are after the link.
macam for multi-cameras
We experienced some problems with multi-camera support for macam when two of the same model of camera were connected to the same system. The issues and fix are detailed on the openFrameworks forum here.
Theo from openFrameworks fixed the macam source, and we have it mirrored on our google code repository:
Download : macam-for-multi-ps3eye
You can fiddle with the camera settings and click “save settings” to adjust the brightness and different parameters of the camera feeds with this software.
FLOSC, AS3 Libraries and examples
These were done by the people at NUI group and packaged with their touchlib software. Instead of downloading all of the touchlib software, all you need are FLOSC and the AS3 Libraries. I separated them out and put them with everything else here.
To use, you first need to open tbeta, then open up the terminal, change the directory to the FLOSC directory:
then run the .jar and specify the ports in and out:
java -jar flosc.jar 3333 3000
then you can try any of the example .swf files in:
Parts and Materials
67″ Samsung HL67A750 Rear Projection TV, from Amazon – $1857
This TV was chosen because of its size and because it uses LED light sources. This means quick startup time, no replacing burned bulbs, and low power consumption. We needed something we can leave on during the day and not worry about having to replace bulbs.
2x PS3 Eye cameras from Amazon – $33 x 2 = $66
These cameras are designed for video processing and are cheap. Ideal!
4x 780nm 100mW Laser Module with 120° line generator, from aixiz.com – $28 x 4 = $112
780 nm laser safety goggles, from aixiz.com – $150
4x 1/8″ Mono Panel-Mount Audio Jack from Radio Shack – 2 packs of 3 @ $4 = $8
4x 1/8” Mono Phone Plug from Radio Shack – 2 packs of 2 @ $3 = $6
Power Supply IC, LM317T from Digikey – $0.77
75K resistor from Digikey -$0.32
200K trimpot from Digikey – $2.76
Grid-Style PC Board with 371 Holes from Radio Shack – $2
Black ABS plastic sheet from McMaster-Carr – $16.34
1/2″ ABS plastic from McMaster-Carr – $6.42
3/4″ ABS plastic from McMaster-Carr -$38.29
4x 10-32 spring plunger from McMaster-Carr – $3.25 x 4 = $13
2x 780 nm infrared bandpass filter from ebay – $15 x 2 =$30
2x “NIR Optical Filter, 796AF35, 25mm diameter ring” from ebay -$50 (email: firstname.lastname@example.org, and ask if he has them and he will post them in his store)
Computer – Office computer (supplied our own)
Wire, heatshrink tubing, epoxy, board connectors, bolts, angle iron, 3V wall brick – found around the office
Total (w/o computer)= $2308.90
Remove the IR filters from the cameras
The cameras have IR-blocking filters. These need to be removed so the cameras can see the IR light from the lasers.
There are two different types of PS3 Eye lenses. We used the older type which the video and walk-through both refer to. We destroyed a new lens learning this. We recommend going to a brick-and-mortar store and looking at the cameras to make sure you get the good older version. The older lens is almost all black and curved, the newer one is flat and has white rings around it.
Remove the back panel and assemblies
In order to connect the power supply to the TV, you will need to take off the back panel. There are 10 screws on the back bottom panel, remove them and the plastic cover. Inside there are two main assemblies that come out really nicely- all in one piece.
The left one (facing the back of the TV) is the one to mount the power supply board to. The black plastic panel in front of this assembly will simply snap out. There are only two screws to remove to get the assembly out: one on the bottom left side, one on the bottom right side. There are five cables to remove. First, disconnect the ribbon cable from the right assembly. There are three more cables to remove on the right, and one on the left. The left assembly should now slide easily to the left side and out.
The right assembly is the one with the projector on it. There are only two screws on the bottom of the assembly to remove. There are three cables to disconnect. One ribbon cable that connects to the left assembly (you probably already disconnected this), one cable on the left side, and one on the right side. The assembly should now slide back and out.
Make a power supply for the lasers
We found a 5V supply on the power supply board of the TV. This supply turns on with the TV, so there is no need to include an extra switch for the lasers. We need to bring the voltage down to the operating voltage of the lasers, so we added our own step-down power supply. We put the LM317T IC on the Radio Shack perfboard, and wired up two resistors according to this schematic found in the datasheet. The resistor calculation basically follows the formula: V = 1.25(1 + R2/R1). We chose R1 as 56K and R2 as 75K. Using the basic equation, it should be about 2.9V, but when tested it was around 3.2V. In the future, we will replace R2 with the 200K trimpot we specified in the materials section so we can turn the lasers down as low as possible without sacrificing performance.
A picture of the finished board. We added some capacitors on the input and output for good measure.
A photo of the power supply on the Samsung TV. The circle on the left is ground, the circle on the right is +5V.
The two boards together. There are some unused threaded holes in the correct locations to mount your Radio Shack board to!
Remove the screen
There are twelve screws surrounding the sides of the screen, remove those.
There are two electrical connectors that go to the front panel. You can find those on the left assembly (looking at the back of the TV). Remove those connectors. The connectors are a real pain to get off because they have clips that require you to squeeze the clips and pull them off. Once these were off we pried the clips open so we didn’t have to go through this every time we removed the back panel.
There are two more screws that should be removed at the bottom from inside the bottom cavity. Remove the two assemblies to clear out the cavity. The two screws at the very top are screwed into the screen and can be recognized as they are in channels.
The screen can now be pulled off the TV (there are four snaps along the bottom that hold it on – you will have to pull it off of those first).
Install the cameras
The two cameras capture the left and right sides of the screen. They have to be mounted no closer to the projector lens than they are in the photos. If they are too close, the top and bottom of the screen will not be captured. In order to test the camera mounting location, we marked the two areas of the screen that need to be captured with red tape
We removed the housings from the PS3 Eye cameras as indicated in AlexP’s post. This should already be done if you have removed the IR filter. We made our own mounting blocks with small bolts and nuts through the board holes. This is a good idea because it will give you precise control if you need to do minute alignments later. Here is one of the mounting boards:
We then cut a hole in the plastic to recess the camera, and bolted a piece of angle iron to the side of the hole. We recommend bolting this for the sake of tuning minor angles and displacements.
To line up the camera properly, install and open macam-for-multi to view the image from the camera. When you can see the entire rectangle of tape, make note of the angle that the mounting board meets the angle iron. Now unbolt the mounting board and grind it down to approximately that angle. Bolt the mounting board back on.
Connect the camera to the computer again, open macam, and get some five-minute epoxy mixed up and ready. Slather some on the place where the mounting board and the angle iron meet. Line the camera up to capture the entire rectangle again. Hold the camera steady and be patient for five minutes while holding it, your arms will probably ache from holding it steady.
Repeat for the other camera.
We taped Infrared filters to the cameras that we got from Edmund Scientific. These are not the best in sunlight, and we are still looking for a good 780 nm band-pass filter that is big enough. These 25mm filters from Knight Optical were too small and cut out the light around the edges of the image. We are waiting for some new ones from eBay and will let you know how they work.
Install the lasers
This step was the most difficult of all. We wanted to make sure we had fine control of the laser alignment. To do this we machined parts to mount the lasers to the frame. You could epoxy these too, but it’s recommended to mount them in a way that allows precise alignment.
We made a quick and dirty prototype of the mounting system. Imagine the black cylinder is the laser and is attached to the bottom plate. By adjusting the screws, you can adjust the height, tilt, yaw, and roll of the module.
We measured the metal braces on the frame and designed a two-part system – the base and the carrier.
The base screws on-to the metal supports that frame and secure the screen.
The carrier holds the laser module and rests on top of the base.
The base has four screws that can be used for fine-tuning the position of the module. The carrier has a spring plunger threaded in, to press the carrier out from the base. Here are some screen captures of the CAD files.
Once we machined the parts, we had to modify them, the metal supports, the bezel, and the corners of the screen to get everything to fit and be secure:
We have some dimensioned PDFs and the CAD files in Solidworks and IGS available on our Google code page.
The PDFs look like this:
Wiring up the lasers
The next step is to wire up the laser modules. We used RadioShack 1/8″ plugs and jacks to connect power to the modules. Because the jacks are mounted to the back of the TV, this system allows us to quickly disconnect the modules in order to align the lasers and remove the screen.
We then ran the wires around and through the slot where the other wires from the screen meet the internal circuitry.
Focusing and testing the lasers
We found that we needed to make a test power supply for focusing the lasers, so we found a 3V power supply in a junk bin and wired up an extra 1/8″ jack to it.
CAUTION! These lasers are extremely dangerous! They are “near IR” so you will see some red light from them, but your eyes aren’t as sensitive to the light as light that is further inside the visible spectrum. You should NEVER look into them, and you should focus them by using a webcam to see the light. Always wear the safety goggles when working with these lasers.
The lasers have lenses to focus them and line generators that screw onto the front. The lasers probably shipped with the line generators on backwards. The ridges are supposed to point outward. We used some Loctite on the lens so it stays focused.
Once you have focused the lasers you can put them in their carriers and use the adjustment screws on the carriers to align them properly. Again, wearing the glasses and using an extra webcam to see the light is the safest way to do this.
Adding ABS side sheilds
You should cut the black ABS plastic sheet into four strips and glue them to the sides of the frame so they extend about an inch from the front of the screen. These will block the laser light from going past the side of the screen and will protect people who might be looking at the TV from the side. Put the textured edge toward the inside of the screen.
Pros and Cons
- Excellent tracking results, even with lots of ambient IR light.
- Zero-force: Feels very natural to work with.
- Comparatively easy to build.
- Comparatively cheap to build.
- Comparatively fast to build.
- Already “sees” the finger slightly before it touches the surface (since the laser light plane extends to about a millimeter above the surface).
- It is very sensitive.
- Availability of infrared lasers*
- Lasers are dangerous for the eyes.
- Potential problems with occlusion.
*As long as you are careful with the lasers, your eyes will be safe: Just accidentally looking into the laser for a second won’t do any harm, it’s just that you shouldn’t stare into them for long. However, since the infrared lasers also emit a bit of (red) light in the visible spectrum, you’ll notice immediately when you’re looking into a laser by accident, so you can turn your head away. It’s really nothing to worry about too much, but it’s most definitely not an ideal setup if you want to work with children, for example!
We used multiple sites to gather information about this system. Two of the most notable sources of information were:
and all of the great information at NUI Group