Kimchi and Chips' blog

Kinect + Projector experiments

Using Padé projection mapping to calibrate Kinect’s 3D world with a projector.

1. Using the kinect camera, we can scan a 3D scene in realtime.
2. Using a video projector, we can project onto a 3D scene in realtime.

Combining these, we re-project images onto geometry to create a new technique for augmented reality

Previous videos (for process)

The pipeline is:

1. Capture Depth at CameraXY (OpenNI)
2. Convert to image of WorldXYZ
3. Pade transformation to create WorldXYZ map in ProjectorXY
4. Calculate NormalXYZ  map in ProjectorXY
5. Guassian Blur X of NormalXYZ in ProjectorXY
6. Guassian Blur Y of NormalXYZ in ProjectorXY
7. Light calculations on NormalXYZ, WorldXYZ maps in ProjectorXY

Tags: MapTools

This entry was posted on Wednesday, January 12th, 2011 at 10:30 pm and is filed under Idea, Projection, Research, Wave of Kimchi and Chips. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

12 Responses to “Kinect + Projector experiments”

1. Kinect-Augmented Reality, as Projection Mapping Meets Depth Sensing (Hint: It’s Awesome) Says:
   January 14th, 2011 at 5:55 pm

   blogpost on Create Digital Motion

2. Kimchi and Chips blog » Blog Archive » Kinect + Projector experiments – media and arts technology Says:
   January 14th, 2011 at 7:03 pm

   blogpost on qmat

3. kinectとプロジェクターで波動拳 | Tom-Style Says:
   January 17th, 2011 at 4:04 pm

   [...] kimchiandchips via [...]

4. Augmented Reality mit Kinect und Projektor: Hadouken! - Kinect Says:
   January 18th, 2011 at 7:51 pm

   [...] Kimchi and Chips zeigen uns, was mit eine Kombination aus Kinect und einem Projektor möglich ist: Elliot – offensichtlich genannt “Chips” – verwandelt seinen Arm in eine virtuelle Fackel und zeigt dann einige beeinruckende Licht- und Schattenspielereien. Grundlage des System ist ein Mapping des vom Projektor erzeugten Bildes auf die von Kinect erfassten Raumkoordinaten. Der Projektor erzeugt dann eine virtuelle Lichtquelle an der Stelle im Raum, wo sich die Hand des Nutzers befindet. Erkennt Kinect nun, dass sich andere Oberflächen im nur – theoretisch vorhandenen Lichtschein – der virtuellen Fackel befinden, so wird die entsprechende Oberfläche beleuchtet. Der Projektor vergrößert also den beleuchteten Bereich um die getrackte Hand herum, aber nur dort, wo sich wirklich eine Oberfläche befindet. Elliot demonstriert dies an der Wand, der Decke und an seinem eigenen Körper, was eigentlich interessanter ist, da dieser keine simple Form aufweist. [...]

5. Luis Rodriguez Says:
   January 18th, 2011 at 11:06 pm

   muy bueno

6. The Lemonary » Nearness Says:
   February 20th, 2011 at 12:57 am

   [...] points there. The NITE skeleton tracking component is available for OSX, as is the OSCeleton code. Kimchi and Chips have videos showing the promise of motion tracking + projection [...]

7. State of the art I – Object recognition with Kinect | mapinect Says:
   April 26th, 2011 at 2:41 pm

   [...] Street Fighter fireball effect with virtual dynamic light source. For more information go to the Kimchi and Chips’ blog [...]

8. Kinect hacks get all surgical and hadouken | VG247 Says:
   May 19th, 2011 at 2:15 pm

   [...] Elliot’s videos through here in case the description above confused you a [...]

9. Kinect + Projector experiments « Arch Comp Project Says:
   August 6th, 2011 at 12:15 am

   [...] Online references: http://www.kimchiandchips.com/blog/?p=544 [...]

10. B.K. Says:
    December 28th, 2011 at 11:27 am

    Hey, is the sourcecode for this available?
    If not, did you do this for the official sdk from microsoft?

11. elliot Says:
    December 28th, 2011 at 2:15 pm

    @BK – check http://www.kimchiandchips.com/blog/?p=725 for source / examples / text / explanation video

    kinect interfacing was performed with OpenNI

12. Mauricio(México) Says:
    April 15th, 2012 at 12:12 am

    amazing

Leave a Reply