Kinect to vanish

This project was born out of experimentations with depth mapping and skeleton tracking with Kinect. It employs a few peculiarities and quirks of Kinect to create opportunities for augmenting a live human performance.

Note: Jump to the video after the technical description for a quick overview.

Detailed Technical description

On a technical level, this project utilizes a quirk in Microsoft Kinect to create manipulations in the depth image.

First we can capture a background to use throughout the session. Then we identify a person in front of the camera through depth image skeleton tracking. The RGB pixels corresponding to the user’s location on the screen are then overlaid on the background which was captured before. This way we can basically superimpose a live video feed of a person on a background, which can be a picture captured from the real space, as is the case now, or in principal any image.

Having a static background beneath a live person on the screen allows some interesting opportunities, we were looking at some for live performance. A simple one which we use in the video below is to have the user in front of the camera when sampling a new background. An image of the user is then part of the background, but the live video of the user can still be overlaid. We can create a trace of the user’s motion this way, for example, or we can have the user interact with his own freeze frame past self, as in the video.

The quirk which we explored in Kinect was based on its limitation of having a minimum range for the depth camera. It meant that skeleton tracking would not work on any of these object which is too close to the Kinect. The way we used it was to create occlusion in path of the user, so that part of his/her body could not be tracked, and correspondingly no RGB pixels were shown on the screen, effectively the part of the user so covered was invisible on the screen.

A second person using their hand close to the Kinect could manipulate how much of the other person’s body is visible. Even the user themselves could come close to the camera and in this process have their bodies gradually disappear, or even make portions of their body disappear using their own hand.