How it works

#

A Kinect 2 depth camera looks at the wall. The program keeps a model of the empty wall and subtracts it from each depth frame. What is left is filtered, and each remaining blob becomes the outline of a climber.

A calibration window maps the camera image to the projector. You drag four markers and two region-of-interest points, and the program saves them. A Lua script draws the activity with the projector and asks the program whether a point on the wall touches a climber.

Activities

#

Activities are Lua scripts. You switch between them with the arrow keys, and the space bar reloads the current one without restarting the program. The scripts use the openFrameworks API, Box2D physics and the Kinect data: the number of blobs, the outline of each and a test whether a point is inside one.

• Coins. Balls fall down the wall under Box2D gravity. Touch a ball with your body and you score. Balls that reach the floor are misses. A round lasts 60 seconds, and gravity grows with the level.
• Position. A picture is shown and you must fill its shape with your body. The target is a 60 percent match.
• Totoro. Your silhouette is a mask that reveals a picture chosen at random from a set.
• Trail. The silhouette is painted on the wall and fades slowly, so it leaves a trail.
• Box. The wall is a grid of 32 by 16 tiles. Tiles light up when you touch them, and the colour changes at intervals.

Limits

#

The project targets Windows and the Kinect 2. It relies on a model of the empty wall, so it needs a background reset when the scene changes. Game texts in the scripts are in Polish.

What came next

#

The project continues as KABE, which replaces the Kinect with an ordinary camera and a pose-detection neural network.