KABE: an interactive climbing wall

KABE succeeds Cinder Climb, my environment for interactive games on a climbing wall, or on any wall. The first Cinder Climb tests were recorded on 11 March 2016 and used a Kinect 2 depth camera. KABE drops the Kinect. It works with an ordinary camera and finds people with a pose-detection neural network.

How it works

#

1. A camera watches the wall. Each frame is cropped to a region of interest and passed through YOLOv8 Pose, run with ONNX Runtime on CUDA when available, otherwise on the CPU.
2. The detected body keypoints are smoothed over time, weighted by the model’s confidence, and joined into 19 “bones”.
3. The bones are drawn as thick lines and merged into one body silhouette with Clipper2.
4. A four-point homography maps the silhouette from camera space to projector space, so what the projector draws lines up with the real climber.
5. The activity is a Lua script. It can call the openFrameworks drawing functions and collide(x, y, radius), which says whether a point on the wall touches a person.

What it can do

#

• Several climbers at once. Every detected person gets their own silhouette, and a script can ask how many people the camera sees.
• Touch detection for any shape. A script asks whether a circle at a point on the wall overlaps a body. It can ask about one point or a whole list of points in a single call.
• Stable tracking. Keypoints are blended with the previous frame and low-confidence points are ignored. A silhouette disappears only after one second without a detection, so a single missed frame does not make it flicker.
• Any wall, any camera. An ordinary USB webcam is enough. There is no depth sensor and no special markers.
• Live script changes. You edit a Lua script, press the space bar or r, and the activity restarts without restarting the program.

Setup and calibration

#

You need a PC, a camera that sees the wall and a projector. The program opens two windows. One goes to the projector. The other is the control window with the camera preview, the silhouettes and two FPS counters, one for inference and one for drawing. A slider there selects the activity.

Calibration is done with the mouse in the preview. You drag four markers onto the corners of the area the projector covers, which defines the mapping between camera and projector. Then you drag two points to crop the part of the image that goes to the model. A smaller region means less work for the model. Both are saved, so the next start needs no calibration. A red dot in the projector window follows the mouse through the mapping, so you can check that it lines up with the wall.

Writing an activity

#

Each activity is a Lua file. Files named game_*.lua are picked up automatically, start.lua is always first, and the arrow keys switch between them.

A script defines setup, update, draw and exit, and can add key and mouse callbacks. A click in the camera preview reaches the script as a click at the matching projector position. That is handy for testing without a climber. The script gets:

• machine:collide(x, y, radius), true when the point touches a person;
• collideAll(points, radius), which takes a table of points and returns a table of booleans, cheaper than many single calls;
• the person count and the screen size;
• the whole openFrameworks API, so images, shapes, fonts and sound are all available.

The project notes also list a few Lua base classes: objects with position and fade, touchable shapes, a collision manager, a Box2D wrapper and sprite animation. Two starter games are named there, a box game and a points game.

Performance

#

The architecture notes expect about 30 frames per second of inference on a CUDA GPU and about 10 on a CPU. Drawing runs at 60 frames per second. These are expectations from the notes, with no measurements behind them. The notes also estimate the gain from the tuning at 45 to 70 percent.

Hardware and limits

#

• Windows 10 or 11 (64 bit), built with Visual Studio 2022 and C++20.
• An NVIDIA GPU with CUDA is recommended. A CPU run needs a code change and a rebuild.
• One camera and one projector. The projector size is fixed at 1024 by 768.
• The model path is fixed in the code.
• The source is in a private repository.

Planned activities

#

Some are meant for people with disabilities: putting out points along a vertical line, one-sided and alternating, putting out segments of a picture around the whole wall, sorting elements by category, and following the hands of a clock.

Others are games for any climber: escaping water that rises from below or fire from above, erasing a picture with your body, catching eggs (the higher, the more points), smearing paint with the whole body for children who take their first steps on holds, matching a shape shown on the wall and holding the pose, and a “jumping frog” that hops from hold to hold.