---
title: "KABE: an interactive climbing wall"
description: "An interactive climbing wall: a camera finds the climbers with a pose-detection model, and a projector turns the wall into a game written in Lua. This page follows one frame through the code."
date: 2025-12-30T00:20:26+01:00
lastmod: 2026-11-03T00:42:26+01:00
url: https://klimek.link/projects/kabe/
author: "Marcin Klimek"
source: https://klimek.link/projects/kabe/
---

# KABE: an interactive climbing wall

**KABE** is the successor of [Cinder Climb](/projects/cc/), my environment for interactive games on a climbing wall. A camera watches the wall, a neural network finds people in the picture and a projector draws a game around them. Games are Lua scripts, so to change one I edit a text file and press the space bar.


I'll follow one camera frame through the program, from raw pixels to the shape that the projector draws and Lua scripts can touch. Everything here comes from the source, and where my code does something odd, I say so.

## What came before

The first Cinder Climb tests were recorded on 11 March 2016 with a Kinect 2 depth camera. That version clips depth images to a near and a far distance, stretches contrast, blurs, erodes and dilates, and hands the result to the OpenCV MOG2 background subtractor with shadow detection on. A contour finder with a minimum and maximum area turns what's left into blobs. Each outline gets smoothed and simplified, goes through a homography into projector coordinates, and that's what Lua sees.

My Cinder Climb box game walks over the vertices of every blob. For each vertex it divides by the tile size and marks the tile under it as touched, on a grid of 32 by 16 tiles. Touched tiles fade out, wait 30 frames and fade back in, so every tile is a tiny state machine. Tiles come back in a new colour every 120 frames.

Everything there hangs on the model of the empty wall. The code has a reset-background switch and a timer for rebuilding it (`Analysis.cpp`), and the last commit in the old repository is titled "searching for a better way to segment foreground/background". The Kinect also ties that version to one particular sensor.

KABE asks where the people are, and a pose model answers that for any ordinary camera.

A good part of the surrounding code carried over. Both repositories have a two-window layout, a sensing window with four markers and two region-of-interest points, a homography helper and an XML file with the calibration. Lua is different. Cinder Climb has a small binding layer of its own, and KABE uses the `ofxLua` addon.

## Two windows and one thread

`main.cpp` creates two windows. The projector window is 1024 by 768, not resizable, with vertical sync on, and it sits at x = 1920 so that it lands on a second display. The control window is 1200 by 1000 with vsync off, placed right next to it, and it shares an OpenGL context with the projector window. Both positions are written into `main.cpp` and assume a primary display of 1920 pixels. I haven't made them configurable.

Each window is an `ofApp` (`ofAppGui` and `ofAppProjector`), and both get a pointer to one `Machine` object. `Machine` owns the camera, the ONNX model, a `Processor` that turns detections into shapes, and the Lua state. `ofAppGui::update` grabs a camera frame and calls the script's `update`, and `ofAppProjector::draw` calls the script's `draw`.

My architecture notes describe a GUI thread, a projector thread and a process thread. As far as I can tell from `main.cpp`, both windows run in one openFrameworks main loop, one after another, so the only thread I start myself is the inference loop in `Machine::run`, a `std::jthread`. Three roles describe it better.

*Diagram: KABE: from camera frame to a silhouette in projector space, and the three roles*

I ask the camera for 960 by 540, half of the 1920 by 1080 in the constants file. Frames go into two buffers. `capture()` writes into whichever buffer isn't marked readable, bumps an atomic frame counter and flips the readable index. The inference loop polls that counter every millisecond. When it sees a new number, it counts skipped frames, copies the readable buffer and gets to work. The skipped-frame counter exists, and nothing displays it yet.

Before double buffering there was a mutex around a single frame, and my review notes flagged it as a source of contention between capture and inference. The preview in the control window reads from the same two buffers and never waits for the model.

## From a frame to keypoints

The worker crops each frame to the region of interest, the rectangle I drag in the preview. Cropped pixels are wrapped in an OpenCV matrix without copying, and `AutoBackendOnnx::predict_once` takes over.

The matrix goes from RGB to BGR in place and gets resized to 640 by 640 with linear interpolation. That's a plain stretch with no letterbox padding, so a tall or wide region is squeezed before the model sees it, and keypoints are scaled back later with separate factors for x and y. Regions that aren't square distort people a little on the way in. Nothing in the repository measures what that costs in accuracy.

Then I make a float tensor. Divide by 255, split the interleaved image into three planes (HWC to CHW) and write them straight into a buffer that lives in the class. This used to be a `new float[]` on every frame, and the optimisation notes estimate that reusing the buffer, together with the other changes below, cut the peak allocation rate by about 97 percent.

The model is YOLOv8m Pose, exported to ONNX and run with ONNX Runtime. The session turns on the memory arena, the memory pattern optimisation and parallel execution. My code asks for the CUDA provider, after checking which providers the runtime reports. If CUDA is missing or appending it throws, the program prints a message and carries on with the CPU, so a machine without CUDA works with no changes, only slower. The CUDA request is a constant in the constructor of `Machine`, and so is the model path, `data/checkpoints/yolov8m-pose.onnx`. When that file is missing, the program prints a message and exits.

The wrapper class reads image size, stride and class names from model metadata, although `predict_once` has 640 by 640 written in.

Post-processing is the usual YOLO routine. The output tensor is transposed so that each row is one prediction, with four box numbers, class scores and 51 keypoint values (17 points with x, y and a confidence each). For every row the code takes the best class score and drops the row at 0.45 or lower. `cv::dnn::NMSBoxes` removes overlapping boxes at an IoU threshold of 0.45. Whatever survives gets scaled from 640 by 640 back to the size of the cropped region and clipped to its edges.

## From keypoints to a body

A list of 17 COCO keypoints isn't something a game can collide with, so every person becomes a shape. The skeleton is a table of 19 pairs of keypoint indices (shoulder to elbow, hip to knee and so on), and a pair becomes a bone when both ends have a confidence of at least 0.5.

Each bone turns into a thick quad. I take the direction between two points, rotate it by 90 degrees to get a normal and offset both ends by half the width, which is 20 pixels in camera space. All quads of one person go to Clipper2 for a union with the non-zero fill rule, and the result is inflated by 5 with round joins and simplified at a tolerance of 1. What comes out is a smooth polygon that looks like a fat stick figure.


The code keeps only the first path of the Clipper2 result. If a missing keypoint splits a person into two separate pieces, the second piece is dropped. That happens in `Shape::mergePolylines`.

Every vertex of the polygon is divided by the size of the cropped region and sent through the homography, which gives coordinates between 0 and 1 in projector space. Bones get the same treatment one by one, and the projector window draws them as lines after the script's `draw` has finished. There's no switch for that overlay yet. A collision test looks at the bounding box of the transformed polygon first and runs a point-in-polygon test second.

### Smoothing

Raw detections jitter. A wrist can jump a few pixels between two frames for no reason, and a keypoint can fall under the confidence threshold for a single frame. `Processor` blends each keypoint with the same keypoint from the previous frame.

```cpp
score    = (conf_p * conf_p + conf * conf) / (conf_p + conf);
factor   = conf_p + score;
result_x = (conf_p * x_p + x * score) / factor;
result_y = (conf_p * y_p + y * score) / factor;
```

`conf_p` and `x_p` come from the previous frame, and with equal confidences the result is a plain average of both frames. The same formula sits in a comment above `Processor::process` as a short Python function. Two details are easy to miss. The previous frame is the raw detection, not the smoothed one, so this blends two frames and isn't a running filter. And the confidence that gets stored is the previous one, which means a point that passed the 0.5 check a frame ago passes it again.

A consequence of the formula is that a new detection with very low confidence still gets a vote almost as big as the previous one, because the score moves toward the previous confidence. With conf_p at 0.9 and conf at 0.1, weights come out at 0.9 and 0.82.

Frames are paired by position in the list, so person number two now is blended with person number two a frame ago. There's no tracking. When two people swap places in the list, their keypoints get mixed for a frame.

When the model finds nobody, old shapes stay for one second (`SHAPE_TIMEOUT_MS` is 1000) and then disappear. That's why silhouettes don't flicker when the model misses a frame or two.

## The homography and the calibration

Camera and projector don't look at the wall from the same place, so mapping between them takes a homography, which needs four points in one image and the four matching points in the other. My solver is a small piece of code adapted from an openFrameworks forum post, and the header says so. It builds an 8 by 8 linear system from the four point pairs, following a multiple-view-geometry textbook, solves it with Gaussian elimination and returns a 4 by 4 matrix. To apply it, I multiply a point by that matrix and divide by `w`.

In the calibration, the four markers in the preview are corners of the area that the projector covers, as the camera sees them. They get normalised against the region of interest and mapped onto the unit square, so the homography goes from camera space inside the region to projector space.

*Diagram: Schematic of the calibration in the control window preview, not a screenshot*


The whole state is four markers and two region-of-interest points, saved as `marker-x-0`, `marker-y-0` and so on, plus `roi-x-0` and `roi-y-0`, into `cc-settings.xml` after every change. When the file is loaded, the program checks that it isn't garbage. If two or more points came back as (0, 0), or two markers are identical, or two region-of-interest points are, everything resets to defaults.

The mouse works on the preview, and the area that accepts it is the preview rectangle scaled by 1.1, so a marker near the edge can still be grabbed. To check the calibration without climbers, the projector window draws a dot at the mouse position, pushed through the same homography. A click in the preview also reaches the script, converted to a click at the matching projector pixel.


The second review round found one slip here. In `mouseReleased`, the y coordinate was divided by the height of the preview and x wasn't. The current code has both without division.

## Lua

Scripts live in a `scripts` folder. The list is built when the program starts and holds `start.lua` plus every file that begins with `game_`, sorted by path. The check for that prefix is written with a backslash, so this part only works on Windows. As far as I can tell, `start.lua` ends up first because its forward slash sorts before the backslash. The list becomes a slider in the control panel, and the script reloads when the slider changes, which the arrow keys do as well.

Reloading goes through `ofAppGui::reloadScript`, and order matters. The old script gets its `exit` call, then a fresh Lua state is created, so nothing carries over from one game to the next. The search path is extended with `data/scripts`, which lets scripts `require` library modules. After that, `LuaMachine` is registered with a constructor and four members (`collide`, two screen-size getters and `get_person_count`), and an object called `machine` becomes a global. A smoke call, `machine:collide(0,0,1)`, runs right away to make sure the binding works. Then the script itself runs, followed by its `setup`.

An earlier version of that smoke call had a typo, `machine::collide`, which isn't Lua. The second review caught it.

For a game, the main question is whether a point on the wall touches a person. `machine:collide(x, y, radius)` answers for one point, with `x` and `y` in projector pixels (0 to 1024 and 0 to 768). It takes a copy of the current list of shapes under a shared lock and asks each shape in turn.

The second function is `collideAll(points, radius)`. It takes a Lua table of `{x=, y=}` points and returns a table of booleans.

```lua
local hits = collideAll({ {x = 10, y = 20}, {x = 30, y = 40} }, 30)
-- hits is { true, false } when only the first point is inside a person
```

It's written against the raw Lua C API. `registerCollideAll` pushes a pointer to `LuaMachine` as light userdata and wraps it in a C closure, so the function finds its object as an upvalue. A call copies the list of shapes once, loops over the table with `lua_rawgeti` and `lua_getfield`, and gives `false` for any entry that isn't a table. For a game with 32 by 16 tiles, one `collideAll` call replaces 512 separate `collide` calls and 512 trips through the binding layer. Nothing in the repository measures the difference.

The `radius` argument has a catch. It's passed down to `Shape::collide` and never used there, because the test is the bounding box and then a point-in-polygon check on the centre point. Silhouettes are already inflated by 5 pixels in camera space, so a point close to a body counts, but a circle that only overlaps the edge of a body doesn't. Cinder Climb behaved differently. Its `point_in_blobs` did use a distance, and a point counted when it was inside a blob or within that distance of the closest point on its outline.

Lua's garbage collector runs one incremental step every 30 frames in `ofAppGui::update`. It used to run a full collection every frame, and the second review counted that as a noticeable per-frame cost, as an estimate.

Cinder Climb scripts use coordinates up to 10.24 by 7.68, and KABE uses projector pixels. A script quoted in the review notes has `deltaX = 1024/numX`, so the box game moved to pixels along the way.

My notes also list a few Lua base classes (an object with position and fade, a touchable shape with a state machine, a collision manager that batches queries, a Box2D wrapper and sprite animation) and two starter games, a box game and a points game. They aren't in the repository. The `bin` folder isn't tracked, so neither scripts nor the model file are there.


## What changed after the first working version

The repository holds two code review notes and a list of optimisations applied after them. Together they describe what changed once the first version worked.

I reused the ONNX input buffer, so a frame costs no allocation there. Preview textures used to be created on every draw call, and now one texture is created once and reallocated only when the camera size changes. The FPS counter caches its text and rebuilds the string only when the number changes. Shape code reserves its vectors, and `Line` and `Shape` have move constructors. `Processor` holds a `std::shared_mutex`, so drawing and Lua calls can read shapes at the same time and only the writer locks exclusively. The previous frame sits in a `std::optional`, `process` takes a `std::span`, and magic numbers moved into a constants file.

The notes put the overall gain at 45 to 70 percent in frames per second, and they say plainly that it's an estimate. The architecture notes expect about 30 frames per second of inference with CUDA and about 10 on a CPU, with drawing at 60 and vsync. These are expectations from the notes, not measurements.

Some leftovers remain. `draw_frame_from_roi` warps the quad between the four markers into a 640 by 640 picture and draws it next to the preview, and it's commented out where it's called. `set_fbo_texture` is empty. The constructor has a path to a test video next to the camera index, and `Line::collide` has a block of projection code after a `return false` that never runs.

## What it needs and where it stops

KABE runs on Windows 10 or 11, 64 bit. It builds with Visual Studio 2022 (toolset v143) and C++20, and glm and zlib come from vcpkg in manifest mode. OpenCV arrives with the openFrameworks addons, ONNX Runtime with CUDA support is bundled in the `external` folder, and Clipper2 sits in the source tree. The repository carries a copy of openFrameworks and the addons `ofxGui`, `ofxGuiExtended` and `ofxLua`. The project notes list one common build error, a project file asking for the v145 toolset on a machine that has only v143.

It supports one camera (the first device) and one projector, and the projector size is fixed at 1024 by 768. The source is in a private repository.

## Planned activities

Some of the planned activities are meant for people with disabilities: putting out points along a vertical line, one-sided and alternating, putting out segments of a picture around the whole wall, sorting elements by category, and following the hands of a clock. The others are games for any climber: escaping water that rises from below or fire from above, erasing a picture with your body, catching eggs (the higher, the more points), smearing paint with the whole body for children who take their first steps on holds, matching a shape shown on the wall and holding the pose, and a "jumping frog" that hops from hold to hold.

Source: https://klimek.link/projects/kabe/