KABE is the successor of Cinder Climb, my environment for interactive games on a climbing wall. A camera watches the wall, a neural network finds people in the picture and a projector draws a game around them. Games are Lua scripts, so to change one I edit a text file and press the space bar.
I’ll follow one camera frame through the program, from raw pixels to the shape that the projector draws and Lua scripts can touch. Everything here comes from the source, and where my code does something odd, I say so.
What came before#
The first Cinder Climb tests were recorded on 11 March 2016 with a Kinect 2 depth camera. That version clips depth images to a near and a far distance, stretches contrast, blurs, erodes and dilates, and hands the result to the OpenCV MOG2 background subtractor with shadow detection on. A contour finder with a minimum and maximum area turns what’s left into blobs. Each outline gets smoothed and simplified, goes through a homography into projector coordinates, and that’s what Lua sees.
My Cinder Climb box game walks over the vertices of every blob. For each vertex it divides by the tile size and marks the tile under it as touched, on a grid of 32 by 16 tiles. Touched tiles fade out, wait 30 frames and fade back in, so every tile is a tiny state machine. Tiles come back in a new colour every 120 frames.
Everything there hangs on the model of the empty wall. The code has a reset-background switch and a timer for rebuilding it (Analysis.cpp), and the last commit in the old repository is titled “searching for a better way to segment foreground/background”. The Kinect also ties that version to one particular sensor.
KABE asks where the people are, and a pose model answers that for any ordinary camera.
A good part of the surrounding code carried over. Both repositories have a two-window layout, a sensing window with four markers and two region-of-interest points, a homography helper and an XML file with the calibration. Lua is different. Cinder Climb has a small binding layer of its own, and KABE uses the ofxLua addon.
Two windows and one thread#
main.cpp creates two windows. The projector window is 1024 by 768, not resizable, with vertical sync on, and it sits at x = 1920 so that it lands on a second display. The control window is 1200 by 1000 with vsync off, placed right next to it, and it shares an OpenGL context with the projector window. Both positions are written into main.cpp and assume a primary display of 1920 pixels. I haven’t made them configurable.
Each window is an ofApp (ofAppGui and ofAppProjector), and both get a pointer to one Machine object. Machine owns the camera, the ONNX model, a Processor that turns detections into shapes, and the Lua state. ofAppGui::update grabs a camera frame and calls the script’s update, and ofAppProjector::draw calls the script’s draw.
My architecture notes describe a GUI thread, a projector thread and a process thread. As far as I can tell from main.cpp, both windows run in one openFrameworks main loop, one after another, so the only thread I start myself is the inference loop in Machine::run, a std::jthread. Three roles describe it better.
I ask the camera for 960 by 540, half of the 1920 by 1080 in the constants file. Frames go into two buffers. capture() writes into whichever buffer isn’t marked readable, bumps an atomic frame counter and flips the readable index. The inference loop polls that counter every millisecond. When it sees a new number, it counts skipped frames, copies the readable buffer and gets to work. The skipped-frame counter exists, and nothing displays it yet.
Before double buffering there was a mutex around a single frame, and my review notes flagged it as a source of contention between capture and inference. The preview in the control window reads from the same two buffers and never waits for the model.
From a frame to keypoints#
The worker crops each frame to the region of interest, the rectangle I drag in the preview. Cropped pixels are wrapped in an OpenCV matrix without copying, and AutoBackendOnnx::predict_once takes over.
The matrix goes from RGB to BGR in place and gets resized to 640 by 640 with linear interpolation. That’s a plain stretch with no letterbox padding, so a tall or wide region is squeezed before the model sees it, and keypoints are scaled back later with separate factors for x and y. Regions that aren’t square distort people a little on the way in. Nothing in the repository measures what that costs in accuracy.
Then I make a float tensor. Divide by 255, split the interleaved image into three planes (HWC to CHW) and write them straight into a buffer that lives in the class. This used to be a new float[] on every frame, and the optimisation notes estimate that reusing the buffer, together with the other changes below, cut the peak allocation rate by about 97 percent.
The model is YOLOv8m Pose, exported to ONNX and run with ONNX Runtime. The session turns on the memory arena, the memory pattern optimisation and parallel execution. My code asks for the CUDA provider, after checking which providers the runtime reports. If CUDA is missing or appending it throws, the program prints a message and carries on with the CPU, so a machine without CUDA works with no changes, only slower. The CUDA request is a constant in the constructor of Machine, and so is the model path, data/checkpoints/yolov8m-pose.onnx. When that file is missing, the program prints a message and exits.
The wrapper class reads image size, stride and class names from model metadata, although predict_once has 640 by 640 written in.
Post-processing is the usual YOLO routine. The output tensor is transposed so that each row is one prediction, with four box numbers, class scores and 51 keypoint values (17 points with x, y and a confidence each). For every row the code takes the best class score and drops the row at 0.45 or lower. cv::dnn::NMSBoxes removes overlapping boxes at an IoU threshold of 0.45. Whatever survives gets scaled from 640 by 640 back to the size of the cropped region and clipped to its edges.
From keypoints to a body#
A list of 17 COCO keypoints isn’t something a game can collide with, so every person becomes a shape. The skeleton is a table of 19 pairs of keypoint indices (shoulder to elbow, hip to knee and so on), and a pair becomes a bone when both ends have a confidence of at least 0.5.
Each bone turns into a thick quad. I take the direction between two points, rotate it by 90 degrees to get a normal and offset both ends by half the width, which is 20 pixels in camera space. All quads of one person go to Clipper2 for a union with the non-zero fill rule, and the result is inflated by 5 with round joins and simplified at a tolerance of 1. What comes out is a smooth polygon that looks like a fat stick figure.
The code keeps only the first path of the Clipper2 result. If a missing keypoint splits a person into two separate pieces, the second piece is dropped. That happens in Shape::mergePolylines.
Every vertex of the polygon is divided by the size of the cropped region and sent through the homography, which gives coordinates between 0 and 1 in projector space. Bones get the same treatment one by one, and the projector window draws them as lines after the script’s draw has finished. There’s no switch for that overlay yet. A collision test looks at the bounding box of the transformed polygon first and runs a point-in-polygon test second.
Smoothing#
Raw detections jitter. A wrist can jump a few pixels between two frames for no reason, and a keypoint can fall under the confidence threshold for a single frame. Processor blends each keypoint with the same keypoint from the previous frame.
score = (conf_p * conf_p + conf * conf) / (conf_p + conf);
factor = conf_p + score;
result_x = (conf_p * x_p + x * score) / factor;
result_y = (conf_p * y_p + y * score) / factor;conf_p and x_p come from the previous frame, and with equal confidences the result is a plain average of both frames. The same formula sits in a comment above Processor::process as a short Python function. Two details are easy to miss. The previous frame is the raw detection, not the smoothed one, so this blends two frames and isn’t a running filter. And the confidence that gets stored is the previous one, which means a point that passed the 0.5 check a frame ago passes it again.
A consequence of the formula is that a new detection with very low confidence still gets a vote almost as big as the previous one, because the score moves toward the previous confidence. With conf_p at 0.9 and conf at 0.1, weights come out at 0.9 and 0.82.
Frames are paired by position in the list, so person number two now is blended with person number two a frame ago. There’s no tracking. When two people swap places in the list, their keypoints get mixed for a frame.
When the model finds nobody, old shapes stay for one second (SHAPE_TIMEOUT_MS is 1000) and then disappear. That’s why silhouettes don’t flicker when the model misses a frame or two.
The homography and the calibration#
Camera and projector don’t look at the wall from the same place, so mapping between them takes a homography, which needs four points in one image and the four matching points in the other. My solver is a small piece of code adapted from an openFrameworks forum post, and the header says so. It builds an 8 by 8 linear system from the four point pairs, following a multiple-view-geometry textbook, solves it with Gaussian elimination and returns a 4 by 4 matrix. To apply it, I multiply a point by that matrix and divide by w.
In the calibration, the four markers in the preview are corners of the area that the projector covers, as the camera sees them. They get normalised against the region of interest and mapped onto the unit square, so the homography goes from camera space inside the region to projector space.
The whole state is four markers and two region-of-interest points, saved as marker-x-0, marker-y-0 and so on, plus roi-x-0 and roi-y-0, into cc-settings.xml after every change. When the file is loaded, the program checks that it isn’t garbage. If two or more points came back as (0, 0), or two markers are identical, or two region-of-interest points are, everything resets to defaults.
The mouse works on the preview, and the area that accepts it is the preview rectangle scaled by 1.1, so a marker near the edge can still be grabbed. To check the calibration without climbers, the projector window draws a dot at the mouse position, pushed through the same homography. A click in the preview also reaches the script, converted to a click at the matching projector pixel.
The second review round found one slip here. In mouseReleased, the y coordinate was divided by the height of the preview and x wasn’t. The current code has both without division.
Lua#
Scripts live in a scripts folder. The list is built when the program starts and holds start.lua plus every file that begins with game_, sorted by path. The check for that prefix is written with a backslash, so this part only works on Windows. As far as I can tell, start.lua ends up first because its forward slash sorts before the backslash. The list becomes a slider in the control panel, and the script reloads when the slider changes, which the arrow keys do as well.
Reloading goes through ofAppGui::reloadScript, and order matters. The old script gets its exit call, then a fresh Lua state is created, so nothing carries over from one game to the next. The search path is extended with data/scripts, which lets scripts require library modules. After that, LuaMachine is registered with a constructor and four members (collide, two screen-size getters and get_person_count), and an object called machine becomes a global. A smoke call, machine:collide(0,0,1), runs right away to make sure the binding works. Then the script itself runs, followed by its setup.
An earlier version of that smoke call had a typo, machine::collide, which isn’t Lua. The second review caught it.
For a game, the main question is whether a point on the wall touches a person. machine:collide(x, y, radius) answers for one point, with x and y in projector pixels (0 to 1024 and 0 to 768). It takes a copy of the current list of shapes under a shared lock and asks each shape in turn.
The second function is collideAll(points, radius). It takes a Lua table of {x=, y=} points and returns a table of booleans.
local hits = collideAll({ {x = 10, y = 20}, {x = 30, y = 40} }, 30)
-- hits is { true, false } when only the first point is inside a personIt’s written against the raw Lua C API. registerCollideAll pushes a pointer to LuaMachine as light userdata and wraps it in a C closure, so the function finds its object as an upvalue. A call copies the list of shapes once, loops over the table with lua_rawgeti and lua_getfield, and gives false for any entry that isn’t a table. For a game with 32 by 16 tiles, one collideAll call replaces 512 separate collide calls and 512 trips through the binding layer. Nothing in the repository measures the difference.
The radius argument has a catch. It’s passed down to Shape::collide and never used there, because the test is the bounding box and then a point-in-polygon check on the centre point. Silhouettes are already inflated by 5 pixels in camera space, so a point close to a body counts, but a circle that only overlaps the edge of a body doesn’t. Cinder Climb behaved differently. Its point_in_blobs did use a distance, and a point counted when it was inside a blob or within that distance of the closest point on its outline.
Lua’s garbage collector runs one incremental step every 30 frames in ofAppGui::update. It used to run a full collection every frame, and the second review counted that as a noticeable per-frame cost, as an estimate.
Cinder Climb scripts use coordinates up to 10.24 by 7.68, and KABE uses projector pixels. A script quoted in the review notes has deltaX = 1024/numX, so the box game moved to pixels along the way.
My notes also list a few Lua base classes (an object with position and fade, a touchable shape with a state machine, a collision manager that batches queries, a Box2D wrapper and sprite animation) and two starter games, a box game and a points game. They aren’t in the repository. The bin folder isn’t tracked, so neither scripts nor the model file are there.
What changed after the first working version#
The repository holds two code review notes and a list of optimisations applied after them. Together they describe what changed once the first version worked.
I reused the ONNX input buffer, so a frame costs no allocation there. Preview textures used to be created on every draw call, and now one texture is created once and reallocated only when the camera size changes. The FPS counter caches its text and rebuilds the string only when the number changes. Shape code reserves its vectors, and Line and Shape have move constructors. Processor holds a std::shared_mutex, so drawing and Lua calls can read shapes at the same time and only the writer locks exclusively. The previous frame sits in a std::optional, process takes a std::span, and magic numbers moved into a constants file.
The notes put the overall gain at 45 to 70 percent in frames per second, and they say plainly that it’s an estimate. The architecture notes expect about 30 frames per second of inference with CUDA and about 10 on a CPU, with drawing at 60 and vsync. These are expectations from the notes, not measurements.
Some leftovers remain. draw_frame_from_roi warps the quad between the four markers into a 640 by 640 picture and draws it next to the preview, and it’s commented out where it’s called. set_fbo_texture is empty. The constructor has a path to a test video next to the camera index, and Line::collide has a block of projection code after a return false that never runs.
What it needs and where it stops#
KABE runs on Windows 10 or 11, 64 bit. It builds with Visual Studio 2022 (toolset v143) and C++20, and glm and zlib come from vcpkg in manifest mode. OpenCV arrives with the openFrameworks addons, ONNX Runtime with CUDA support is bundled in the external folder, and Clipper2 sits in the source tree. The repository carries a copy of openFrameworks and the addons ofxGui, ofxGuiExtended and ofxLua. The project notes list one common build error, a project file asking for the v145 toolset on a machine that has only v143.
It supports one camera (the first device) and one projector, and the projector size is fixed at 1024 by 768. The source is in a private repository.
Planned activities#
Some of the planned activities are meant for people with disabilities: putting out points along a vertical line, one-sided and alternating, putting out segments of a picture around the whole wall, sorting elements by category, and following the hands of a clock. The others are games for any climber: escaping water that rises from below or fire from above, erasing a picture with your body, catching eggs (the higher, the more points), smearing paint with the whole body for children who take their first steps on holds, matching a shape shown on the wall and holding the pose, and a “jumping frog” that hops from hold to hold.