Climbing Beta

You give it a photo of a wall. A fine-tuned YOLO11s-seg model finds the holds, colour clustering in CIELAB works out which ones belong to the same route, and a shortest-path search over four-limb stances returns a sequence of moves. The first full one was 14 moves up an 8-hold green route, picked out of 121 detected holds. On 106 held-out images containing 6,356 holds the detector reaches 0.877 mask mAP50.

There is also a phone app, live at climbing-beta-theta.vercel.app, that runs the detector in the browser, so the photo never leaves the phone. On a 24-megapixel wall photo it finds the same 152 holds as the Python pipeline, in 0.67 seconds.

Two modelling mistakes cost me the most time. The search kept stacking all four limbs on one hold, because coincident contact points drive the strain and balance costs to zero. Then I set the maximum hand-to-foot span to 0.9 times height and deadlocked 7 of the 10 routes I tested, when reaching overhead off a foothold is closer to 1.25 times your height.

Clustering the route was its own fight. Hold colours run in a continuum from orange through yellow to pink, so single linkage chained straight across the gaps and merged 43 of the 76 holds on one wall into a single route.

Then getting the detector onto a phone. PyTorch alone is 558 MB, over Vercel’s 500 MB limit, so I exported the model to ONNX and ran it in the page over WebGPU. Ultralytics’ own browser library crashed on this model, so I wrote the letterboxing and NMS myself.

I never got scale out of the photo by itself. Gym walls are drilled on a regular T-nut grid, so the bolt holes should work as a ruler, but I tried twice across three walls and neither method found them. So scale is an input: you mark two points a known distance apart, or four corners of the bolt grid, and the phone app measures a route that way.

The app can now save a tape-measure check alongside the photo and the taps, so next is collecting those for the first real error figure the geometry has had. After that, tuning the reach limits against routes a person has actually climbed. One of them currently rejects 91% of candidate moves and nothing has tested whether that is right.

The detector still picks up things that are not holds. A calibrated wall plane now removes the ones it can rule out, but a brush on the mat in front of a vertical wall still gets through.