fix(ci): moving checkouts to a shared volume made them leak forever #5605

Seven orphaned checkouts, 1.5 GB, in 75 minutes — on a 150 GB disk shared with five other tenants and already 64% full.

Checkouts used to be mkdtemp'd inside the app container, so a leftover directory vanished with the next container restart (roughly hourly here). Moving them to the shared ci-work volume this morning, so a sibling runner could see them, made them persistent — turning a self-clearing mess into an unbounded one. Each is a full clone plus node_modules, ~214 MB.

A full disk on this box is an outage for every tenant on it — the same failure the broker's container cleanup exists to prevent, arriving by a different route.

Diagnosis, because the obvious suspect was innocent

The in-run cleanup looked wrong: rm(join(checkoutDir, "..")), a path ending in ... I tested it rather than assuming — join() normalises it and the delete works fine. The app logs showed no cleanup errors either, though that proved nothing: the container had been recreated minutes earlier, so only minutes of logs existed.

The real cause is that the cleanup line is never reached. The routine victim is a run killed by the very deploy that recreates the container, and no finally survives SIGKILL. Today's cadence — seven merges in an afternoon, each deploy killing an in-flight run — accounts for all seven orphans exactly.

The fix

A sweeper on the same schedule as reapStuckRuns, removing gluecron-run-* directories older than two hours — against reapStuckRuns marking anything still running after one hour as abandoned, so the extra hour is pure margin. Deleting a live job's working directory would be a far worse bug than keeping a dead one an hour longer.

Steady-state bound is therefore (runs per two hours) × ~214 MB, rather than unbounded.

Two guards, both mutation-verified by deleting them and watching specific tests fail:

• Only directories this code created (gluecron-run- prefix). The volume is shared; removing a directory we didn't create isn't cleanup, it's data loss in someone else's service.
• Only ones past the age limit, with the boundary case kept rather than deleted.

Tests run against a real temp directory rather than a mocked fs, because what's being asserted is what happens to files.

Also corrects a comment

The in-run cleanup was labelled // Cleanup always runs. It isn't in a finally, and SIGKILL can't be caught regardless — a claim the code didn't support, sitting directly above the line whose failure to run produced the leak.