feat(ci): a container per job — jobs isolated from each other, not just the app #5602

The sibling runner isolated jobs from the app. It did not isolate jobs from each other — one long-lived runner means two runs share a filesystem and a process namespace, so one tenant's step can read another's checkout or wait around for it.

That is the last thing standing between this and multi-tenant CI that can honestly be sold. Each job now gets its own container: created, run, harvested, destroyed.

Why a broker rather than a socket in the app

Something has to reach the Docker API, and /var/run/docker.sock is equivalent to root on the host — whoever holds it can docker run -v /:/host --privileged. Arbitrary code execution inside the app container is a product feature here, so mounting the socket there would upgrade every container-scoped compromise into a host-scoped one.

So the broker holds the socket and the app holds an HTTP client. The distinction that makes that worth anything:

The broker takes no container configuration from the caller.

Not the image, mounts, limits, capabilities, user or network — every one is a constant in broker-agent.ts. The app sends a workspace name, a script, an environment and a timeout. A fully compromised app can only ask for jobs the broker was already willing to run.

An API that accepts a container spec is a docker socket with extra steps.

Three modes, no flag day

Each falls through to the next when unconfigured. Both remote modes also require the checkout to be on the shared volume, because dispatching a private temp path to a container that can't open it is worse than running locally. A dev machine needs no configuration at all.

One simplification over the original audit

AUDIT-CI-RUNNER-ISOLATION.md Stage 2 assumed the job would clone over loopback with a short-lived scoped token, and made that token type a prerequisite. It isn't needed. The app already clones into the shared volume, so the broker binds only that run's checkout directory. No credential in the job at all — strictly better than the plan, because a credential that doesn't exist can't leak.

Verification

The tests assert the boundary's shape and are mutation-verified — adding Image: req.image, adding Privileged: true, or mounting the socket into the app service each fails a specific test.

One mutation initially "passed" because a CRLF/\n mismatch made the replacement a silent no-op — the same trap that has bitten twice this session — so the mutation script now asserts it actually changed the file before drawing a conclusion.

What this still does NOT close

Which is why workflow-exec-policy.ts stays, with a header that now gives reasons you can check:

• Egress. Jobs use bridge networking because bun install needs a registry. A hostile step can still make outbound connections — it can simply read far less than before. Deny-by-default is the next increment.
• The default job image is the app's own image, resolved from the broker's container so it always exists and matches the deploy — but it carries this platform's source at /app. Fine for trusted accounts, wrong for strangers.
• Kernel escape, and disk quota on ci-work.

Those two, then that file goes and CI opens to everyone. That's the whole remaining list.

Enabling

Off by default — BROKER_URL and BROKER_TOKEN both empty, and a URL without a token counts as no broker. To roll back after enabling, clear BROKER_URL; execution falls to the sidecar with no code change.

One thing I could not reproduce: in one combined local run of six suites, 1 of 76 tests failed; three subsequent runs were clean and the failing name never printed. Not claiming it's resolved — flagging it, since a flaky test in a merge gate is exactly the "learn to distrust red" problem.