CodeIssuesPull RequestsActionsSecurityInsights
✨ AI
More
Settings

fix(ci): a typecheck that gets OOM-killed now says so, and fits #5566

MergedXSccantynz wants to mergefix/ci-oom-ceilingmainopened 2d ago
ccantynzcommented 2d ago

The symptom, three times this week

==> Typecheck
$ bunx tsc --noEmit
[exit 137 in 22058ms]

137 is 128+9 — SIGKILL. No error, no output, nothing to search for. And because the CI gate blocks merges on a red run, the fix for the ceiling could not land through the gate the ceiling was breaking. 2g on 2026-08-27, 4g today — both set from the app's steady state alone, both fatal.

Measured, not guessed

peak time result
tsc --noEmit 3,349 MB 111s exit 0
bun --smol … tsc 3,441 MB 187s exit 0

The app holds ~800 MB in the same cgroup, because the CI runner executes inside the app container — ~4.2 GB together, which is exactly why a 4 GiB ceiling moved the death from 22s to 36s rather than curing it. The kernel agrees:

oom-kill:constraint=CONSTRAINT_MEMCG ... task=node
Killed process (node) anon-rss:3530020kB

(node there is Bun's own shim in /tmp/bun-node-*, not a second runtime.)

bun --smol is not a way out — it costs 68% more time and saves nothing. skipLibCheck is already on. 6g clears the measured sum with headroom; every other container on the box totals ~400 MB, so the host absorbs it.

Two changes, because the number alone just moves the next surprise

  • scripts/ci-typecheck.sh runs the same command and, on 137 only, prints what was killed, where the bound comes from, and how to confirm it. It refuses to decorate a real type error with a memory story, and it points at the documented co-tenant recreate — a hand-run compose up on that box is what 502'd the site earlier today.
  • The compose ceiling carries its measurements, and the stale "2g suits a small shared box" advice is gone. An .env override silently wins over the compose default — the two-sources-of-truth shape behind most of this week's findings — so that's now stated where someone raising the limit will actually read it.

Both files say plainly that this is a stopgap: the durable fix is moving CI out of the app container, so a build stops being bounded by the running site's memory budget.

Tests

Five, running the real wrapper against a stub bunx: 137 explains itself and still exits 137; exit 1/2 are left undecorated; 0 stays quiet. Plus guards that the workflow calls the wrapper and that the ceiling can't be tidied back under 5g without arguing with the measurement.

Note for merging

The box's .env currently pins GLUECRON_MEM_LIMIT=4g, which overrides the compose default. That line has to move to 6g before this PR's own CI can go green — merging alone won't do it.

ccantynzAI Reviewcommented 2d ago

AI Triage

(no summary)

Priority: medium Risk area: mixed

Suggested labels: (no label suggestions) Suggested reviewers: (no reviewer suggestions)

Suggestions only — nothing has been applied. The PR author stays in control.

gluecron[bot]🤖 botAI Reviewcommented 2d ago

AI review unavailable

The platform's AI balance is exhausted, so AI generation is temporarily unavailable. Nothing was lost. You can queue this as a repair for the internal agent from the repository's Health page, or try again once the balance is restored. The PR is otherwise unchanged.

Cross-repo impact

See what breaks downstream if this PR merges.

Analyze →
⮌ Merged

This pull request was merged into main.

c comment · e edit title · m merge · a approve · r request changes · ? shortcuts