fix(ci): runner concurrency defaults to 1 — the 3-default self-DDoSed the box #5517
ccantynzAI Reviewcommented 10d ago
AI Triage
(no summary)
Priority: medium Risk area: mixed
Suggested labels: (no label suggestions) Suggested reviewers: (no reviewer suggestions)
Suggestions only — nothing has been applied. The PR author stays in control.
gluecron[bot]🤖 botAI Reviewcommented 10d ago
AI review unavailable
The platform's AI balance is exhausted, so AI generation is temporarily unavailable. Nothing was lost. You can queue this as a repair for the internal agent from the repository's Health page, or try again once the balance is restored. The PR is otherwise unchanged.
Cross-repo impact
See what breaks downstream if this PR merges.
⮌ Merged
This pull request was merged into main.
c comment · e edit title · m merge · a approve · r request changes · ? shortcuts
The real root cause of today's three outages (supersedes 950ab5c's iptables theory)
The runner lives inside the app container. #5510's default of 3 let three full-suite CI runs (4,670 tests each) execute concurrently on the one small VPS — CPU starvation made new TCP connections time out platform-wide while kernel ICMP kept answering, which masqueraded perfectly as a firewall wedge. Every "fix" that worked (docker restart ×2) worked by killing the CI runs; the final confirmation is that outage #3 resolved itself when the suites hit their 600s step timeouts, with no intervention at all. The compat-shim CI log had the tell all along:
tsckilled at 744s.Parallelism itself is correct and stays. The default now matches the deployment reality: serial on the one box;
WORKFLOW_CONCURRENCYis an explicit per-host opt-in for a runner with headroom the app doesn't pay for.Operational rule recorded alongside: never dispatch multiple CI runs simultaneously on this box.
7/7 tests, typecheck clean.
🤖 Generated with Claude Code