CodeIssuesPull RequestsActionsSecurityInsights
✨ AI
More
Settings

feat(ci): boot-time sweep requeues runs orphaned by the deploy restart #5468

Merged⚡ AI-generatedXSccantynz wants to mergefix/requeue-runs-orphaned-by-restartmainopened 24d ago
ccantynzcommented 24d ago

Run #257 exposed the class: the runner executes inside the app container, so a push-to-main verify-deploy run is killed by the very deploy it verifies, ~40s in — leaving a zombie "running" row the hourly reaper eventually calls "abandoned". Under reaping alone that workflow could never go green on push (this is also what happened to stuck run #253).

On boot, any run still running from before this process started is dead with certainty (runs execute in-process; one container by design). The sweep fails each honestly as runner_restarted and enqueues one fresh attempt for recent victims, which then executes against the post-deploy world.

Guards:

  • one retry per (workflow, commit) — a workflow whose execution restarts its own container cannot loop
  • 15-minute requeue window — stale zombies are finalised but not re-run ahead of live work on the serial drain
  • bounded (limit 50) and non-throwing, same contract as the reaper

Tests extend workflow-run-abandonment.test.ts: sweep ordering (sweep → reap → drain), one-shot flag set before the await, loop guard, window, bound, catch. This PR's own deploy is the live test: its push-triggered verify run will be killed by its deploy, then requeued by the new boot sweep and go green.

🤖 Generated with Claude Code

ccantynzAI Reviewcommented 24d ago

AI Triage

(no summary)

Priority: medium Risk area: mixed

Suggested labels: (no label suggestions) Suggested reviewers: (no reviewer suggestions)

Suggestions only — nothing has been applied. The PR author stays in control.

gluecron[bot]🤖 botAI Reviewcommented 24d ago

AI review unavailable

The platform's AI balance is exhausted, so AI generation is temporarily unavailable. Nothing was lost. You can queue this as a repair for the internal agent from the repository's Health page, or try again once the balance is restored. The PR is otherwise unchanged.

Cross-repo impact

See what breaks downstream if this PR merges.

Analyze →
⮌ Merged

This pull request was merged into main.

c comment · e edit title · m merge · a approve · r request changes · ? shortcuts