feat(deploy): failed deploys and rollbacks notify the owner #5472
ccantynzAI Reviewcommented 23d ago
AI Triage
(no summary)
Priority: medium Risk area: mixed
Suggested labels: (no label suggestions) Suggested reviewers: (no reviewer suggestions)
Suggestions only — nothing has been applied. The PR author stays in control.
gluecron[bot]🤖 botAI Reviewcommented 23d ago
AI review unavailable
The platform's AI balance is exhausted, so AI generation is temporarily unavailable. Nothing was lost. You can queue this as a repair for the internal agent from the repository's Health page, or try again once the balance is restored. The PR is otherwise unchanged.
Cross-repo impact
See what breaks downstream if this PR merges.
⮌ Merged
This pull request was merged into main.
c comment · e edit title · m merge · a approve · r request changes · ? shortcuts
Correction to yesterday's assessment first: automatic rollback already existed (built 2026-07-21 — last-good image tagging, health+readiness gate with a real DB check, migration-failure capture, retry-loop marker). What was missing is that it's silent: a failed deploy rolls back into journalctl and the owner's next experience is "I keep pushing and nothing ships" with no visible reason.
auto-update.shnow callsscripts/notify-deploy-event.ts(viadocker exec, so it runs inside the app container with DB + email creds) on all three failure paths:Fan-out reuses
spine-alert-fanout(PR #5471): in-app notification + email to every site admin. Best-effort by design — the rolled-back image may predate the notify script and the catastrophic path may have no container at all; the external heartbeat and FAILED_MARKER remain the backstop.Verified by real execution with stubbed git/docker: the failed-migration path produces exactly one
rollback_oknotification with the right cause; healthy deploys produce none. (The bash-execution test suite can't run on the Windows dev box — pre-existing limitation — but runs anywhere with bash.)🤖 Generated with Claude Code