CodeIssuesPull RequestsActionsSecurityInsights
✨ AI
More
Settings

feat(deploy): failed deploys and rollbacks notify the owner #5472

Merged⚡ AI-generatedXSccantynz wants to mergefeat/deploy-failure-alertsmainopened 23d ago
ccantynzcommented 23d ago

Correction to yesterday's assessment first: automatic rollback already existed (built 2026-07-21 — last-good image tagging, health+readiness gate with a real DB check, migration-failure capture, retry-loop marker). What was missing is that it's silent: a failed deploy rolls back into journalctl and the owner's next experience is "I keep pushing and nothing ships" with no visible reason.

auto-update.sh now calls scripts/notify-deploy-event.ts (via docker exec, so it runs inside the app container with DB + email creds) on all three failure paths:

  • rollback_ok — deploy failed, rolled back cleanly; message says whether migrations or health was the cause, and how to retry
  • rollback_failed — both failed; critical wording with the SSH runbook
  • no_last_good — nothing to roll back to

Fan-out reuses spine-alert-fanout (PR #5471): in-app notification + email to every site admin. Best-effort by design — the rolled-back image may predate the notify script and the catastrophic path may have no container at all; the external heartbeat and FAILED_MARKER remain the backstop.

Verified by real execution with stubbed git/docker: the failed-migration path produces exactly one rollback_ok notification with the right cause; healthy deploys produce none. (The bash-execution test suite can't run on the Windows dev box — pre-existing limitation — but runs anywhere with bash.)

🤖 Generated with Claude Code

ccantynzAI Reviewcommented 23d ago

AI Triage

(no summary)

Priority: medium Risk area: mixed

Suggested labels: (no label suggestions) Suggested reviewers: (no reviewer suggestions)

Suggestions only — nothing has been applied. The PR author stays in control.

gluecron[bot]🤖 botAI Reviewcommented 23d ago

AI review unavailable

The platform's AI balance is exhausted, so AI generation is temporarily unavailable. Nothing was lost. You can queue this as a repair for the internal agent from the repository's Health page, or try again once the balance is restored. The PR is otherwise unchanged.

Cross-repo impact

See what breaks downstream if this PR merges.

Analyze →
⮌ Merged

This pull request was merged into main.

c comment · e edit title · m merge · a approve · r request changes · ? shortcuts