Reliability tier 2: SLO dashboard, dedicated CI runner, mirror-schedule keepalive#214
○Openccantynz opened this issue 23d ago
ccantynzAuthorcommented 23d ago
Follow-ups to the 2026-08-09 five-move reliability build (external heartbeat, spine alerting, deploy-failure alerts, CI gate, synthetic journeys). Next tier, in value order:
SLO dashboard on /admin/spine — uptime %, p95 latency, and error-rate trends from data already collected (synthetic_checks durations, platform_errors rates). Today the spine shows current state; professionals also see direction. ~1 day.
Dedicated CI runner container — CI currently executes inside the app container: fine at today's load (24s per suite, serial), but test CPU/memory contends with production traffic and a pathological test could OOM the app. A second compose service (gluecron-runner) sharing the repos volume isolates it. ~1 day.
GitHub mirror schedule keepalive — GitHub disables scheduled workflows after 60 days without repo activity. The heartbeat workflow (the outage watcher of last resort) lives on the mirror, which only gets occasional pushes. #207 (auto mirror-push) fixes this as a side effect; until then, any mirror push resets the clock. Worth landing #207 within the next month for this reason alone.
Journey depth — an open-PR journey (create branch + PR + comment via API with a probe token, then clean up) would exercise the write path the current read-only journeys don't.
STATUS: not started; this issue is the roadmap.
c comment · e edit title · x close/reopen · ? shortcuts
Follow-ups to the 2026-08-09 five-move reliability build (external heartbeat, spine alerting, deploy-failure alerts, CI gate, synthetic journeys). Next tier, in value order:
SLO dashboard on /admin/spine — uptime %, p95 latency, and error-rate trends from data already collected (
synthetic_checksdurations,platform_errorsrates). Today the spine shows current state; professionals also see direction. ~1 day.Dedicated CI runner container — CI currently executes inside the app container: fine at today's load (24s per suite, serial), but test CPU/memory contends with production traffic and a pathological test could OOM the app. A second compose service (
gluecron-runner) sharing the repos volume isolates it. ~1 day.GitHub mirror schedule keepalive — GitHub disables scheduled workflows after 60 days without repo activity. The heartbeat workflow (the outage watcher of last resort) lives on the mirror, which only gets occasional pushes. #207 (auto mirror-push) fixes this as a side effect; until then, any mirror push resets the clock. Worth landing #207 within the next month for this reason alone.
Journey depth — an open-PR journey (create branch + PR + comment via API with a probe token, then clean up) would exercise the write path the current read-only journeys don't.
STATUS: not started; this issue is the roadmap.