Off-box watchdog: investigation findings + Tailscale-based fix design #3836

Originally written by @ccantynz-alt on GitHub.
Imported from https://github.com/ccantynz-alt/jarvis-platform/pull/2

What this is

Doc-only update to docs/OFF-BOX-WATCHDOG.md recording the 2026-07-19 investigation into the watchdog's unreliable execution and the current "down" state, plus the redesign that fixes it.

Findings

1. The current status: down (since 06:03 UTC 2026-07-19) is likely a false alarm. The routine recorded "down" ~3 hours before the commit restoring the :9212 liveness endpoint (e728388) was even authored — a deploy race, not evidence of a dead box. On-box verification still needed (ss -tlnp | grep 9212, systemctl status jarvis-dashboard).
2. Root cause hypothesis for the silent write failures: verified from inside a Claude Code cloud container that the environment's egress allowlist 403-blocks ntfy.sh and the Tailscale control plane, and plain HTTP to the box's raw IP is unroutable. This matches the "reads work, writes vanish" symptom and means the health check itself may not have been reaching the box.
3. One git write from the routine did land (af340f2), so GitHub egress works; per-run logs in the routines web UI remain the only way to see the remaining failures.

Fix design (documented, not yet executed)

Join the watchdog to the tailnet: allowlist *.tailscale.com in the environment's network policy, add an ephemeral pre-authorized tagged auth key as an env var, run tailscaled in userspace mode, probe the gateway at https://jarvis.tailbd6217.ts.net:8443/health, and alert via the gateway's /internal/notify with a scoped token (heartbeat-token pattern) with ntfy/email as the dead-box backstop. Two prerequisites are owner-only (environment network policy + Tailscale admin key) and are called out in the doc.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HA1aZrJyqbEN29hAehKJ4c

Generated by Claude Code