Off-box watchdog: investigation findings + Tailscale-based fix design #3836
1 changed file+73−0
Modifieddocs/OFF-BOX-WATCHDOG.md+73−0View fileUnifiedSplit
@@ -74,6 +74,79 @@ fast; this is a slower backstop that still needs its execution reliability
7474proven before it's trusted for the one case that matters most (the box
7575being fully dead).
7676
77## Investigation 2026-07-19 (cloud session, "box-watchdog issues")
78
79Findings from a Claude Code cloud session that dug into the failures above.
80None of this is fixed yet — it's diagnosis plus a concrete design; the two
81prerequisite changes are Craig-only (environment settings + tailnet key).
82
83### 1. The current "down" state is probably a false alarm
84
85`docs/.watchdog-state.json` on main says `{"status":"down","since":
86"2026-07-19T06:03:14Z"}` (commit `af340f2`, landed 06:03 UTC — so at least
87one git write from the routine HAS succeeded, contrary to the all-writes-fail
88pattern above). But the commit restoring the `:9212` endpoint itself
89(`e728388`) was only authored ~09:00 UTC — three hours AFTER the routine
90recorded "down". The routine was probing an endpoint that didn't exist yet
91(or wasn't deployed/restarted on the box). No up-transition has been
92committed since, which means either the box never got the `:9212` restore
93deployed, or the routine's later runs are still failing to write. On-box
94check: `ss -tlnp | grep 9212` and `systemctl status jarvis-dashboard`.
95
96### 2. Likely root cause of the silent write failures: the cloud
97### environment's egress allowlist
98
99Verified from inside a Claude Code cloud container (same kind of
100environment the routine runs in): outbound traffic goes through an
101allowlisting proxy, and
102
103- `ntfy.sh:443` → **CONNECT rejected 403** (policy denial). The routine's
104 ntfy alert POSTs and the ntfy-cache state design both die here —
105 silently, exactly matching the observed "reads work, writes vanish"
106 symptom for the ntfy designs.
107- Plain HTTP to `66.42.121.161:9212` → not proxied (405 from the proxy for
108 non-CONNECT) and direct egress is firewalled. **If the routine's
109 environment has the same policy, the health check itself may not be
110 reaching the box either**, and a failed check is indistinguishable from
111 a down box → false "down".
112- `controlplane.tailscale.com` / `login.tailscale.com` /
113 `pkgs.tailscale.com` → all 403. So a cloud session cannot currently join
114 the tailnet at all.
115
116GitHub is allowlisted (git push works — see the successful `af340f2`), so
117the earlier git-push failures were probably a different issue (per-run
118execution logs in the web UI are still the only way to see them).
119
120### 3. The fix design: put the watchdog ON the tailnet
121
122"A cloud routine has no path onto the tailnet" (above) is only true under
123the default network policy. The environment's network policy is
124user-configurable, which unlocks a much better design:
125
1261. **Craig, in the environment settings** (claude.ai/code → environment →
127 network policy): allow `*.tailscale.com` (control plane + DERP relays)
128 and `pkgs.tailscale.com`; keep/add `ntfy.sh` only if it stays as backup
129 channel. Note the proxy's no-proxy list already exempts `100.64.0.0/10`
130 (the Tailscale CGNAT range), so tailnet traffic won't fight the proxy.
1312. **Craig, in the Tailscale admin console**: create an **ephemeral,
132 pre-authorized, tagged** auth key (e.g. tag `tag:watchdog`) with ACLs
133 allowing only the gateway/dashboard serve ports on `jarvis`. Put it in
134 the environment as an env var (e.g. `TS_AUTHKEY`) — never in a prompt
135 or this repo.
1363. The routine then runs `tailscaled` in userspace-networking mode, joins
137 as an ephemeral node, and probes
138 `https://jarvis.tailbd6217.ts.net:8443/health` (the gateway — the real
139 front door) instead of the raw-IP `:9212` ping.
1404. Alerting: POST to the gateway's `/internal/notify` with a scoped token
141 (same pattern as `JARVIS_HEARTBEAT_TOKEN_vapron158`) so a "Jarvis box
142 unreachable" alert is spoken through Jarvis's own channel when the
143 gateway is up but something else is wrong — with ntfy (or email via a
144 connector) as the channel for the case that matters most, the box being
145 fully dead and the gateway unreachable.
146
147Until (1) and (2) are done, the watchdog stays as-is: built, plumbed,
148unverified, and not to be trusted (see above).
149
77150## To change the check target, schedule, or alert channel
78151
79152Use the `/schedule` skill or call `RemoteTrigger` directly:
80153
c comment · e edit title · m merge · a approve · r request changes · ? shortcuts