docs(161): the Phase-2 flip runbook + portfolio Caddyfile #5527
2 changed files+155−0
Addeddocs/RUNBOOK-161-CLEAN-REBUILD.md+113−0View fileUnifiedSplit
@@ -0,0 +1,113 @@
1# Runbook — box 161 clean rebuild (Phase 2: Coolify eviction)
2
3Owner-greenlit 2026-08-24 ("let's go big"). Goal: **one clean kernel,
4everything fenced, our caddy owning 80/443, Coolify gone** — zero new
5spend. Phase 1 (cgroup fences) and the compose bridge (#5526) are done;
6this runbook is the flip. Executed by the owner + agent together in a
7~20-minute window; every step has a rollback.
8
9## The census (verified live 2026-08-24)
10
11| Property | Runs as | Listens | Domains |
12|---|---|---|---|
13| Gluecron | docker compose stack | container :3000 (via coolify net) | gluecron.com, www |
14| GateTest web | host systemd `gatetest-web` (Next.js) | **10.0.1.1**:3000 | gatetest.ai/.io, www |
15| GateTest MCP | host systemd `gatetest-mcp` (bun) | *:8787 | mcp.gatetest.ai |
16| AlecRae web | host systemd `alecrae-web` (Next.js) | :4200 | alecrae.com, www, mail |
17| AlecRae api | host systemd `alecrae-api` (Hono/Bun) | :4100 | api.alecrae.com |
18| AlecRae MTA | host systemd `alecrae-mta` (outbound worker) | n/a | — (no ingress) |
19| DavenRoe | host systemd `davenroe-api` (FastAPI) | **10.0.1.1**:8010 | davenroe.com, www |
20| screenshot-to-code | docker (2 containers) | host 0.0.0.0:5173/:7001 | none (was internet-exposed — see §2) |
21| v9klj… app | docker via Coolify | coolify net :3000 (unrouted) | none — Caddy labels on a Traefik proxy = dead config |
22| Coolify control plane | 5 containers | — | ingress duty only |
23
24**10.0.1.1 is the coolify network's gateway IP — it dies with the
25network.** Services bound to it get rebound to 127.0.0.1 in the window
26(§4 step 3). That is the one genuinely order-sensitive part.
27
28## 1. Prerequisites (before scheduling the window)
29
30- [ ] Owner identifies the `v9klj…` app (it is one of the early products,
31 deployed via Coolify with no domain, currently unreachable): keep →
32 it gets a vhost + fence; retire → it gets stopped in the window.
33- [ ] Phase-1 fences applied (`docker update --cpus … --memory … --memory-swap …`
34 per the 2026-08-24 command block).
35- [ ] DOCKER-USER rule for 5173/7001 applied (§2) and made persistent:
36 `apt-get install -y netfilter-persistent iptables-persistent && netfilter-persistent save`.
37- [ ] Optional but recommended: lower DNS TTL for the five non-gluecron
38 domains to 300s a day ahead (fast un-flip if a vhost misbehaves —
39 though rollback here is proxy-level, not DNS, so this is belt).
40- [ ] Confirm each systemd product's bind config location (`systemctl cat
41 alecrae-web gatetest-web davenroe-api` — look for HOST/PORT env or
42 ExecStart flags) so step 3's rebinds are one-line edits, not
43 spelunking during the window.
44
45## 2. The standing security fix (do NOW, independent of the flip)
46
47screenshot-to-code's backend answered the public internet on :7001
48(verified externally 2026-08-24; docker-published ports bypass ufw):
49
50```
51iptables -I DOCKER-USER -p tcp -m multiport --dports 5173,7001 ! -s 100.64.0.0/10 -j DROP
52netfilter-persistent save # after installing iptables-persistent
53```
54
55## 3. Flip-window compose change (staged, applied in the window)
56
57In `/opt/gluecron/docker-compose.standalone.yml` ON THE BOX (merged to the
58repo as the new reality immediately after a successful flip):
59
60- caddy service: `network_mode: host`, Caddyfile mount →
61 `./scripts/portfolio/Caddyfile.portfolio`, drop the `ports:` block
62 (host networking binds directly).
63- gluecron service: add `ports: ["127.0.0.1:3005:3000"]` (caddy's loopback
64 upstream — the coolify network route it replaces dies with Traefik).
65
66## 4. The window (~20 min, owner terminal + agent verifying)
67
681. Announce/prepare: `systemctl stop gluecron-update.timer` (no deploys
69 mid-flip).
702. Rebind host services off 10.0.1.1 → 127.0.0.1:
71 `alecrae-*` and `gatetest-mcp` already bind loopback-or-all — verify;
72 edit `gatetest-web` and `davenroe-api` bind host, then
73 `systemctl daemon-reload && systemctl restart gatetest-web davenroe-api`.
74 Verify: `curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:3000/`
75 (and :8010).
763. Apply §3's compose changes on the box.
774. The flip:
78 `docker stop coolify-proxy` (Traefik releases 80/443)
79 `cd /opt/gluecron && docker compose -f docker-compose.standalone.yml up -d`
80 (caddy binds 80/443 host-side, begins TLS issuance per vhost on first
81 request — expect seconds per domain).
825. Verify every property (agent runs the checklist):
83 `for d in gluecron.com www.gluecron.com gatetest.ai gatetest.io mcp.gatetest.ai alecrae.com api.alecrae.com mail.alecrae.com davenroe.com; do curl -s -o /dev/null -m 15 -w "$d %{http_code}\n" "https://$d/"; done`
84 — every line 200/30x, none 502/000.
856. Point of no return passed → retire Coolify:
86 `docker stop coolify coolify-db coolify-redis coolify-realtime coolify-sentinel`
87 (stop, don't rm, for a week — then rm + image cleanup).
88 The v9klj app: per the prerequisite decision.
897. `systemctl start gluecron-update.timer`; push the compose+Caddyfile
90 reality to the repo so main matches the box (agent).
918. Aftercare (same day): watch /status + peer-watch; `netfilter-persistent
92 save` again if iptables changed; update BUILD_BIBLE §infra +
93 OPERATIONS.md (agent).
94
95## 5. Rollback (any point before step 6)
96
97`docker start coolify-proxy` — Traefik rebinds 80/443 (caddy host-network
98will lose the bind race after a `docker compose stop caddy`), file routes
99unchanged, everything serves exactly as before the window. The rebound
100host services keep working under Traefik too (its urls point at 10.0.1.1,
101so ALSO revert step 2's rebinds if rolling back past it — or update the
102two Traefik yamls to 127.0.0.1, which works under both proxies and makes
103rollback rebind-free; prefer that variant).
104
105## 6. What this unlocks (post-flip queue)
106
107- SSH clone (move #1) gets its clean home: port 2222 published with a
108 DOCKER-USER policy written on purpose, not inherited.
109- The sibling products join the fleet mesh: OnFailure= paging on their
110 units (the gluecron-failure-notify@ pattern ships in-repo), cards on the
111 spine, fences recorded in code.
112- 161's compose + Caddyfile become the single reviewable description of
113 the whole box — infrastructure you can `git blame`.
Addedscripts/portfolio/Caddyfile.portfolio+42−0View fileUnifiedSplit
@@ -0,0 +1,42 @@
1# Portfolio Caddyfile — box 161 after the Phase-2 Coolify eviction.
2# Applied DURING the flip window per docs/RUNBOOK-161-CLEAN-REBUILD.md,
3# not by a routine deploy: caddy cannot bind 80/443 until Traefik stops.
4#
5# Caddy runs host-networked, so every upstream is a loopback port:
6# gluecron container publishes 127.0.0.1:3005->3000 (flip-window compose
7# change); every other product is a host systemd service, rebound from
8# the dying 10.0.1.1 (coolify network gateway) to 127.0.0.1 in the same
9# window. TLS: caddy auto-issues for every vhost on first serve — DNS
10# already points at this box for all six domains.
11
12gluecron.com, www.gluecron.com {
13 encode zstd gzip
14 reverse_proxy 127.0.0.1:3005
15}
16
17gatetest.ai, www.gatetest.ai, gatetest.io, www.gatetest.io {
18 encode zstd gzip
19 reverse_proxy 127.0.0.1:3000
20}
21
22mcp.gatetest.ai {
23 reverse_proxy 127.0.0.1:8787
24}
25
26alecrae.com, www.alecrae.com, mail.alecrae.com {
27 encode zstd gzip
28 reverse_proxy 127.0.0.1:4200
29}
30
31api.alecrae.com {
32 reverse_proxy 127.0.0.1:4100
33}
34
35davenroe.com, www.davenroe.com {
36 encode zstd gzip
37 reverse_proxy 127.0.0.1:8010
38}
39
40# screenshot-to-code deliberately has NO vhost: its ports (5173/7001) are
41# tailnet-only after the DOCKER-USER rule (see runbook §2). Give it a real
42# domain here if it ever becomes a served product.
043
c comment · e edit title · m merge · a approve · r request changes · ? shortcuts