Code Issues Pull Requests Actions Security InsightsSettings

fix(ops): caddy cannot run on the co-tenant box, so stop starting it #5565

MergedXSccantynz wants to mergefix/caddy-cotenant-inert→mainopened 2d ago

4 changed files+76−15

Modifieddocs/ops/OPERATIONS.md+10−0View fileUnified Split

@@ -76,6 +76,16 @@ docker network connect coolify gluecron-gluecron-1 2>/dev/null || true

7676curl -s -o /dev/null -w '%{http_code}\n' https://gluecron.com/ # expect 200

7777```

7878

79**caddy does not run on the 161 box** (2026-08-29). It binds host :80/:443

80and Coolify's Traefik owns both, so it only ever reached `Created` — and it

81made `compose up` return non-zero on every tick, which the deploy script

82explained away by name. `scripts/docker-compose.coolify-override.yml` now

83puts caddy behind a `dedicated-edge` profile that nothing activates, so the

84co-tenant stack is gluecron + autoheal and a non-zero `up` means something

85again. Dedicated hosts never layer that override, so caddy stays their edge

86with automatic Let's Encrypt certs. When Phase 2 evicts Coolify, the override

87stops being layered and caddy takes the ports — no change needed here.

7989**Logs:** `journalctl -u gluecron-update` for the deploy itself;

8090`docker compose -f docker-compose.standalone.yml logs gluecron` for the app.

8191

Modifiedscripts/auto-update.sh+10−8View fileUnified Split

@@ -178,12 +178,14 @@ GIT_BRANCH="$BRANCH"

178178BUILD_SHA="$GIT_SHA"

179179export GIT_SHA GIT_BRANCH BUILD_SHA

180180

181# NOTE: `|| true` is deliberate. On this Coolify co-tenant box the compose

182# also defines a `caddy` service that tries to bind host :80/:443, which

183# Coolify's proxy already owns — so `up` reports a non-zero exit for caddy

184# even though the app (gluecron) started fine. Under `set -e` that would abort

185# the deploy BEFORE the coolify reattach + health gate below. We tolerate the

186# partial failure here and instead gate on the app's OWN health further down.

181# NOTE: tolerating a non-zero `up` is deliberate — under `set -e` it would

182# abort the deploy BEFORE the coolify reattach + health gate below, and the

183# app's OWN health is the real success signal. What it must NOT do is name a

184# cause: this used to blame the co-tenant caddy port conflict, which was true

185# every single tick, so a genuine compose failure inherited a ready-made

186# excuse and looked routine. caddy is now profiled out of the co-tenant stack

187# (scripts/docker-compose.coolify-override.yml), so a non-zero exit here is

188# once again something that deserves to be read.

187189deploy_event started

188190

189191# Build FIRST, then migrate on the new image, then switch. Until 2026-08-19

@@ -219,7 +221,7 @@ fi

219221# --remove-orphans: services deleted from the compose file (the dead

220222# gluecron-postgres, removed 2026-08-24) get their containers cleaned up on

221223# the next deploy instead of running forever as unmanaged strays.

222$COMPOSE up -d --remove-orphans || echo "$(date -Is) compose up returned non-zero (likely the co-tenant caddy port conflict) — continuing to app health gate"

224$COMPOSE up -d --remove-orphans || echo "$(date -Is) WARNING: compose up returned non-zero — continuing to the app health gate, which is what decides this deploy"

223225

224226# Co-tenant ingress reattach (Coolify boxes only).

225227# `docker compose up --build` recreates the gluecron container, which DROPS

@@ -317,7 +319,7 @@ fi

317319echo "$(date -Is) rolling back to $prev_sha" >&2

318320git reset --hard "$prev_sha"

319321docker tag "$IMAGE:last-good" "$IMAGE:latest"

320$COMPOSE up -d --no-build --remove-orphans || echo "$(date -Is) compose up (rollback) returned non-zero (likely the co-tenant caddy port conflict) — continuing to health gate"

322$COMPOSE up -d --no-build --remove-orphans || echo "$(date -Is) WARNING: compose up (rollback) returned non-zero — continuing to the health gate"

321323

322324if docker network inspect coolify >/dev/null 2>&1; then

323325 docker network connect coolify gluecron-gluecron-1 2>/dev/null || true

Modifiedscripts/docker-compose.coolify-override.yml+17−0View fileUnified Split

@@ -17,3 +17,20 @@ services:

1717networks:

1818 coolify:

1919 external: true

21 # Caddy cannot run here. It binds host :80/:443, and on this box Coolify's

22 # Traefik ("coolify-proxy") already owns both — so every `compose up` left a

23 # caddy container stuck in `Created` and returned non-zero. auto-update.sh

24 # swallowed that exit code and attributed it to "the co-tenant caddy port

25 # conflict", which meant a REAL compose failure would have been swallowed

26 # under the same excuse. A permanently-tolerated error is not a tolerated

27 # error, it is a blind spot.

28 #

29 # Giving caddy a profile that nothing activates keeps it out of the co-tenant

30 # `up` entirely. Dedicated hosts never layer this file, so there caddy stays

31 # a default service and still comes up with automatic Let's Encrypt certs.

32 # Dies with Phase 2 alongside the rest of this override: when Coolify is

33 # evicted, auto-update.sh stops including this file and caddy takes the edge.

34 caddy:

35 profiles:

36 - dedicated-edge

Modifiedsrc/__tests__/auto-update-deploy.test.ts+39−7View fileUnified Split

@@ -388,18 +388,50 @@ describe("no script anywhere swallows a migration failure", () => {

388388

389389describe("failures that must NOT fail the deploy", () => {

390390 test("a non-zero `compose up` is tolerated and gated on health instead", async () => {

391 // The real false positive here, and the reason the migration fix could

392 // not simply drop the `||`. This box is a Coolify co-tenant: the compose

393 // file defines a caddy service that cannot bind :80/:443 because the

394 // host proxy owns them, so `up` legitimately returns non-zero while the

395 // app starts fine. Treating that as fatal would abort every deploy

396 // before the health gate that actually decides.

391 // The reason the migration fix could not simply drop the `||`: under

392 // `set -e` a non-zero `up` aborts the deploy before the health gate that

393 // actually decides. Tolerating it stays correct.

394 //

395 // What changed 2026-08-29: the toleration used to NAME the co-tenant

396 // caddy port conflict as the cause. That was true on every single tick,

397 // so a genuine compose failure arrived pre-excused and read as routine.

398 // caddy is now profiled out of the co-tenant stack, and this asserts the

399 // message no longer explains away an exit code it cannot inspect.

397400 const script = await Bun.file(SCRIPT).text();

398401 expect(script).toMatch(

399 /\$COMPOSE up -d --remove-orphans \|\| echo[\s\S]{0,160}continuing to app health gate/

402 /\$COMPOSE up -d --remove-orphans \|\| echo[\s\S]{0,200}health gate/

400403 );

404 expect(script).not.toMatch(/compose up returned non-zero \(likely/);

401405 // And it still succeeds end to end with docker returning 0 everywhere.

402406 const r = await run({});

403407 expect(r.exitCode).toBe(0);

404408 });

405409});

410

411describe("the co-tenant override keeps caddy out of the stack", () => {

412 // INCIDENT 2026-08-29: caddy sat in `Created` on the 161 box because it

413 // binds host :80/:443 and Coolify's Traefik owns both. Harmless on its own

414 // — but it made `compose up` return non-zero on EVERY tick, and the deploy

415 // script explained that away by name. A permanently-expected failure is

416 // indistinguishable from a new one, which is the whole trap.

417 const OVERRIDE = resolve("scripts/docker-compose.coolify-override.yml");

418

419 test("caddy is behind a profile nothing activates", async () => {

420 const yaml = await Bun.file(OVERRIDE).text();

421 expect(yaml).toMatch(/^\s{2}caddy:/m);

422 expect(yaml).toMatch(/profiles:\s*\n\s*-\s*dedicated-edge/);

423 // Nothing may turn it on: no --profile flag anywhere in the deploy path.

424 const script = await Bun.file(SCRIPT).text();

425 expect(script).not.toMatch(/--profile/);

426 });

427

428 test("the override still declares the ingress network it exists for", async () => {

429 // The profile addition must not disturb the reason this file exists:

430 // attaching gluecron to `coolify` at CREATE time. Losing that silently

431 // reintroduces the 502-on-every-recreate this override was written for.

432 const yaml = await Bun.file(OVERRIDE).text();

433 expect(yaml).toMatch(/^\s{2}gluecron:/m);

434 expect(yaml).toMatch(/networks:\s*\n\s*-\s*default\s*\n\s*-\s*coolify/);

435 expect(yaml).toMatch(/^\s{2}coolify:\s*\n\s*external:\s*true/m);

436 });

437});

406438

c comment · e edit title · m merge · a approve · r request changes · ? shortcuts