docs(ci): the attribution guarantee promised more than the gate delivers #5601
2 changed files+134−13
Modifieddocs/CI-ATTRIBUTION-GUARANTEE.md+47−13View fileUnifiedSplit
@@ -1,6 +1,7 @@
11# The CI attribution guarantee
22
3> **When CI fails because of us, we say so, and it does not block your merge.**
3> **When CI fails because of us, we say so — in the reason text, in the
4> statistics, and in what we ask you to do about it.**
45
56This is a promise about behaviour that is already implemented and tested, not a
67roadmap item. Every claim below names the code that makes it true, so it can be
@@ -36,13 +37,32 @@ They render differently on purpose: muted grey rather than red, `⟳` rather tha
3637`✗`, and wording that says *"interrupted by a platform restart — not your
3738code"* or *"the worker running it died — not your code"*.
3839
39### 2. Our failure never blocks your merge
40### 2. Our failure is never reported as yours, and clears itself
4041
41The merge gate distinguishes `CI did not complete — re-run the workflow` from
42`CI failed: <reason>` (`src/lib/gate.ts`). An infrastructure conclusion is not
43counted against you, and a run interrupted by our restart is requeued
44automatically — once per (workflow, commit), so a persistently broken commit
45does not loop.
42**Corrected 2026-08-31.** This section previously read *"Our failure never
43blocks your merge"*, and the code has never done that. `decideCiGate` in
44`src/lib/gate.ts` returns `passed: false` for an infrastructure conclusion —
45it blocks, and says `CI did not complete — re-run the workflow` instead of
46`CI failed`.
47
48The code is right and the old wording was the overstatement, so the wording
49changed. A run that did not finish has not judged your code, and a gate that
50waved it through would be doing the thing this whole document exists to
51refuse: scoring a check that could not be performed as one that passed.
52
53What we actually promise here is narrower and worth more:
54
55- **The reason names us.** `CI did not complete — <what broke on our side>` is
56 a different sentence from `CI failed: <your test>`, and you never have to
57 guess which one you are looking at.
58- **You are not asked to do anything about it.** A run interrupted by our
59 restart is requeued automatically — once per (workflow, commit), so a
60 persistently broken commit does not loop. In the ordinary case the block
61 clears on its own within minutes without you touching anything.
62- **It does not count against your repo's health** — see §3.
63
64If one of our failures ever blocks you and does NOT clear itself, that is a
65bug in the requeue path, and it is ours to fix.
4666
4767### 3. Our failures do not poison your statistics
4868
@@ -50,13 +70,27 @@ does not loop.
5070platform that counts its own outages against your repo's health score is
5171lying to you with your own data.
5272
53### 4. An outage of ours never blocks a merge on a check we could not run
73### 4. An outage of OUR OWN advisory service does not block your merge
74
75Added 2026-08-30 after an AI provider outage; heading narrowed 2026-08-31 to
76match what the code does. An `AI Review` check that could not run is recorded
77`skipped` with a reason, and a *required* status check is satisfied by that
78specific skip — but ONLY by that one. A skipped `CI` check, a skipped
79security scan, and an AI review disabled in settings are all still refused.
80
81The line between this and §2 is not "whose fault was it" — both are ours. It
82is **what went unverified**:
83
84- AI review is something WE add on top of your tests. When our provider is
85 down, blocking your merge means punishing you for our outage over a check
86 you never asked for.
87- CI is YOUR tests. When a run does not complete, nothing has checked your
88 code at all, and merging it would be shipping on no evidence.
5489
55Added 2026-08-30 after an AI provider outage: an `AI Review` check that could
56not run is recorded `skipped` with a reason, and a *required* status check is
57satisfied by that specific skip — but ONLY by that one. A skipped `CI` check,
58a skipped security scan, and an AI review disabled in settings are all still
59refused, because "we could not look" is not "we looked and it was fine".
90So the same sentence — "we could not look" is not "we looked and it was
91fine" — decides both, and lands differently because the stakes differ. It is
92the only extra check credited, and adding a second one to that list should
93require the same argument to be made again out loud.
6094
6195### 5. A failure tells you what it was
6296
Addedsrc/__tests__/attribution-guarantee-honesty.test.ts+87−0View fileUnifiedSplit
@@ -0,0 +1,87 @@
1/**
2 * The customer-facing promise must not outrun the code.
3 *
4 * docs/CI-ATTRIBUTION-GUARANTEE.md is written to be checked rather than
5 * believed — it names the code behind each claim. That only works if the
6 * claims stay true, and one of them had not been true for as long as the
7 * document existed: §2 was headed "Our failure never blocks your merge" while
8 * `decideCiGate` returned `passed: false` for every infrastructure
9 * conclusion. It blocks, and always did.
10 *
11 * Nobody noticed because a document does not fail a build. That is the same
12 * shape as the defects this session kept finding — something that reports a
13 * state it does not have — and it is worse here, because this file is the
14 * thing we would point a customer at.
15 *
16 * So the two are pinned to each other. If the gate's behaviour changes, this
17 * fails and the promise gets rewritten deliberately rather than quietly
18 * falling out of date.
19 */
20
21import { describe, expect, test } from "bun:test";
22import { readFileSync } from "fs";
23import { resolve } from "path";
24import { decideCiGate, type CiRunRow } from "../lib/gate";
25
26const root = resolve(import.meta.dir, "../..");
27const doc = readFileSync(resolve(root, "docs/CI-ATTRIBUTION-GUARANTEE.md"), "utf8");
28
29function run(over: Partial<CiRunRow> = {}): CiRunRow {
30 return {
31 workflowId: "w1",
32 name: "CI",
33 status: "failure",
34 conclusion: "runner_restarted",
35 startedAt: new Date(),
36 createdAt: new Date(),
37 ...over,
38 };
39}
40
41describe("the document does not promise more than the gate delivers", () => {
42 test("an infrastructure failure BLOCKS — so the doc must not say otherwise", () => {
43 const verdict = decideCiGate([run()], [{ id: "w1", name: "CI" }]);
44 expect(verdict.passed).toBe(false);
45
46 // Checked on HEADINGS only, and this is a deliberate narrowing rather
47 // than a weakening. The body legitimately quotes the old wording while
48 // explaining why it was wrong, and a test that forbade the phrase
49 // outright would force the correction note to be written around it —
50 // contorting the honest paragraph to satisfy the honesty check.
51 //
52 // The rule: no section may PROMISE a non-blocking merge, unless it is
53 // scoped to our own advisory check (§4, the AI review). That is the one
54 // exception, and it earned its place with an argument that is spelled
55 // out in the document.
56 const headings = doc
57 .split(/\r?\n/)
58 .filter((l) => l.startsWith("#"))
59 .map((l) => l.toLowerCase());
60 const offenders = headings.filter(
61 (h) => /blocks? your merge/.test(h) && !h.includes("advisory")
62 );
63 expect(offenders).toEqual([]);
64 });
65
66 test("it blocks with OUR name on it, not by calling the code failed", () => {
67 // The promise that survived: the reason text distinguishes the two.
68 const verdict = decideCiGate([run()], [{ id: "w1", name: "CI" }]);
69 expect(verdict.details).toContain("CI did not complete");
70 expect(verdict.details).not.toContain("CI failed");
71 });
72
73 test("a genuine test failure still says CI failed", () => {
74 const verdict = decideCiGate(
75 [run({ conclusion: "failure" })],
76 [{ id: "w1", name: "CI" }]
77 );
78 expect(verdict.passed).toBe(false);
79 expect(verdict.details).toContain("CI failed");
80 });
81
82 test("the doc still points at the file that decides this", () => {
83 // A promise that names no code is a promise that cannot be checked, which
84 // is the one thing this document says it is not.
85 expect(doc).toContain("src/lib/gate.ts");
86 });
87});
088
c comment · e edit title · m merge · a approve · r request changes · ? shortcuts