fix(ai): an outage is deferred, not judged as unfixable #5576
ccantynzAI Reviewcommented 2d ago
AI Triage
(no summary)
Priority: medium Risk area: mixed
Suggested labels: (no label suggestions) Suggested reviewers: (no reviewer suggestions)
Suggestions only — nothing has been applied. The PR author stays in control.
gluecron[bot]🤖 botAI Reviewcommented 2d ago
AI review unavailable
The platform's AI balance is exhausted, so AI generation is temporarily unavailable. Nothing was lost. You can queue this as a repair for the internal agent from the repository's Health page, or try again once the balance is restored. The PR is otherwise unchanged.
Cross-repo impact
See what breaks downstream if this PR merges.
⮌ Merged
This pull request was merged into main.
c comment · e edit title · m merge · a approve · r request changes · ? shortcuts
The diagnosis shipped in
c434f13answered its question on the first live call, and the answer wasn't a code bug:The account is out of credit, so every AI feature is failing — healer, review, triage, spec-to-PR, chat, semantic search. That part is the owner's to fix. This PR is the part that was ours.
What was ours
The healer recorded that billing error as
ai.ci.gave_up— the identical permanent verdict it writes when the model has read the logs and judged a failure genuinely unfixable.hasMarker()skips marked runs forever, so topping up the account would not have retried a single run that failed during the outage. They were marked unhealable by an error that never looked at them.Today's recurring defect in another costume: a check that never ran, recorded as a check that came back negative — the same shape as the readiness gate that passed by skipping, and the CI gate where a run that never executed counted as green.
The fix, and the rule behind it
The four diagnoses where the model was never reached now leave no marker —
ai_unavailable,client_init_failed,claude_call_failed,run_load_failed. The run stays eligible and is retried when the environment recovers.The dividing rule is billing, not taste. Anything that reached the model and produced an answer we couldn't use (
unparseable_response,model_said_unfixable,no_usable_paths) cost money and stays marked — otherwise one bad response would re-bill on every tick for the 24 hours a run remains inside the healer's window.Deferrals stay visible:
ai.ci.deferredcarries the diagnosis but is absent fromhasMarker()'sINclause, so it informs without blocking. Written at most once per run, so a multi-hour outage can't flood the audit log at five runs a tick.Tests
ai.ci.deferredis pinned out ofhasMarker()'s clause — adding it there would silently revert this fix.Correcting the record on c434f13
I raised
max_tokens2048 → 16000 there and called truncation the probable cause. It was not the current cause. The change stands on its own merits, but the instrumentation is what earned its place — it produced the real answer within one call of going live.