CodeIssuesPull RequestsActionsSecurityInsights
✨ AI
More
Settings

fix(ai): an outage is deferred, not judged as unfixable #5576

MergedXSccantynz wants to mergefix/healer-defer-transientmainopened 2d ago
ccantynzcommented 2d ago

The diagnosis shipped in c434f13 answered its question on the first live call, and the answer wasn't a code bug:

claude_call_failed: 400 invalid_request_error
"Your credit balance is too low to access the Anthropic API."

The account is out of credit, so every AI feature is failing — healer, review, triage, spec-to-PR, chat, semantic search. That part is the owner's to fix. This PR is the part that was ours.

What was ours

The healer recorded that billing error as ai.ci.gave_upthe identical permanent verdict it writes when the model has read the logs and judged a failure genuinely unfixable. hasMarker() skips marked runs forever, so topping up the account would not have retried a single run that failed during the outage. They were marked unhealable by an error that never looked at them.

Today's recurring defect in another costume: a check that never ran, recorded as a check that came back negative — the same shape as the readiness gate that passed by skipping, and the CI gate where a run that never executed counted as green.

The fix, and the rule behind it

The four diagnoses where the model was never reached now leave no markerai_unavailable, client_init_failed, claude_call_failed, run_load_failed. The run stays eligible and is retried when the environment recovers.

The dividing rule is billing, not taste. Anything that reached the model and produced an answer we couldn't use (unparseable_response, model_said_unfixable, no_usable_paths) cost money and stays marked — otherwise one bad response would re-bill on every tick for the 24 hours a run remains inside the healer's window.

Deferrals stay visible: ai.ci.deferred carries the diagnosis but is absent from hasMarker()'s IN clause, so it informs without blocking. Written at most once per run, so a multi-hour outage can't flood the audit log at five runs a tick.

Tests

  • The two sets must partition the union exactly — a ninth diagnosis added without classifying it fails the build rather than defaulting to permanent.
  • The literal production 400 payload is pinned transient.
  • ai.ci.deferred is pinned out of hasMarker()'s clause — adding it there would silently revert this fix.
  • Verified by mutation: reclassifying the billing failure as permanent fails the suite.

Correcting the record on c434f13

I raised max_tokens 2048 → 16000 there and called truncation the probable cause. It was not the current cause. The change stands on its own merits, but the instrumentation is what earned its place — it produced the real answer within one call of going live.

ccantynzAI Reviewcommented 2d ago

AI Triage

(no summary)

Priority: medium Risk area: mixed

Suggested labels: (no label suggestions) Suggested reviewers: (no reviewer suggestions)

Suggestions only — nothing has been applied. The PR author stays in control.

gluecron[bot]🤖 botAI Reviewcommented 2d ago

AI review unavailable

The platform's AI balance is exhausted, so AI generation is temporarily unavailable. Nothing was lost. You can queue this as a repair for the internal agent from the repository's Health page, or try again once the balance is restored. The PR is otherwise unchanged.

Cross-repo impact

See what breaks downstream if this PR merges.

Analyze →
⮌ Merged

This pull request was merged into main.

c comment · e edit title · m merge · a approve · r request changes · ? shortcuts