feat(ai): learn the provider is down once, instead of rediscovering it everywhere #5582
gluecron[bot]🤖 botAI Reviewcommented 2d ago
AI review unavailable
The platform's AI balance is exhausted, so AI generation is temporarily unavailable. Nothing was lost. You can queue this as a repair for the internal agent from the repository's Health page, or try again once the balance is restored. The PR is otherwise unchanged.
ccantynzAI Reviewcommented 2d ago
AI Triage
(no summary)
Priority: medium Risk area: mixed
Suggested labels: (no label suggestions) Suggested reviewers: (no reviewer suggestions)
Suggestions only — nothing has been applied. The PR author stays in control.
Cross-repo impact
See what breaks downstream if this PR merges.
⮌ Merged
This pull request was merged into main.
c comment · e edit title · m merge · a approve · r request changes · ? shortcuts
Root cause behind today's whole class of symptom.
isAiAvailable()is a configuration check, not a health check:export function isAiAvailable(): boolean { if (config.aiProvider === "openai") return !!config.aiBaseUrl; return !!config.anthropicApiKey; }With a valid key and a zero balance it returns
true. So everyif (!isAiAvailable())guard in the codebase passes and the 400 lands separately in each caller — which is how one billing lapse became a different-looking bug in a dozen subsystems: 405 healer give-ups, required status checks blocking merges, an empty flywheel, and nothing anywhere saying "the provider is down".What this adds
noteAiFailure()records an account-level failure;aiOutage()reports it.Deliberately narrow — only failures that repeat for every caller until a human tops up or fixes a key. A 429 or a 500 is retryable and must not trip it; treating a rate limit as an outage would switch the AI tier off for ten minutes over a burst. That's the false positive worth guarding, so it's mutation-tested: widening the pattern to include rate limits fails the suite.
Bounded by a TTL, not a flag someone must clear — a top-up recovers on its own without a restart. Nothing here may leave the AI tier off longer than the outage itself.
First consumer
The CI healer now checks before spending the request rather than after. Five runs a tick, every tick, is a lot of round trips spent rediscovering an exhausted balance — and its failure path feeds the cache, so the first caller to hit the wall tells the rest.
It reports
ai_unavailable, already classified TRANSIENT, so a run is never permanently marked unhealable by an outage it had nothing to do with.What this deliberately does NOT do
It does not change
isAiAvailable()'s meaning. Flipping that tofalseduring an outage would makegate.tsrecord AI Review as skipped fornot_configuredrather thanai_unavailable— which, after the required-check fix ina497f10, would silently reinstate the merge block that fix just removed.I verified
runAiReviewreads settings only (gate.ts:779) before writing a line of this.