CodeIssuesPull RequestsActionsSecurityInsights
✨ AI
More
Settings

feat(ai): learn the provider is down once, instead of rediscovering it everywhere #5582

MergedXSccantynz wants to mergefeat/ai-outage-cachemainopened 2d ago
ccantynzcommented 2d ago

Root cause behind today's whole class of symptom.

isAiAvailable() is a configuration check, not a health check:

export function isAiAvailable(): boolean {
  if (config.aiProvider === "openai") return !!config.aiBaseUrl;
  return !!config.anthropicApiKey;
}

With a valid key and a zero balance it returns true. So every if (!isAiAvailable()) guard in the codebase passes and the 400 lands separately in each caller — which is how one billing lapse became a different-looking bug in a dozen subsystems: 405 healer give-ups, required status checks blocking merges, an empty flywheel, and nothing anywhere saying "the provider is down".

What this adds

noteAiFailure() records an account-level failure; aiOutage() reports it.

Deliberately narrow — only failures that repeat for every caller until a human tops up or fixes a key. A 429 or a 500 is retryable and must not trip it; treating a rate limit as an outage would switch the AI tier off for ten minutes over a burst. That's the false positive worth guarding, so it's mutation-tested: widening the pattern to include rate limits fails the suite.

Bounded by a TTL, not a flag someone must clear — a top-up recovers on its own without a restart. Nothing here may leave the AI tier off longer than the outage itself.

First consumer

The CI healer now checks before spending the request rather than after. Five runs a tick, every tick, is a lot of round trips spent rediscovering an exhausted balance — and its failure path feeds the cache, so the first caller to hit the wall tells the rest.

It reports ai_unavailable, already classified TRANSIENT, so a run is never permanently marked unhealable by an outage it had nothing to do with.

What this deliberately does NOT do

It does not change isAiAvailable()'s meaning. Flipping that to false during an outage would make gate.ts record AI Review as skipped for not_configured rather than ai_unavailable — which, after the required-check fix in a497f10, would silently reinstate the merge block that fix just removed.

I verified runAiReview reads settings only (gate.ts:779) before writing a line of this.

gluecron[bot]🤖 botAI Reviewcommented 2d ago

AI review unavailable

The platform's AI balance is exhausted, so AI generation is temporarily unavailable. Nothing was lost. You can queue this as a repair for the internal agent from the repository's Health page, or try again once the balance is restored. The PR is otherwise unchanged.

ccantynzAI Reviewcommented 2d ago

AI Triage

(no summary)

Priority: medium Risk area: mixed

Suggested labels: (no label suggestions) Suggested reviewers: (no reviewer suggestions)

Suggestions only — nothing has been applied. The PR author stays in control.

Cross-repo impact

See what breaks downstream if this PR merges.

Analyze →
⮌ Merged

This pull request was merged into main.

c comment · e edit title · m merge · a approve · r request changes · ? shortcuts