Graceful Degradation When LLM Providers Go Down

A provider outage is only one of three ways an LLM stack breaks, and it is arguably the easiest to handle. Hard failures are the 5xx errors and timeouts that make a status page turn red. Soft failures return a 200 OK even though the output itself is truncated, malformed, or quietly wrong. Latency failures leave the service technically up while p99 response times triple, degrading the product without tripping a single alert. Treating "my provider is down" as the only failure shape is how teams end up unprepared for the other two, which are more common and harder to catch.

The gap between advertised and actual uptime means this needs to be built into the architecture from the start. LLM providers target 99 to 99.5% uptime, a figure that sounds close to enterprise-grade until it's set against the SLAs teams take for granted on their own cloud infrastructure, which run 6 to 14 times tighter.

Soft failures are the ones that do the most damage precisely because nothing announces them. Anthropic's September 2025 post-mortem described three overlapping infrastructure bugs that caused a portion of Claude Sonnet requests to be silently misrouted or degraded over a period of weeks, and standard HTTP monitoring showed nothing abnormal the entire time. Requests were succeeding by every metric a typical dashboard tracks. The damage was in the content of the responses, not the transport layer, which is exactly where most monitoring setups don't look.

Failure modes also differ by provider architecture, so a playbook built around one vendor's failure signature will not generalize cleanly to another's. A 2025 empirical analysis of 792 incidents across major providers found that OpenAI's architecture tended toward compartmentalized failures, contained within one service, while Anthropic's tighter service interdependencies produced more cascading cross-service impacts. Rate limiting adds a fourth wrinkle that sits outside the hard/soft/latency taxonomy: a provider that hasn't gone down at all but simply can't serve the request right now, returning HTTP 429 once requests-per-minute or tokens-per-minute limits are crossed. None of these four problems look alike from the outside, and none of them get solved by the same fix.

Correlated infrastructure and the "just add a second provider" assumption

Running two providers instead of one helps, but it does not deliver the independence the math implies, because the providers are not actually independent. They share cloud regions, network backbones, and increasingly the same accelerator supply chains, so a failure at one layer down the stack can take multiple "separate" vendors down at once.

The November 18, 2025 Cloudflare outage is the clean case: it took down ChatGPT and Claude at the same time through shared infrastructure, neutralizing a multi-provider fallback strategy that had performed exactly as intended just months earlier, in June. A strategy that worked once doesn't get credit for working reliably if the thing it was protecting against turns out to have a shared point of failure. The pattern repeated on a larger scale in 2026.

Hosting adds a second, additive layer of exposure on top of the model provider itself. The October 2025 AWS US-EAST-1 outage ran roughly fifteen hours, affected more than 140 AWS services, and disrupted hundreds of platforms that had nothing to do with each other except the region they ran in. An AI vendor and its hosting provider are two separate points of failure that stack rather than cancel out. Once that stacking is accounted for, real multi-provider deployments land closer to 99.6 to 99.8% effective uptime, well short of the 99.99%-plus figure the independence assumption would suggest on paper.

The obvious objection is that most of the time, none of this matters, because the system just works. That's true on average and beside the point. The failure isn't distributed evenly across normal operation, it concentrates in a correlated tail, and the September 2026 event is a documented instance of that tail rather than a hypothetical worst case. Multi-provider routing is still necessary. It just isn't sufficient on its own, which is the reason the rest of this piece is about the layers built on top of it.

What real outages cost when no fallback exists

The clearest illustration of what's at stake comes from healthcare. The June 10, 2025 OpenAI outage showed how quickly a single-provider dependency cascades into operational failure across unrelated industries. The June 10, 2025 OpenAI outage ran roughly fifteen hours globally, and clinical AI systems across 340 hospitals went down at the same moment, forcing emergency departments to revert to paper triage mid-shift. Average triage time rose from 18 minutes to 47 minutes at hospitals caught without a fallback plan. Hospitals that had kept a rule-based backup running in parallel maintained 22-minute triage times, and physicians barely noticed the AI was down, though these hospital-impact figures come from the Towards AI piece by Piyoosh Rai and are self-reported, not independently corroborated.

At the hospitals hit hardest, nobody had performed manual triage in eight months, which turned out to be the more useful lesson than the outage itself. Paper ESI reference guides had been digitized two years earlier and never printed again, and the backup paper forms that did exist ran out after two hours. The fallback plan existed on a slide somewhere. It had not been rehearsed, stocked, or checked, so when it was needed, it wasn't there in any usable form.

The financial and operational toll spreads well past healthcare. Major outages in 2025 shut down production for hours at a time and cost billions of dollars in aggregate across global companies. Because LLM access runs through a small number of large cloud providers, an issue at any one of them propagates fast across dozens, sometimes hundreds, of dependent businesses simultaneously.

Set against that is a plainer contrast. Teams that came through the OpenAI outage in December 2024 and the Anthropic infrastructure degradation in August 2025 without meaningful user impact shared a specific set of things they'd built before either incident started, none of it improvised: multi-provider routing already running in production, fallback models already mapped to primaries, cost alerts already configured, and feature-degradation decisions already written down rather than debated in the moment. That's the pivot point for the rest of this piece: what exactly those teams had built, and why each piece of it does a job the others don't.

Detection before failover: measuring what standard monitoring misses

None of the fallback engineering that follows matters if the system can't tell a failure is happening. Standard API monitoring, request counts, error rates, p50 latency, covers the hard-failure case well and almost nothing else. Soft degradation, latency-tail blowouts, and token-limit exhaustion all happen while standard metrics look completely normal.

p99 latency, tracked separately per provider and per model, matters more than the average for the same reason a single slow checkout line matters more to the customer stuck in it than the store's average wait time. Averages absorb outliers. Users experience the outlier. Error rate needs to be split by error type as well, because a 429 and a 503 call for opposite responses: a surge in 429s signals quota exhaustion, while a surge in 503s means the provider itself is struggling. Confusing the two leads to the wrong remediation, retrying harder into a rate limit or backing off from a provider that's actually fine.

Token consumption needs its own line item, tracked against limits by tokens-per-minute rather than requests-per-minute alone, since agents and RAG pipelines routinely exhaust TPM ceilings well before they hit any request-count ceiling. Fallback activation rate deserves a place on the same dashboard: a gradual creep upward is often the earliest available signal that something is degrading at the primary provider, well before that provider's own status page says anything. Output quality itself needs a signal too, built by sampling a slice of production responses and running lightweight evals to catch shifts in length, structure, or semantic similarity against what's expected.

Single-metric alerting misses most of this. A p99 spike on its own might be a fluke of network weather. A p99 spike combined with a climbing error rate and a rising fallback activation rate is an incident, and alerting logic needs to be built around combinations rather than thresholds on any one number in isolation.

Health checks built on a fixed timeout alone are a weak proxy for "is this provider working," because a slow response and a hung connection look identical for the first several seconds of waiting. Combining a timeout threshold with error-rate tracking closes most of that gap. Traditional circuit breakers trip on HTTP errors alone, but an LLM circuit breaker needs a second trigger for quality degradation, a distinction production deployments still underweight.

The four mechanisms that do the rerouting

Detection tells a system that something is wrong. Four mechanisms decide what happens next, and each one answers a different version of that question: health checks, circuit breakers, fallback chains, and hedged requests. Teams that build only one or two of these tend to discover the rest of the gap live, during an actual incident, which is the most expensive way to find out.

Health checks set the definition of "down" in the first place, and that definition determines how quickly the system reacts to a real failure. Combining a timeout threshold with error-rate tracking, watching for a rising share of 5xx or 429 responses, gives a far cleaner signal than a fixed timeout alone.

Circuit breakers move through three states: closed, where traffic flows normally; open, where traffic stops going to a provider already failing; and half-open, where a trial request checks whether the provider has recovered. The purpose is to stop every new request from retrying into an outage and piling additional latency on top of it. Microsoft's Azure Architecture Center documents the same pattern as a way of letting a failing dependency recover instead of getting hit with a retry storm the moment it comes back up. Production deployments converge on similar numbers, with roughly five failures to trip the breaker open, a 60-second cooldown before testing recovery, an alert at above 5% error rate, and a critical alert above 15%. Done well, this cuts failover latency from the 10-plus seconds a system would otherwise spend waiting out timeouts down to milliseconds of proactive detection. Retries feeding into this need to respect provider-specific rate-limit headers like Retry-After and the various x-ratelimit-* fields; a naive immediate retry turns a single 429 into a thundering-herd storm that makes the underlying problem worse.

Fallback chains are the ordered list of backups a system works through once the primary is marked unhealthy. Ordering that list by output fidelity to the primary, rather than by cost, usually wins in practice: a cheap backup that breaks the downstream workflow costs more in cleanup than it ever saved in API spend. Every model in production needs a documented fallback mapping written before any incident happens, spelling out known behavioral differences in verbosity, JSON adherence, and instruction-following, so the switch at 2 a.m. means the switch at 2 a.m. is not the first time anyone's seen how the backup actually behaves. Fallback and load balancing differ: load balancing distributes normal traffic across providers continuously, while fallback triggers only on failure, and a team that's built one sometimes assumes it covers the other. Multiple API keys within a single organization typically share one underlying quota rather than multiplying available headroom, and content-filter rejections need a policy remediation path separate from the fallback chain, since they aren't a transient failure to route around.

Hedged requests work differently from all three of the above: fire a duplicate request to a backup after a short delay, then use whichever response comes back first. Google's 2013 "The Tail at Scale" paper is the origin of the technique, and it found that hedging after a short delay cut 99.9th-percentile latency on a distributed lookup by more than 95%, at a cost of only 2% more total requests. Hedging protects against the slow-but-technically-up provider, not the fully-down one, which is a distinct failure shape from anything a circuit breaker addresses. Duplicate requests mean duplicate load and duplicate spend, and that trade-off needs to be made deliberately rather than by default.

Streaming failover is the case none of these four mechanisms handle cleanly. Switching providers mid-generation, after partial output has already reached the user, produces a broken experience no matter how fast the switch happens. The only clean solution is triggering failover before the first token goes out, not after.

Put side by side, the four mechanisms leave a clear map of what each one misses. Health checks don't catch a provider that's up but returning degraded output. Fallback chains don't protect against behavioral differences in what the backup actually produces. Circuit breakers don't catch the original failure, they only stop the retry storm once the failure is already underway. Hedged requests do nothing for outright downtime, only for tail latency.

The silent failure that rerouting alone cannot fix: output format and quality drift

A request that succeeds after failover is not the same thing as a request that succeeds correctly. A different model can change output format, tone, and structure without ever throwing an error, and that gap is the one most monitoring setups never catch.

The concrete version of this looks routine right up until it isn't. A support agent's fallback chain fires exactly as designed. A backup provider can respond quickly with formatted output that differs slightly from the primary's. No alert fires anywhere, because as far as the system can tell, every request succeeded. What actually happened is that the backup model formatted its structured output slightly differently than the primary, the downstream parser was never tested against that shape, and ticket after ticket came back malformed while every dashboard stayed green.

Agents built on top of these chains carry a sharper version of the same risk. A slow model, a rate-limited API, or a hallucinated tool call can degrade an agent's output quality well before anything resembling a hard crash occurs. Better-designed responses build in graceful degradation at the task level rather than treating success as binary. A research agent can return the evidence it's gathered so far and queue the synthesis step instead of failing outright, a coding agent can drop from making file edits to offering read-only diagnosis, and a support agent can stop trying to fully resolve a ticket and instead produce a clean, structured handoff for a human. Each of those is a designed fallback behavior, not a crash, and each preserves partial value instead of losing the whole request.

Closing the gap takes work that has to happen before an incident, not during one. Fallback model output shapes need to be tested against downstream parsers ahead of time, and the known behavioral differences, verbosity, JSON adherence, instruction-following, need to be written down explicitly for every fallback mapping in production. Output quality monitoring is the detection mechanism for this, sampling a slice of production responses and running lightweight evals against them. It doesn't need to score quality with any precision. It needs to notice when output length, structure, or semantic similarity to the expected pattern shifts in a way that matters, and say so before malformed tickets pile up downstream of a system that thinks everything is fine.