What Happens When OpenAI Goes Down? Designing LLM Fallback Chains
Every AI product has a moment it hasn't planned for. The status page turns orange. Error rates climb. Your support chatbot, your document assistant, your AI-powered search, every feature built on that one model API, stops working at the same time.
It doesn't matter how good your prompts are or how well your agent reasons. If the model behind it can't be reached, your product is down.
OpenAI isn't uniquely unreliable. Every major LLM provider, including Anthropic, Google and others, has had outages, slowdowns and capacity problems. These are complex, heavily used systems, and occasional failures are a fact of life. The question isn't whether your provider will have a bad day. It's what your product does when it happens.
The answer, for production AI systems, is a fallback chain. In this article, we'll explain what that is, how a well-designed one works, and the design decisions that separate a fallback chain that saves you during an outage from one that fails right alongside your provider. Designing and testing these chains is a core part of our AI Hardening service.
Provider failures aren't just outages
When people think about provider problems, they picture a full outage: every request failing. In reality, that's only one of several ways things go wrong, and often not the most common.
Type of failure | What it looks like | Why it's dangerous |
Full outage | Every request returns an error | Your AI features stop completely |
Partial degradation | Some requests fail, or one model is affected while others work | Harder to detect, and users get inconsistent experiences |
Slowdowns | Requests succeed, but take many times longer than normal | Requests pile up and can freeze the rest of your app |
Rate limiting | Requests are rejected because you've hit usage limits | Usually happens at your busiest moments |
Regional issues | Problems in one region or data center | Some users are affected while dashboards look fine |
Model changes and retirements | A model version is updated or deprecated | Behavior shifts or calls fail, even with no outage at all |
A good fallback strategy handles all of these, not just the dramatic full outage. Provider failures are also only one of the ways AI products break. We cover the others in 9 Ways AI Agents Fail After Launch (and How We Guard Against Each)
What is an LLM fallback chain?
A fallback chain is an ordered list of options your system works through when a model call fails. Instead of one model and one outcome, every request has a plan B, a plan C and a final safe answer.
Here's what a typical chain looks like, from first choice to last resort:
Level 1: Retry the primary model
Many failures are brief. A single retry, after a short delay, often succeeds. But retries must be limited and spaced out. Hammering a struggling provider with repeated requests makes things worse and can trigger rate limits.
Level 2: Switch to an alternative model from the same provider
Sometimes one model is affected while others from the same provider keep working. A similar model from the same provider can be a quick, low-friction fallback, since prompts and formats usually carry over well.
Level 3: Switch to a second provider
This is the level that protects you from a provider-wide outage. Requests go to a comparable model from a different provider, for example from OpenAI to Anthropic or Google. This is the most important level, and also the one that needs the most preparation, because different models respond differently to the same prompt.
Level 4: Fall back to a smaller or self-hosted model
If both main providers are struggling, a smaller, faster model, hosted elsewhere or on your own infrastructure, can still handle simpler tasks: answering common questions, classifying requests or summarizing text. It may not match the quality of your primary model for complex work, but it keeps the product useful.
Level 5: Serve from cache
For frequently asked questions, a cached answer from earlier may be perfectly good. Semantic caching, which matches similar questions rather than only exact ones, can cover a surprising share of traffic during an outage.
Further reading: Why Your AI App's API Bill Keeps Growing (and Where the Money Goes) [ADD LINK once this blog is published]
Level 6: Degrade gracefully
When nothing else can answer safely, the user gets a clear, honest message, and the request is handed off where possible: to a human support agent, a ticket queue or a "we'll email you" flow. Context is preserved, so nobody has to start again. What the user should never see is a blank screen, a spinning icon or a technical error.
Not every product needs all six levels. A simple internal tool might need only retries and a second provider. A customer-facing agent handling thousands of conversations a day usually benefits from the full chain.
Five design decisions that make or break a fallback chain
Listing backup models is the easy part. These are the decisions that determine whether the chain actually works when you need it.
1. Deciding when to fall back
Falling back only on hard errors isn't enough. A provider that takes 40 seconds to respond is effectively down for your users. Good chains trigger on errors, rate limit responses, timeouts and unusually high latency.
This is where a circuit breaker comes in. When failures for one provider cross a threshold, the circuit "opens," and requests skip that provider entirely for a while, instead of each user waiting for a timeout. The system then tests the provider periodically and moves traffic back gradually once it's healthy. Without this, every request during an outage pays the full cost of a failed attempt before reaching the fallback.
2. Making prompts work across models
A prompt tuned carefully for one model often performs worse on another. Different models follow instructions differently, handle system messages differently and format tool calls differently. If your fallback model receives the same prompt untested, the outage might be survived on paper while users get noticeably worse answers.
The fix is to treat prompts as model-specific where it matters: keep a shared core, adapt the parts that need it for each model in the chain, and test each version.
3. Keeping outputs consistent
Your application expects responses in a particular shape. A fallback model that returns slightly different structure, extra text or a different tool call format can break downstream code just as badly as an outage. Every response, from every level of the chain, should pass through the same output validation before it's used. A normalization layer that converts each provider's response into one standard format makes this much easier.
4. Testing fallback quality before you need it
The worst time to discover your backup model can't handle your use case is in the middle of an outage. Every model in the chain should be run against the same evaluation suite as your primary model: real questions, edge cases and known failure cases, scored against clear criteria. This tells you exactly what quality to expect at each level, and which tasks a smaller model should and shouldn't handle.
Further reading: Prompt Changes Broke Production: Why AI Apps Need Evals in CI [ADD LINK once this blog is published]
5. Accounting for cost, data and compliance
Fallback models have different prices, so an outage can shift your costs suddenly. Budgets and alerts should cover every provider, not just the primary.
Data matters too. If your contracts or regulations limit where customer data can be processed, every provider in the chain must meet the same requirements, including data location, retention and agreements in place. A fallback that violates your data commitments isn't a fallback you can use.
Not sure your AI product would survive a provider outage? We design and test fallback chains as part of our AI Hardening work. Get a free 30-minute review →
Common fallback mistakes
These are the issues we see most often when reviewing AI products that already have some kind of fallback in place:
Mistake | What goes wrong |
Unlimited retries | Retrying a failing provider over and over slows every request and can trigger rate limits |
No timeouts | The fallback never starts, because the first request is still waiting |
Fallback configured, never tested | Expired keys, missing permissions or broken prompts are only discovered mid-outage |
Same prompt for every model | The product stays up, but answer quality drops sharply |
No output validation on fallbacks | Different response formats break the app in new ways |
No visibility into fallback use | Nobody knows how often the chain is used, or that the primary has been degraded for days |
Agents switching models mid-task | A multi-step agent changes model halfway through and loses track of its plan or tool formats |
That last one deserves a note. For multi-step agents, it's often safer to decide the model at the start of a task and keep it for the whole run, or to restart the task cleanly on the fallback model, rather than switching midway.
How to know your fallback chain actually works
A fallback chain that has never been tested is a hope, not a safeguard. Before relying on one, we run deliberate failure drills in a staging environment:
Block the primary provider completely, and confirm traffic moves to the next level
Add artificial delays, and check that timeouts and circuit breakers trigger correctly
Simulate rate limit responses during a traffic spike
Disable two levels at once, and confirm the user still gets a graceful response
Run the full evaluation suite against each fallback model, and compare quality
Check that alerts fire, dashboards show the switch, and traces record which model handled each request
Once in production, keep watching. Dashboards should show how much traffic each level handles, how often the circuit breaker opens, and how answer quality and cost compare across models. If your fallback is quietly handling 20% of traffic, you want to know about it.
Our Monitoring & Observability service sets up exactly these dashboards and alerts, including full tracing of which model handled each request.
Quick self-check: would your AI product survive an outage?
If your main LLM provider went down right now, what would your users see?
Do your model calls have timeouts, and limited retries with backoff?
Is there a second provider configured, with working keys and access?
Have your prompts been tested on every fallback model?
Does every response, from every model, pass the same output validation?
Do you have a circuit breaker, or does every request wait to fail first?
Do fallback providers meet the same data and compliance requirements as your primary?
Have you ever simulated an outage to see what happens?
Would your team be alerted the moment fallbacks start being used?
If you can't confidently answer yes to most of these, an outage will likely hit your users directly.
Build for the bad day
LLM providers will keep improving, but no provider can promise perfect uptime. The AI products that earn customer trust are the ones that keep working when their dependencies don't.
A well-designed fallback chain is one of the most valuable safeguards you can add to an AI product, and one of the easiest to get subtly wrong. At Codersarts, designing, building and testing fallback chains is a core part of our AI Hardening service, alongside output validation, guardrails, cost controls and evaluations. Want a full review of your system first? Start with a Launch Readiness Audit.
Related reading
9 Ways AI Agents Fail After Launch (and How We Guard Against Each)
Why Your AI App's API Bill Keeps Growing (and Where the Money Goes) [ADD LINK once this blog is published]
Prompt Changes Broke Production: Why AI Apps Need Evals in CI [ADD LINK once this blog is published]
Comments