Model fallback decisioning
Model fallback decisioning keeps one client response alive when an upstream model fails. It evaluates the failed attempt against delivery evidence, then either stops, hands control to the client, or opens the next ordered model with a safe restart or continuation state.
Fallback boundary
Fallback sits between model routing and response delivery. The routing layer supplies one primary target followed by ordered fallback candidates, while the fallback runner owns the sequential attempt loop; see Application execution and routing and Execution orchestration.
Candidate sequence
Each candidate names a provider-model target and may carry a reasoning preference. The runner treats the primary target and its fallbacks as one ordered response plan, so a later provider is used only after the current attempt returns a failure that permits a transition.
The candidate list also carries the original request's public model identity outside this subsystem. The fallback trace can therefore show the upstream transition without changing what the client asked for.
References: src/…/routing.py• L52-76 src/…/execution.py• L401-424
Failure assessment
Failure assessment turns one failed candidate into a small set of safe actions. It combines the failure category, whether the candidate emitted useful output, what the client has already received, and whether another candidate remains.
Recovery evidence
An ExecutionFailure is the protocol-neutral failure record. It carries a stable failure kind, status code, retry policy, and optional stream evidence. The stream evidence describes source state such as a normal stop, structured output, unsafe content, and whether a tool handoff is allowed.
The delivery state supplies the other side of the decision. It records whether any content was released and can provide a snapshot of the delivered prefix. That prefix includes text, thinking, tool-call state, safety eligibility, and an optional native continuation seed.
References: src/…/failures.py• L9-31 src/…/stream_recovery.py• L12-21 src/…/stream_recovery.py• L63-83 src/…/stream_delivery.py• L11-50
Recovery actions
The decision function returns one of three typed outcomes:
| Outcome | Meaning |
|---|---|
Fail | Raise the current failure to the client. |
Handoff | End the response because completed tool calls were released and the client can act on them. |
Fallback | Open the next candidate with a restart or continuation seed. |
The decision rules protect already released output. A clean, portable prefix can continue on another model. A prefix with unsafe content, structured output, released tool calls, or missing native continuation data stops instead. A retryable attempt with completed tool calls can use handoff when both the source and delivery state allow it.
When no stream evidence exists, the decision is more limited. The runner can preserve an existing continuation seed if the failed replacement produced nothing. Otherwise, released content makes the failure terminal. A missing next candidate also makes every otherwise valid fallback decision terminal.
References: src/…/decision.py• L18-85 src/…/stream_recovery.py• L85-145
Candidate execution
The runner executes candidates one at a time and presents their output as one async response stream. It does not expose empty provider chunks, and it closes each provider stream before it changes models.
Attempt lifecycle
When the request begins, the runner acquires the request's delivery context and starts with no continuation seed. For each candidate it records the current route, opens the provider stream, consumes meaningful output, and marks the candidate committed after the first chunk with frames or completed calls.
A replacement begins a new delivery attempt before it opens. The runner passes the current continuation seed to the candidate opener, then closes the prior provider stream in all cases. If the attempt succeeds, the runner ends the sequence. If it fails, it sends the failure and delivery evidence to the decision function.
This flow shows how routing, provider opening, delivery evidence, and the next decision form one response lifecycle.
flowchart TD
A[Ordered candidates] --> B[Open current candidate]
B --> C[Consume meaningful chunks]
C --> D{Stream completes?}
D -->|Yes| E[End response]
D -->|Failure| F[Close provider stream]
F --> G[Assess failure and delivery]
G -->|Fail| H[Raise failure]
G -->|Handoff| I[End for client tool action]
G -->|Fallback| J[Begin next attempt]
J --> B
References: src/…/runner.py• L20-63 src/…/runner.py• L78-107 src/…/trace.py• L97-117
Continuation transitions
A restart opens the next model without a prior output snapshot. A continuation opens it with a ContinuationSeed, which is a copy of output items in the client's wire protocol. This lets a compatible provider continue text or thinking without replaying an unsafe or incomplete prefix.
The runner marks the delivery state when it selects continuation or handoff. These markers let the outer response path distinguish a model transition from an ordinary stream end.
References: src/…/runner.py• L109-129 src/…/stream_recovery.py• L24-60 src/…/stream_delivery.py• L45-50
Fallback traces
Fallback traces make model transitions visible without changing the response contract. They record the failed target, the next target, the position in the candidate list, the failure class, and the recovery choice.
Transition events
The started event is emitted when a failure permits a move to the next candidate. It includes provider-model references, API-key positions, failure status, retry policy, recovery action, and recovery reason. The selected event is emitted only after the replacement produces its first meaningful output, which separates an attempted transition from a candidate that actually delivered output.
Both events identify the request and wire API. They may also include the generation identifier, so one request with multiple generations can be inspected without merging its traces.
References: src/…/traces.py• L12-56 src/…/traces.py• L59-81 src/…/runner.py• L63-76 src/…/trace.py• L72-95
Fallback state types
The fallback boundary uses small immutable values to keep provider details out of the decision rules. These types carry the candidate input, the failure evidence, the selected recovery action, and the wire format needed to preserve output.
Candidate opener contract
CandidateStreamOpener is the runner's provider-neutral opening contract. It receives the candidate index, the provider-model candidate, and an optional continuation seed. It returns an async stream of normalized StreamOutput values, while the caller remains responsible for consuming and closing that stream.
WireApi limits continuation state to the supported client protocols: messages, responses, gemini, and chat. This keeps continuation data tied to the protocol that originally delivered it.
References: src/…/types.py• L3-15 src/…/__init__.py• L1-6
Decision state
CandidateFailure is the decision input for one failed attempt. It combines optional stream context with retryability and a flag that says whether the candidate produced meaningful output. Fallback carries the next seed, the recovery action, and a reason so the runner can both execute the transition and explain it in a trace.
The public entry point is stream_model_candidates. Provider-specific execution code supplies the opener and the delivery context, while this subsystem keeps the fallback policy shared across wire APIs; see Accounts, models, and reasoning policy.
References: src/…/decision.py• L18-46 src/…/runner.py• L20-36 src/…/execution.py• L82-107 src/…/execution.py• L455-463
Change checklist
When these files or sections are changed, remember to consider these:
- Keep the recovery action and reason values aligned with
assess_recovery, because traces and continuation behavior use both values as a shared contract. - Keep
CandidateStreamOpenerinputs aligned with every wire-specific opener, especially the continuation seed and candidate-local reasoning choice. - Preserve the distinction between a candidate being attempted and being selected. The selected trace must remain tied to the first meaningful replacement output.
- Keep delivery transition state aligned with response streaming so continuation and handoff do not look like ordinary successful completion.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| A provider releases tool calls but the attempt is not safe for handoff | The runner stops instead of asking another model to continue an ambiguous tool state. | Require retryability plus both source and delivered-prefix handoff permission. |
| A delivered prefix has no native continuation seed | A model switch after released text could duplicate or corrupt the response. | Stop with missing_native_prefix rather than continue. |
| The opener returns no async iterator and no execution failure | The candidate loop raises a type error instead of silently treating the attempt as a valid stream. | Validate the provider stream before consumption. |
| A provider stream fails during consumption | The stream may remain open and obscure the original failure. | Close it in the runner's cleanup path and preserve the active failure. |
Key files
| File | Why to read it |
|---|---|
| src/…/runner.py | Shows the ordered candidate loop and the boundary between execution, delivery, and decisioning. |
| src/…/decision.py | Defines the stop, handoff, restart, and continuation choices after a failure. |
| src/…/traces.py | Defines the trace view of fallback attempts and successful replacement selection. |
| src/…/types.py | Defines the provider-neutral opener and supported wire API types. |
| src/…/stream_recovery.py | Defines the evidence and continuation rules that make a model transition safe. |
| src/…/stream_delivery.py | Defines request-local evidence about content already released to the client. |
| src/…/execution.py | Shows how routed requests build the candidate list and connect provider openers to the runner. |