Application execution and routing
The application layer turns a client request into a streamed response from a suitable connected provider account and model. It keeps public model identity separate from provider identity, applies reasoning intent, manages safe fallback, and exposes narrow ports between API work and runtime services. See Execution orchestration for the request flow, Accounts, models, and reasoning for selection inputs, and Model fallback decisioning for recovery choices.
Execution boundary
The execution boundary joins protocol handlers to provider integrations. A handler supplies a resolved route and a runtime resolver. The application opens each candidate account and model, passes the provider-specific request context, then returns provider output as a stream.
The same candidate lifecycle serves Messages, Responses, Gemini, and Chat Completions requests. Each protocol keeps its own provider operation while the shared execution path records route data, estimates input size, supplies model metadata, and carries continuation state when recovery is safe.
This diagram shows how a public request becomes a provider stream.
flowchart LR
Client[Client request] --> Adapter[API adapter]
Adapter --> Route[Resolved route]
Route --> Executor[Application executor]
Executor --> Resolver[Account and provider resolver]
Resolver --> Provider[Selected provider model]
Provider-->>Executor[Stream output]
Executor-->>Adapter[Public response stream]
Adapter-->>Client[Client response]
References: src/…/execution.py• L74-126 src/…/execution.py• L385-478 src/…/ports.py• L21-111
Model routing
Model routing maps the public model name to a primary provider model and an ordered set of fallback targets. The resolved route keeps the public name for the response while each target supplies the provider ID, provider model, account number, and reasoning preference needed for execution.
Routing can select a configured target, repair a retired model reference, apply a model-family override, and add configured fallbacks. Native Messages adds a protocol rule: only fallback providers that support native passthrough can receive that request.
Each candidate is opened in order. A candidate becomes active only after it emits usable output. If it fails before output, execution can try the next candidate. If it fails after output, the application checks the released content and provider failure state before it continues. A safe text or thinking prefix can seed a continuation. Unsafe structured output or an ambiguous stream stops the request to avoid duplicate work.
This diagram shows the boundary between a provider failure and fallback selection.
flowchart TD
Start[Open candidate] --> Read[Read stream]
Read -->|No usable output| Next[Open next candidate]
Read -->|Usable output| Active[Keep candidate]
Active -->|Normal end| Done[Complete response]
Active -->|Retryable failure| Assess[Assess released content]
Assess -->|Safe prefix| Continue[Seed continuation]
Assess -->|Unsafe or unclear| Stop[Stop response]
Continue --> Next
Next --> Start
References: src/…/execution.py• L385-478 src/…/runner.py• L20-129 src/…/decision.py• L18-85 src/…/stream_recovery.py• L20-106 src/…/stream_delivery.py• L11-50
Reasoning policy
The reasoning policy is the application-level description of whether extended reasoning is off, on, or left to the provider. The application resolves client intent and route settings before a provider adapter translates that policy into its own request format.
Fallbacks resolve their reasoning preference from the original request. This prevents one fallback from inheriting a computation choice that came from an earlier target. The public model name remains stable even when a fallback changes the provider model or reasoning mode.
References: src/…/execution.py• L80-107 src/…/execution.py• L137-178 src/…/execution.py• L202-239 src/…/execution.py• L271-307 src/…/execution.py• L332-367 src/…/reasoning.py• L19-123
Response collection
Response collection supports application flows that must inspect a complete private provider response before deciding what to expose. It collects frames without claiming public delivery, resets its private buffer when execution starts a replacement attempt, and validates that the selected wire protocol reached a complete terminal event.
This capability is used by automatic web search. The application first asks a provider whether a hidden search tool is needed. It then either replays the original response or runs the local search and emits server-tool result frames. This keeps the provider decision private while preserving one public response contract.
References: src/…/response_collection.py• L1-58 src/…/response_collection.py• L68-129 src/…/service.py• L81-163 src/…/service.py• L189-258
Runtime lease
The request runtime lease keeps one provider generation available for the full response. The API acquires it before execution and releases it after stream completion or failure. Provider resolution and model metadata lookup therefore use the same settings view and provider set.
The lease also carries the generation identity used in trace data. Fallback resolution stays inside that generation, so a response does not switch to a new runtime state while it is already streaming.
This sequence shows the lease from request admission through response delivery.
sequenceDiagram
participant API as API route
participant Runtime as Runtime lease
participant Executor as Application executor
participant Provider as Provider
API->>Runtime: Acquire generation
Runtime-->>API: Settings and resolver
API->>Executor: Execute routed request
Executor->>Runtime: Resolve account and provider
Runtime-->>Executor: Provider and model metadata
Executor->>Provider: Start stream
Provider-->>Executor: Response chunks
Executor-->>API: Public stream
API->>Runtime: Release after stream lifetime
References: src/…/ports.py• L92-114 src/…/execution.py• L81-87 src/…/routes.py• L42-102
Integration ports
Integration ports define the capabilities that application services need without binding them to provider or runtime implementations. They keep execution, model metadata, runtime generations, task control, and web-tool access behind stable boundaries.
| Port or capability | Application responsibility |
|---|---|
ProviderPort | Stream a request through one provider protocol and carry reasoning, model metadata, headers, and continuation state. |
ProviderResolver | Select the connected provider account for a provider ID and account number. |
RequestRuntimeLease | Retain one provider generation and expose settings, readiness, provider resolution, and model metadata. |
ModelCatalogPort | Read a coherent view of configured models and cached model information. |
TaskController | Stop managed work without exposing transport-specific resources. |
| Web tools port | Search or fetch through the application web-tool workflow. |
The executor looks up model metadata by the resolved provider and provider model. This keeps capability data aligned when a request uses a fallback. The ports let runtime and provider modules change without changing the application contract.
References: src/…/ports.py• L21-176 src/…/execution.py• L163-178 src/…/execution.py• L224-239 src/…/ports.py• L1-120
Web-tool execution
The web-tool service handles supported server-side search and fetch requests within the routed Messages path. It validates whether the request is automatic or forced, uses the same executor for the private model decision, and emits provider-shaped tool events for local results.
Automatic search runs only the allowed number of searches and limits concurrent work. Fetch requests pass through an egress policy before the local client accesses a URL. Search results can be filtered by domain, while failures become tool error blocks instead of exposing client or network details.
References: src/…/service.py• L58-121 src/…/service.py• L123-258 src/…/service.py• L260-323
Change checklist
When these files or sections are changed, remember to consider these:
- Keep provider target identity and model metadata lookup aligned. A fallback must receive metadata for its own provider model.
- Keep candidate attempt state aligned with response collection. A replacement attempt must not replay private frames from the failed candidate.
- Keep protocol terminal validation aligned with every supported wire format. A collector must reject an incomplete response before a private result is reused.
- Keep fallback recovery aligned with stream delivery state. A continuation is safe only when the released prefix and provider failure support it.
- Keep provider cleanup before the next candidate starts. A closed stream must not leave an active upstream request during fallback.
- Keep runtime lease lifetime aligned with the public response stream. Early release can invalidate the provider generation during delivery.
- Keep web-tool planning aligned with routed Messages execution. Automatic search must use the same route and request identity as the final response.
- Keep web-fetch egress policy aligned with URL handling. A change to allowed schemes or private-network rules changes the safety boundary.
- Keep web-search limits aligned with emitted tool events. The event stream must show when a requested search is not executed because the limit was reached.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| A provider fails before usable output. | Execution can move to the next configured candidate. | Assess the failure inside the shared candidate lifecycle. |
| A provider fails after unsafe or unclear output. | The response stops instead of risking duplicate work. | Require stream recovery evidence before continuation. |
| A private response ends without a protocol terminal event. | The application rejects it before automatic web-tool logic uses it. | Validate the collected frames for the selected protocol. |
| A replacement attempt begins while old private frames remain. | The next decision can include output from the failed candidate. | Clear collected frames when the attempt revision changes. |
| Runtime generation changes during a response. | Provider state or model metadata can become inconsistent with the stream. | Hold one runtime lease until response delivery ends. |
| A web fetch targets a blocked scheme or private network. | The fetch is rejected and a tool error is returned. | Apply the web-fetch egress policy before the client call. |
| Automatic search returns an unexpected tool or malformed query. | The provider response is treated as a protocol failure. | Accept only the hidden search tool with one non-empty query field. |
Key files
| File | Why to read it |
|---|---|
| src/…/execution.py | Shows the shared provider execution path, streaming, fallback, recovery, and runtime metadata use. |
| src/…/routing.py | Defines public model routing, target identity, fallback order, and protocol filtering. |
| src/…/runner.py | Runs ordered candidates and controls stream handoff or continuation after failure. |
| src/…/decision.py | Defines the safe next step after a candidate failure. |
| src/…/ports.py | Defines the application contracts for providers, runtime generations, model metadata, and task control. |
| src/…/response_collection.py | Defines private response buffering and protocol-complete validation. |
| src/…/reasoning.py | Resolves client and route reasoning intent. |
| src/…/service.py | Coordinates automatic search, forced web tools, local results, and safe fetch behavior. |
| src/…/stream_recovery.py | Defines the conditions for restart, continuation, handoff, or stop after stream failure. |
| src/…/routes.py | Shows where the runtime lease surrounds API request handling and response streaming. |