Introducing a new way to review your PRs.

Read the announcement

alishahryar1 / free-claude-code

Use Claude Code, Codex, Antigravity, VSCode, Pi, and OpenCode (and 6 other harnesses) for free (1.3B+ free tokens) from your terminal, app, IDE, or phone, and now from the browser with native browser sessions (multi-harness + multi-model) like OpenClaw (voice supported + ToS friendly)

View on GitHub

28 documents · Updated Nov 20, 2026

docs / execution-orchestration

Execution orchestration

Execution orchestration turns an accepted request into one client-visible provider operation. It connects the resolved route to provider streams, permits safe fallback when a candidate fails, maps failures to the public protocol, and keeps output and provider lifetimes aligned. See Application execution and routing for route selection and Accounts, models, and reasoning policy for the inputs that shape a route.

Execution lifecycle

The request crosses clear boundaries between the API, application execution, and provider integration. The API accepts the request and owns its public format. Application execution selects and advances candidates. The provider integration owns the upstream operation.

This diagram shows how an accepted request becomes client-visible output.

flowchart LR
    Client[Accepted request] --> API[API adapter]
    API --> Route[Resolved route]
    Route --> Local{Local result?}
    Local -->|Yes| Output[Public output]
    Local -->|No| Executor[Provider executor]
    Executor --> Candidate[Provider candidate]
    Candidate --> Stream[Provider stream]
    Stream --> Decision{Failure?}
    Decision -->|No| Output
    Decision -->|Safe recovery| Candidate
    Decision -->|Unsafe or exhausted| Error[Mapped error]
    Error --> Output
    Output --> Client

Request acceptance

The HTTP layer accepts schema-valid requests on the supported inference paths and assigns a request ID before execution starts. The request then enters an adapter that chooses the correct public output contract, such as a complete result or a server-sent event stream.

The handler acquires one runtime generation for the request before it creates the execution stream. This gives the operation one consistent provider view while configuration changes may publish a newer generation.

References: src/…/app.py• L36-46 src/…/routes.py• L42-102 src/…/request_ids.py• L20-62

Routing context

The handler converts the public request into a routed request. This context keeps the public model for client-visible data while carrying the provider model, ordered fallback candidates, and reasoning policy used for execution.

Some requests produce a local result before the provider boundary. Other requests enter the shared executor, which applies the same candidate lifecycle across the supported wire APIs.

References: src/…/messages.py• L101-148 src/…/responses.py• L56-97 src/…/routing.py• L190-214

Runtime ownership

Runtime ownership keeps an active request attached to the provider generation that it started with. A generation can become old while a live response still uses it. Cleanup waits until its leased responses finish.

Generation lease

The generation lease gives execution access to one published provider set and its settings. Short responses release the lease after completion. Streaming responses release it only after the body is consumed, closed, or cancelled.

This prevents a configuration reload or shutdown from closing a provider that still serves an active response.

References: src/…/provider_manager.py• L39-90 src/…/provider_manager.py• L129-135 src/…/provider_manager.py• L229-286 src/…/routes.py• L42-102

Provider invocation boundary

The provider port is the narrow boundary between application execution and a vendor integration. Execution resolves the provider for the current candidate and passes the provider model, request ID, public model, reasoning policy, request headers, token estimate, and any continuation state.

The executor opens the first candidate first. It opens a later candidate only after the current operation fails and its delivered output can be restarted or continued safely. This keeps provider choice in the application while provider-specific transport stays behind the provider port.

References: src/…/ports.py• L16-43 src/…/execution.py• L158-463 src/…/execution.py• L465-645 src/…/base.py• L34-73

Provider operation control

Provider integrations control each physical upstream attempt inside the logical request operation. The application executor controls which routed candidate is active. These two levels provide bounded recovery without treating a retry at one provider as a switch to another provider.

Candidate admission

Before an upstream call starts, provider admission applies the provider’s rate and concurrency limits. The provider may retry a transient attempt within its own operation. Only after that operation returns a failure does application execution assess whether it can use the next routed candidate.

This separation prevents provider retries from consuming fallback targets too early and keeps load control with the provider integration.

References: src/…/admission.py• L74-176 src/…/admission.py• L398-466 src/…/admission.py• L710-816 src/…/execution.py• L527-619

Stream recovery

The executor tracks the difference between provider chunks and output that the client can use. A failed candidate can restart, continue with a portable text or reasoning prefix, hand off through a supported tool path, or stop when the current state is unsafe.

When recovery continues on another candidate, the executor passes a continuation seed and marks the delivery state as a continuation. It does not start an unrelated second response after an unsafe committed result.

References: src/…/stream_recovery.py• L10-106 src/…/stream_delivery.py• L11-50 src/…/stream_recovery.py• L36-164 src/…/execution.py• L549-619

Progress watchdog

The active provider stream has a progress deadline. Each non-empty chunk renews that deadline. If no progress arrives in time, execution ends with a final timeout instead of waiting without a bound.

The same deadline also limits cleanup when a stream closes without a separate active error. This prevents stalled work from holding the request open indefinitely.

References: src/…/execution.py• L465-645

Transport ownership

Each provider stream remains owned by the attempt that opened it until that attempt closes. Cleanup closes the stream before it releases provider capacity. A cleanup error is recorded without replacing the error that explains the request outcome.

Client cancellation follows the same path. It stops the response stream and lets the active attempt release its resources.

References: src/…/http.py• L25-105 src/…/admission.py• L245-364 src/…/trace.py• L65-85 src/…/execution.py• L569-579

Failure mapping

The system separates the meaning of a provider failure from the wire format seen by the client. Provider policy classifies raw failures into stable execution failures. API adapters then express that meaning in the selected protocol.

Canonical execution failures

An ExecutionFailure records the kind, status, safe message, and retry permission for one failed provider operation. Recovery uses its retry permission before the API adds request and upstream context to the final response.

The failure model distinguishes cases such as authentication, permission, invalid request, rate limit, overload, timeout, upstream failure, and provider unavailability. Error text is bounded and sensitive values are removed before it reaches logs or clients.

References: src/…/failures.py• L7-50 src/…/failure_policy.py• L61-95 src/…/diagnostics.py• L104-173

Protocol error envelopes

The API maps the shared failure into the error shape required by the selected public protocol. Before a stream sends its first public event, the adapter can return a normal HTTP error. After output starts, it sends a terminal error event inside that protocol’s stream.

Final execution errors also tell clients not to retry the exhausted request through the proxy.

References: src/…/response_streams.py• L177-213 src/…/messages.py• L227-294 src/…/responses.py• L121-172 src/…/errors.py• L34-83 src/…/errors.py• L25-67

Output handling

Provider execution uses a shared stream lifecycle. Public adapters then shape that stream as a complete response or as protocol-specific events. This lets fallback and failure rules remain common while output rules stay visible to each client type.

Messages output

The Messages adapter aggregates the shared stream when the client requests a complete JSON result. For streaming requests, it emits Anthropic server-sent events and places a terminal error in the stream when failure occurs after the first event.

The first-event boundary decides whether a failure uses an HTTP error response or an in-stream error event.

References: src/…/messages.py• L161-225 src/…/response_streams.py• L216-252 src/…/emitter.py• L33-49

Responses output

The Responses adapter preserves the public Responses request and output state while execution uses its routed provider model. Its stream assembler turns shared events into Responses events for text, reasoning, tool calls, completion, incomplete output, or failure.

If failure occurs after output starts, the assembler closes active output blocks and emits a failed Responses result. It does not expose a raw provider exception as the public response.

References: src/…/responses.py• L56-97 src/…/stream.py• L19-75 src/…/assembler.py• L32-161

Response lifetime

The managed response owns the provider generation lease until its body finishes or closes. Cleanup runs once even when cancellation or a stream error interrupts the response.

Client disconnect handling cancels inference work on the supported paths. That cancellation flows through response cleanup and closes the active provider attempt.

References: src/…/response_streams.py• L42-118 src/…/response_streams.py• L153-174 src/…/request_lifetime.py• L10-76

Change checklist

When these files or sections are changed, remember to consider these:

  • Keep the runtime lease attached to the full response body. Early release can close a provider generation during a live stream.
  • Keep provider admission retry separate from routed-candidate fallback. A retry must not skip or consume a fallback target before recovery assessment.
  • Keep stream recovery aligned with delivery evidence and continuation seeds. A candidate change must not produce mixed or unsafe output.
  • Keep progress enforcement aligned with chunk reads and cleanup. A stalled stream must not hold the request or provider capacity without a bound.
  • Keep ExecutionFailure mappings aligned across public protocols. A new failure kind needs a status, wire error, and terminal-stream rule.
  • Keep pre-output and post-output failure handling aligned. The same failure needs an HTTP form before output and a protocol event after output.
  • Keep client disconnect cancellation connected to response and provider cleanup. A break at either boundary can leave work or capacity open.

Important failure modes

TriggerConsequenceGuard
A candidate fails after it releases an unsafe output state.Execution stops instead of starting an incompatible second response.Use delivery evidence and recovery assessment before changing candidates.
A provider stream produces no non-empty chunk before its deadline.The request ends with a bounded timeout.Renew the deadline only for non-empty chunks and enforce it during stream handling.
A request stream ends without its required terminal marker.The result is treated as a stream failure rather than a valid completion.Require the provider-specific terminal marker during provider stream processing.
A client disconnects during inference.The request is cancelled and provider work must stop.Propagate cancellation through response cleanup and attempt closure.
A finalized error contains raw credentials or excessive upstream detail.Sensitive or excessive data can reach logs or clients.Redact sensitive text and cap upstream detail before formatting.
A streaming response releases its generation lease too early.A live response can use a retired or closed provider runtime.Bind release to the managed response lifetime.

Key files

FileWhy to read it
src/…/execution.pyDefines the shared candidate lifecycle, provider boundary, fallback decision, progress handling, and stream tracing.
src/…/routes.pyShows request acceptance, runtime lease acquisition, and response-lifetime binding.
src/…/routing.pyDefines the routed request context consumed by execution.
src/…/provider_manager.pyExplains generation publication, lease retention, and provider runtime cleanup.
src/…/admission.pyShows provider rate, concurrency, attempt, and retry control.
src/…/stream_recovery.pyDefines safe transitions after a provider stream failure.
src/…/stream_delivery.pyRecords client-visible delivery evidence used by recovery.
src/…/failure_policy.pyMaps raw provider failures to shared meanings and retry decisions.
src/…/response_streams.pyShows first-event errors, terminal stream errors, and response cleanup.
src/…/assembler.pyShows how shared stream events become Responses output and failure events.