Provider integration layer
The provider integration layer gives the gateway one boundary for many model vendors. It accepts gateway request shapes, sends them through vendor transports, streams gateway response formats, and reports provider model metadata. Shared admission and failure rules protect each provider while adapter profiles handle vendor differences. Application execution selects this layer through Application execution and routing. Provider construction and refresh are covered in Provider runtime and discovery. Vendor-specific compatibility behavior is cataloged in Vendor provider adapters, and public wire conversion is covered in Protocol adapters.
Provider boundary
The gateway treats each provider as a source of model responses and model metadata. This boundary keeps request routing independent from credentials, transport shape, model capability data, and vendor failure behavior.
Common provider contract
Every provider supports the gateway's request validation, response streaming, model listing, and cleanup needs. The contract covers Anthropic Messages, Gemini, and OpenAI Responses requests. Native Messages execution is optional for providers that expose that wire format directly, and native Gemini execution is available to providers that expose the GenerateContent protocol. Continuation state can also cross the provider boundary when recovery needs to resume an interrupted response.
| Contract area | Purpose |
|---|---|
| Native Messages execution | Pass through an already prepared Anthropic Messages request when the provider supports that protocol. |
| Messages execution | Turn a gateway request into an Anthropic SSE response stream. |
| Gemini execution | Turn a gateway request into a Gemini response stream when the provider supports that path. |
| Responses execution | Turn a gateway request into OpenAI Responses SSE events. |
| Model listing | Report provider model IDs and known capability metadata. |
| Cleanup | Close provider-owned network and authentication resources. |
References: src/…/base.py• L20-126 src/…/provider.py• L264-374
Provider request flow
The gateway resolves a provider from a model reference, applies that provider's request and capability rules, and then turns the upstream result back into the selected gateway stream. The provider owns this translation, so the rest of the application does not need to know each vendor's request fields or stream markers. A shared attempt loop now orders credential refresh and request correction before admission recovery. It can then restart a clean request, continue delivered content, or return one final provider failure.
This diagram shows the main relationship between gateway requests, the provider boundary, vendor responses, model metadata, and bounded stream recovery.
flowchart LR
A[Gateway request] --> B[Provider boundary]
B --> C[Vendor request]
C --> D[Vendor response stream]
D --> B
B --> E[Gateway response stream]
B --> F[Provider model metadata]
D -. recoverable interruption .-> G[Bounded continuation]
G --> C
References: src/…/base.py• L56-126 src/…/loop.py• L117-206 src/…/loop.py• L216-328
Request-scoped endpoint identity
Provider credentials and endpoint data belong to the request attempt that uses them. A request endpoint supplies one snapshot of the base URL, headers, and account identity. A rejected credential can cause one forced refresh before the response is committed, but it does not mutate a shared client used by other requests.
This ownership model also keeps replay provenance tied to the provider connection that produced it. The layer can use that provenance when it negotiates history or decides whether a partial stream is safe to continue.
References: src/…/endpoint.py• L6-26 src/…/endpoint_types.py• L8-25 src/…/loop.py• L235-272
History and continuation state
Provider history handling keeps persisted reasoning tied to the connection that produced it. Native reasoning records, tool-call data, and other provider state are treated as source evidence rather than inferred from public text alone.
When a stream fails, recovery can restart the request before public content is released. After release, it can continue text or repair a tool call only when the captured source state can be combined with the delivered response. Otherwise the provider returns one canonical failure instead of sending a second incompatible response. History negotiation can also learn a working replay mode for later requests without placing account or transcript data in the shared cache key.
References: src/…/history_negotiation.py• L30-49 src/…/history_negotiation.py• L112-224 src/…/loop.py• L310-328
Transport support
Most integrations use a shared OpenAI-compatible path, but the layer also supports transports with different request, stream, or discovery rules. All paths present the same provider boundary to the gateway.
OpenAI-compatible Chat transport
The main shared transport sends requests to OpenAI-compatible Chat Completions endpoints. A provider profile describes endpoint shape, reasoning fields, tool behavior, model-list response shape, and small request changes that a vendor needs. This supports hosted vendors and local servers without making each one implement a separate gateway protocol.
The profile path covers remote and local integrations such as OpenRouter, Gemini, Mistral, Cohere, Cloudflare, LM Studio, llama.cpp, and Ollama. Profiles can also describe aliases, input types, context limits, output limits, and pagination when a vendor catalog differs from the common shape. Focused behavior hooks handle stateful or unusual cases. Cloudflare, for example, converts numeric text deltas to text and maps its reasoning field into the gateway's thinking output.
The shared path has separate output forms for native Chat, Messages, Gemini, and Responses clients. This lets one upstream stream serve several public protocols while keeping each public lifecycle valid.
References: src/…/__init__.py• L15-59 src/…/provider.py• L43-158 src/…/transport.py• L501-531 src/…/client.py• L73-105
Native Messages transport
The native Messages transport sends Anthropic Messages requests to providers that expose that protocol. It resolves thinking and output controls from the request and known model capabilities, then presents the upstream SSE stream in the gateway's Messages format or converts it for Responses, Chat, and Gemini clients. Native relays preserve upstream block order and signed reasoning metadata instead of adding synthetic replay records.
The transport validates controls before opening a stream. It keeps manual thinking, adaptive thinking, and output effort distinct. It removes a reasoning control and retries only when the upstream explicitly rejects that control before unsafe output is committed. Early output is held for a short recovery window, so a transient cutoff can restart without duplicate public content. Later failures can use a bounded continuation when the captured response state is safe to resume.
The direct Anthropic provider adds API-key authentication, workspace headers, authenticated model discovery, and per-model capability records to this path. Its native passthrough preserves a prepared Messages request while still using the shared admission, framing, failure, and continuation rules. Custom providers reuse the same transport when their endpoint uses the Anthropic Messages format.
References: src/…/provider.py• L105-262 src/…/transport.py• L111-236 src/…/transport.py• L339-627 src/…/custom.py• L95-153
Native Gemini transport
The native Gemini transport sends GenerateContent requests to Google's Gemini API and converts its native parts into the gateway's requested stream. It supports Gemini, Messages, Chat, and Responses ingress by selecting the matching output presenter. Native Gemini clients can receive native parts without losing finish state or usage data.
The stream keeps thought parts, thought signatures, and function calls as replayable source state. It rejects unsupported candidates, grounding data, malformed tool calls, and streams without a finish reason instead of producing an incomplete gateway response. It uses the same admission, bounded correction, early retry, and continuation rules as the other provider paths.
References: src/…/client.py• L54-89 src/…/client.py• L91-140 src/…/client.py• L142-287 src/…/transport.py• L82-134 src/…/transport.py• L135-350
Responses transport
The shared Responses transport sends native OpenAI Responses requests and can also present Messages, Gemini, and Chat requests through the Responses protocol. Presenters keep the public stream format separate from upstream event handling, so a provider can adapt events without changing the gateway contract.
The public OpenAI API provider uses this transport with an API key and model-list discovery. Its correction path removes temperature or top_p only when the API clearly says that the selected model supports only its default value and the field matches the value sent on that attempt. The connected OpenAI/Codex path uses request-scoped account credentials and keeps model discovery tied to the connected account.
References: src/…/transport.py• L110-194 src/…/transport.py• L236-358 src/…/transport.py• L360-710 src/…/provider.py• L31-119 src/…/provider.py• L50-170
Catalog-directed transport
Some providers learn the model route from a vendor catalog. OpenCode uses one public model selector while its catalog chooses the upstream model ID and whether the request uses Chat Completions or Responses. The catalog publishes a complete last-good view rather than a partial set of routes.
This model supports providers with separate generation and discovery paths. The same provider boundary can therefore cover standard model endpoints, account-scoped catalogs, and native vendor catalogs.
References: src/…/provider.py• L81-151 src/…/provider.py• L202-252 src/…/provider.py• L326-421
Custom endpoint transport
Custom providers let an operator describe a compatible endpoint without adding a vendor-specific integration. The provider selects Anthropic Messages, OpenAI Chat Completions, or OpenAI Responses and reuses the matching transport with a configured reasoning mode.
The custom path can advertise fixed model IDs or discover models from the endpoint. Messages endpoints use Messages discovery and capability parsing. OpenAI-shaped endpoints use the shared model client. A custom reasoning mode must represent the caller's request; an exact token budget is rejected when the selected mode supports only named effort or a provider default.
References: src/…/custom.py• L95-219 src/…/custom.py• L203-216 src/…/custom.py• L393-418
Admission and failure policy
The provider layer controls how calls enter a provider and how failures affect later calls. These rules limit load, avoid duplicate recovery storms, and turn varied upstream errors into stable gateway failures.
Provider admission
Each provider has a shared capacity budget and a separate recovery episode for each model. A normal call must pass a rate limit and a concurrency limit. One logical ProviderExecution owns the physical-attempt budget, so a correction, retry, continuation, or tool repair cannot create an unbounded set of calls.
When several calls for the same model see the same transient failure, they share one recovery episode. One call performs the next probe while the others wait. Calls for different credentials can share provider capacity while retaining separate recovery state. Discovery uses the same attempt lifecycle and retains its last provider failure when a deadline cancels retry backoff.
| Control | Effect |
|---|---|
| Rate window | Limits provider attempts over a configured period. |
| Concurrency bulkhead | Limits active calls to one provider. |
| Attempt budget | Bounds retries for one logical operation. |
| Shared recovery episode | Coalesces recovery for one model into one probe and one backoff schedule. |
| Provider capacity | Applies provider-wide rate and concurrency limits across credential controllers. |
Validated limits come from settings. The admission registry shares capacity by provider while keeping controller recovery state separate for each credential identity. Reconfiguration updates live controllers without resetting occupancy or recovery history.
References: src/…/admission_policy.py• L11-33 src/…/admission_registry.py• L14-74 src/…/admission.py• L109-243 src/…/admission.py• L496-595
Failure classification
The layer separates errors that can improve after a retry from errors that need a client or operator change. Rate limits, upstream server failures, timeouts, connection faults, TLS read faults, and selected malformed protocol results are normally retryable. Authentication, permission, billing, invalid request, request-too-large, and context-window failures are final when their meaning is clear.
After recovery is exhausted, the layer maps the retained upstream error to a stable failure kind, status, and user-facing message. Provider behavior can override this meaning when a vendor uses a misleading status or a provider-specific error shape. Canonical meanings include overload and context-window exhaustion. The result hides unsafe upstream details while retaining enough detail for diagnosis.
References: src/…/canonical_failures.py• L7-57 src/…/failure_reporting.py• L22-78 src/…/loop.py• L361-401
Stream recovery
An opened stream has a separate recovery policy from a failed request setup. The provider briefly holds early output so a retryable cutoff can restart without exposing partial output. After public content is committed, the layer can continue generated text or repair an incomplete tool call only when the captured source state is safe to resume. If it is not safe, the layer closes open response blocks and returns the canonical failure.
The attempt loop applies pure stream rules after it handles repairs and admission. It distinguishes a failure before an attempt, a failure on an open attempt, and a failure after delivery. A truncated provider stream is retryable only when its protocol requires a terminal marker and the attempt budget and delivery state still permit a safe next step.
This diagram shows how stream state changes the result of a transient failure.
flowchart TD
A[Provider stream starts] --> B{Transient failure?}
B -- No --> C[Map final failure]
B -- Yes --> D{Output committed?}
D -- No --> E[Retry within attempt budget]
D -- Yes --> F{Recoverable source state?}
F -- Yes --> G[Continue text or repair tools]
F -- No --> C
E --> H[Complete stream]
G --> H
References: src/…/decision.py• L10-123 src/…/loop.py• L274-328 src/…/recovery_errors.py• L4-35 src/…/http.py• L70-133
Request correction policy
Request correction is a bounded change to one failed attempt. The layer can refresh authentication or remove a provider field only before public content is committed and only when the error identifies a safe correction. This keeps a late refresh or body downgrade from replaying content that the client already received.
Common corrections include history representation, reasoning controls, stream options, output caps, and provider-specific request fields. Each correction has a limited retry kind, and the corrected body is separate from the original request. The corrector tries a rejected history record first, then one-use provider steps, and only then moves to another history mode for a direct HTTP 400.
References: src/…/corrector.py• L25-88 src/…/loop.py• L235-272 src/…/loop.py• L416-422
Reasoning compatibility
Reasoning compatibility keeps the gateway's computation intent separate from each vendor's wire fields. The layer uses model metadata when it can, and it reshapes or removes a reasoning control only after a recognized provider response proves that the sent field is not supported.
This prevents one provider error from causing broad guesses about model identity. Focused adapters can learn accepted values, map a budget to a vendor field, preserve native signatures, or reject an intent that the account model does not advertise. The shared transport still owns the retry budget and public result.
References: src/…/client.py• L61-98 src/…/reasoning.py• L26-92 src/…/transport.py• L959-1001 src/…/transport.py• L202-225
Model listing
Model listing turns vendor catalogs into the gateway's usable model set. The layer validates the response shape, filters models that cannot serve gateway requests, keeps capability data when the vendor exposes it, and supplies the result to routing and model-list responses.
Model metadata
Each discovered model has a provider model ID and optional capability metadata. The metadata can state reasoning support, accepted input types, context-window size, and maximum output size. An unknown value preserves uncertainty when the vendor does not report the capability.
Common OpenAI-shaped lists and vendor-specific catalogs follow the same goal: advertise only usable model records. Anthropic discovery uses authenticated cursors and retains thinking and token-limit capabilities. Gemini keeps models that support GenerateContent and retains native thinking and token limits. OpenAI-compatible profiles can apply provider filters, aliases, tags, required fields, and bounded pagination. Custom endpoints either publish fixed model IDs or use the discovery format of their selected wire protocol.
References: src/…/provider.py• L142-192 src/…/client.py• L91-140 src/…/provider.py• L100-213 src/…/custom.py• L393-418 src/…/client.py• L144-174
Gateway model names
The gateway keeps raw model IDs under their provider and exposes names in the form provider_id/model_id. This prevents two vendors with the same raw model name from colliding. The API layer can encode these references into gateway-discoverable IDs and can add a no-thinking variant when a client must disable reasoning.
Discovery is limited to providers that are valid for the current settings, connected accounts, or custom endpoint definitions. Optional local endpoints do not make the whole refresh fail when they are unused. Discovery errors become safe, stable messages rather than exposing upstream URLs or raw error text. For construction, refresh, and runtime lifecycle, see Provider runtime and discovery.
References: src/…/provider.py• L144-213 src/…/client.py• L91-140 src/…/provider.py• L149-151
Vendor integration catalog
The provider catalog is the source of truth for supported provider IDs, display names, authentication style, default endpoints, local status, and provider-specific settings. Construction gives each catalog entry one owner: a shared OpenAI-compatible profile, a focused special adapter, a custom endpoint definition, or a connected-account provider.
Hosted model providers
The hosted catalog spans direct vendor APIs, model gateways, and hosted inference services. The gateway presents them through the same model and stream boundary even when their credentials, endpoint paths, model filters, or reasoning fields differ.
| Group | Providers |
|---|---|
| Direct vendor APIs | Anthropic, OpenAI API, NVIDIA NIM, Groq, xAI, QwenCloud Token Plan, DeepInfra, SiliconFlow, Nebius, Chutes, Featherless AI, Azure OpenAI, Gemini, Google Vertex AI, DeepSeek, Mistral, Mistral Codestral, Amazon Bedrock, Hugging Face, Cohere, MiniMax, Cerebras, SambaNova, Fireworks, Novita AI, Cloudflare, Z.ai Coding Plan, Z.ai API, Poolside AI |
| Connected or subscription access | OpenAI / ChatGPT, GitHub Copilot, ClinePass, Kimi Code, QwenCloud Coding, OpenCode Zen, OpenCode Go |
| Model gateways and hosted access | OpenRouter, Together AI, Vercel AI Gateway, TokenRouter, NaraRoute, ZenMux, Kilo.ai, Wafer, Agnes AI, W&B Inference |
| User-defined endpoints | Custom providers using Anthropic Messages, OpenAI Chat Completions, or OpenAI Responses |
Entries with separate quotas or endpoints are separate providers. Examples include Mistral and Codestral, QwenCloud Token Plan and Coding Plan, and Z.ai Coding Plan and API.
References: src/…/__init__.py• L41-59 src/…/provider.py• L68-119 src/…/custom.py• L95-153
Local model providers
Local providers connect the gateway to model servers on the user's machine rather than to a hosted vendor. LM Studio, llama.cpp, and Ollama use OpenAI-compatible Chat transport with local default endpoints. Their model visibility depends on the local service and on whether the provider is relevant to the current configuration.
References: src/…/__init__.py• L41-59
Connected account providers
Connected account providers use sign-in state and a short-lived account lease instead of a static inference key. GitHub Copilot discovers models for the current account and dispatches each request to Messages, Chat Completions, or Responses according to the account's current model data.
The lease keeps account identity, model choice, endpoint, and upstream protocol together for one request. Account loss or a changed egress becomes a stable authentication or availability failure. Cleanup waits for active leases and streams before it closes shared account resources.
References: src/…/provider.py• L74-140 src/…/provider.py• L247-407 src/…/provider.py• L50-170
Compatibility adapters
Most providers need only declarative compatibility data. Focused adapters cover stateful authentication, custom model catalogs, native stream conversion, special request fields, vendor error semantics, custom endpoint formats, and provider-owned auxiliary transports. The adapter catalog explains these differences in more detail: Vendor provider adapters.
Provider construction is checked at startup. Every catalog ID must have exactly one owner: a profile, a special factory, or an injected connected-account implementation. The direct Anthropic and OpenAI API providers are special factories because they own native protocol or public-key-specific behavior. Custom providers are created from their endpoint definition and must select one supported wire format.
References: src/…/__init__.py• L41-59 src/…/provider.py• L105-154 src/…/provider.py• L68-119 src/…/custom.py• L95-219
Change checklist
When these files or sections are changed, remember to consider these:
- Keep provider catalog entries and construction owners in one-to-one agreement. A mismatch can make a provider visible but impossible to construct.
- When a model-list response changes, update its filter and capability rules with the gateway model naming assumptions.
- When a provider changes its endpoint or authentication mode, check both generation and discovery. These paths can use different URLs or credentials.
- When retry or failure meanings change, preserve the distinction between retryable upstream faults and final client faults.
- When stream parsing changes, check the early holdback and public commitment boundary. Emitting partial SSE too early can make a later retry duplicate output.
- When a connected-account model can change egress during refresh, keep the model lease and endpoint snapshot tied to the same account revision.
- When an endpoint refresh changes, keep the forced refresh pending until the next attempt resolves a new snapshot. Do not mutate credentials on a shared client.
- When a vendor changes its reasoning fields or error shape, keep correction bounded to a recognized model or payload.
- When a custom provider changes its reasoning format or wire protocol, keep its definition aligned with transport and model discovery.
- When native Messages or Gemini replay changes, preserve signed reasoning, tool metadata, and source evidence for later history and continuation.
- When OpenAI Responses request correction changes, remove only a field named by the upstream error and unchanged on the sent attempt.
- When admission limits change, update validated settings and runtime capacity together so current occupancy and recovery state remain intact.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| A catalog entry has no matching profile, special factory, or connected implementation | The provider is advertised but cannot be constructed | Startup construction checks require one owner for each provider ID |
| A model catalog has an invalid shape, repeated pagination, or no usable model IDs | The provider's models are not published to routing | Model discovery validates records, page progress, and usable IDs |
| Several calls for one provider model fail at the same time | Independent retries amplify load and rate limits | Admission coalesces them into one recovery episode and one probe |
| A request correction runs after public content is committed | The client can receive duplicate or inconsistent output | The attempt loop allows refresh and correction only before commitment |
| A stream fails after native reasoning or hosted state was delivered | A continuation can produce a response that cannot be joined to the first one | Source recovery tracks native evidence and blocks unsafe handoff |
| A native Messages stream ends without its terminal event | The client receives an incomplete response | Messages framing requires a valid stop event before success |
| A native Gemini stream has multiple candidates, unsupported parts, or no finish reason | The gateway can return misleading or incomplete output | Gemini stream validation requires one supported candidate and a terminal reason |
| A vendor rejects a reasoning field with an unrelated error | A broad retry can remove valid caller intent or hide the real failure | Compatibility adapters match the recognized field and error shape |
| A custom endpoint uses a wire format that cannot represent its reasoning mode | Requests fail late or silently change the requested computation | Custom definitions select a supported transport and reject unrepresentable budgets |
| A connected account changes while a request lease is active | The request can use stale credentials or the wrong upstream route | Account leases check identity, model, endpoint, and egress before dispatch |
| A local endpoint is absent during an unused discovery pass | An irrelevant local service can block model refresh | Optional local discovery treats connection and timeout failures as non-blocking |
Key files
| File | Why to read it |
|---|---|
| src/…/base.py | Defines the provider boundary for Messages, Gemini, Chat, and Responses requests. |
| src/…/loop.py | Shows the shared order for correction, admission, retry, stream recovery, and final failure. |
| src/…/admission.py | Coordinates provider capacity, attempt budgets, and shared recovery episodes. |
| src/…/admission_registry.py | Shows how provider capacity and credential-specific recovery controllers are retained. |
| src/…/canonical_failures.py | Defines stable overload and context-window failure meanings and provider override hooks. |
| src/…/failure_reporting.py | Maps exhausted provider errors to safe client-facing failures. |
| src/…/corrector.py | Defines the bounded order for history, reasoning, and provider request corrections. |
| src/…/decision.py | Defines when a failed stream can restart, recover, or fail. |
| src/…/endpoint.py | Keeps endpoint refresh request-scoped and pending until the next attempt resolves it. |
| src/…/provider.py | Shows direct Anthropic credentials, discovery, capability records, and native Messages access. |
| src/…/transport.py | Shows shared Messages translation, stream framing, correction, and recovery. |
| src/…/client.py | Shows native Gemini discovery and conversion from all gateway request protocols. |
| src/…/transport.py | Shows native GenerateContent execution with shared admission and recovery. |
| src/…/__init__.py | Exposes the profile-driven OpenAI-compatible provider family. |
| src/…/provider.py | Shows shared Chat provider ownership and model discovery. |
| src/…/transport.py | Shows Chat request conversion, stream presentation, correction, and recovery. |
| src/…/transport.py | Shows shared Responses execution, event presentation, correction, and recovery. |
| src/…/provider.py | Illustrates public-key OpenAI access and narrow sampling correction. |
| src/…/provider.py | Illustrates connected OpenAI account discovery and request-scoped credentials. |
| src/…/provider.py | Shows account leases and dispatch across Copilot's three upstream egresses. |
| src/…/client.py | Shows a focused Cloudflare adapter for account-scoped discovery and unusual Chat deltas. |
| src/…/custom.py | Shows how user-defined endpoints reuse the shared Messages, Chat, and Responses transports. |
| src/…/provider.py | Shows catalog-driven model routing between Chat and Responses. |