Introducing a new way to review your PRs.

Read the announcement

alishahryar1 / free-claude-code

Use Claude Code, Codex, Antigravity, VSCode, Pi, and OpenCode (and 6 other harnesses) for free (1.3B+ free tokens) from your terminal, app, IDE, or phone, and now from the browser with native browser sessions (multi-harness + multi-model) like OpenClaw (voice supported + ToS friendly)

View on GitHub

28 documents · Updated Nov 20, 2026

docs / provider-integrations

Provider integration layer

The provider integration layer gives the gateway one boundary for many model vendors. It accepts gateway request shapes, sends them through vendor transports, streams gateway response formats, and reports provider model metadata. Shared admission and failure rules protect each provider while adapter profiles handle vendor differences. Application execution selects this layer through Application execution and routing. Provider construction and refresh are covered in Provider runtime and discovery. Vendor-specific compatibility behavior is cataloged in Vendor provider adapters, and public wire conversion is covered in Protocol adapters.

Provider boundary

The gateway treats each provider as a source of model responses and model metadata. This boundary keeps request routing independent from credentials, transport shape, model capability data, and vendor failure behavior.

Common provider contract

Every provider supports the gateway's request validation, response streaming, model listing, and cleanup needs. The contract covers Anthropic Messages, Gemini, and OpenAI Responses requests. Native Messages execution is optional for providers that expose that wire format directly, and native Gemini execution is available to providers that expose the GenerateContent protocol. Continuation state can also cross the provider boundary when recovery needs to resume an interrupted response.

Contract areaPurpose
Native Messages executionPass through an already prepared Anthropic Messages request when the provider supports that protocol.
Messages executionTurn a gateway request into an Anthropic SSE response stream.
Gemini executionTurn a gateway request into a Gemini response stream when the provider supports that path.
Responses executionTurn a gateway request into OpenAI Responses SSE events.
Model listingReport provider model IDs and known capability metadata.
CleanupClose provider-owned network and authentication resources.

References: src/…/base.py• L20-126 src/…/provider.py• L264-374

Provider request flow

The gateway resolves a provider from a model reference, applies that provider's request and capability rules, and then turns the upstream result back into the selected gateway stream. The provider owns this translation, so the rest of the application does not need to know each vendor's request fields or stream markers. A shared attempt loop now orders credential refresh and request correction before admission recovery. It can then restart a clean request, continue delivered content, or return one final provider failure.

This diagram shows the main relationship between gateway requests, the provider boundary, vendor responses, model metadata, and bounded stream recovery.

flowchart LR
    A[Gateway request] --> B[Provider boundary]
    B --> C[Vendor request]
    C --> D[Vendor response stream]
    D --> B
    B --> E[Gateway response stream]
    B --> F[Provider model metadata]
    D -. recoverable interruption .-> G[Bounded continuation]
    G --> C

References: src/…/base.py• L56-126 src/…/loop.py• L117-206 src/…/loop.py• L216-328

Request-scoped endpoint identity

Provider credentials and endpoint data belong to the request attempt that uses them. A request endpoint supplies one snapshot of the base URL, headers, and account identity. A rejected credential can cause one forced refresh before the response is committed, but it does not mutate a shared client used by other requests.

This ownership model also keeps replay provenance tied to the provider connection that produced it. The layer can use that provenance when it negotiates history or decides whether a partial stream is safe to continue.

References: src/…/endpoint.py• L6-26 src/…/endpoint_types.py• L8-25 src/…/loop.py• L235-272

History and continuation state

Provider history handling keeps persisted reasoning tied to the connection that produced it. Native reasoning records, tool-call data, and other provider state are treated as source evidence rather than inferred from public text alone.

When a stream fails, recovery can restart the request before public content is released. After release, it can continue text or repair a tool call only when the captured source state can be combined with the delivered response. Otherwise the provider returns one canonical failure instead of sending a second incompatible response. History negotiation can also learn a working replay mode for later requests without placing account or transcript data in the shared cache key.

References: src/…/history_negotiation.py• L30-49 src/…/history_negotiation.py• L112-224 src/…/loop.py• L310-328

Transport support

Most integrations use a shared OpenAI-compatible path, but the layer also supports transports with different request, stream, or discovery rules. All paths present the same provider boundary to the gateway.

OpenAI-compatible Chat transport

The main shared transport sends requests to OpenAI-compatible Chat Completions endpoints. A provider profile describes endpoint shape, reasoning fields, tool behavior, model-list response shape, and small request changes that a vendor needs. This supports hosted vendors and local servers without making each one implement a separate gateway protocol.

The profile path covers remote and local integrations such as OpenRouter, Gemini, Mistral, Cohere, Cloudflare, LM Studio, llama.cpp, and Ollama. Profiles can also describe aliases, input types, context limits, output limits, and pagination when a vendor catalog differs from the common shape. Focused behavior hooks handle stateful or unusual cases. Cloudflare, for example, converts numeric text deltas to text and maps its reasoning field into the gateway's thinking output.

The shared path has separate output forms for native Chat, Messages, Gemini, and Responses clients. This lets one upstream stream serve several public protocols while keeping each public lifecycle valid.

References: src/…/__init__.py• L15-59 src/…/provider.py• L43-158 src/…/transport.py• L501-531 src/…/client.py• L73-105

Native Messages transport

The native Messages transport sends Anthropic Messages requests to providers that expose that protocol. It resolves thinking and output controls from the request and known model capabilities, then presents the upstream SSE stream in the gateway's Messages format or converts it for Responses, Chat, and Gemini clients. Native relays preserve upstream block order and signed reasoning metadata instead of adding synthetic replay records.

The transport validates controls before opening a stream. It keeps manual thinking, adaptive thinking, and output effort distinct. It removes a reasoning control and retries only when the upstream explicitly rejects that control before unsafe output is committed. Early output is held for a short recovery window, so a transient cutoff can restart without duplicate public content. Later failures can use a bounded continuation when the captured response state is safe to resume.

The direct Anthropic provider adds API-key authentication, workspace headers, authenticated model discovery, and per-model capability records to this path. Its native passthrough preserves a prepared Messages request while still using the shared admission, framing, failure, and continuation rules. Custom providers reuse the same transport when their endpoint uses the Anthropic Messages format.

References: src/…/provider.py• L105-262 src/…/transport.py• L111-236 src/…/transport.py• L339-627 src/…/custom.py• L95-153

Native Gemini transport

The native Gemini transport sends GenerateContent requests to Google's Gemini API and converts its native parts into the gateway's requested stream. It supports Gemini, Messages, Chat, and Responses ingress by selecting the matching output presenter. Native Gemini clients can receive native parts without losing finish state or usage data.

The stream keeps thought parts, thought signatures, and function calls as replayable source state. It rejects unsupported candidates, grounding data, malformed tool calls, and streams without a finish reason instead of producing an incomplete gateway response. It uses the same admission, bounded correction, early retry, and continuation rules as the other provider paths.

References: src/…/client.py• L54-89 src/…/client.py• L91-140 src/…/client.py• L142-287 src/…/transport.py• L82-134 src/…/transport.py• L135-350

Responses transport

The shared Responses transport sends native OpenAI Responses requests and can also present Messages, Gemini, and Chat requests through the Responses protocol. Presenters keep the public stream format separate from upstream event handling, so a provider can adapt events without changing the gateway contract.

The public OpenAI API provider uses this transport with an API key and model-list discovery. Its correction path removes temperature or top_p only when the API clearly says that the selected model supports only its default value and the field matches the value sent on that attempt. The connected OpenAI/Codex path uses request-scoped account credentials and keeps model discovery tied to the connected account.

References: src/…/transport.py• L110-194 src/…/transport.py• L236-358 src/…/transport.py• L360-710 src/…/provider.py• L31-119 src/…/provider.py• L50-170

Catalog-directed transport

Some providers learn the model route from a vendor catalog. OpenCode uses one public model selector while its catalog chooses the upstream model ID and whether the request uses Chat Completions or Responses. The catalog publishes a complete last-good view rather than a partial set of routes.

This model supports providers with separate generation and discovery paths. The same provider boundary can therefore cover standard model endpoints, account-scoped catalogs, and native vendor catalogs.

References: src/…/provider.py• L81-151 src/…/provider.py• L202-252 src/…/provider.py• L326-421

Custom endpoint transport

Custom providers let an operator describe a compatible endpoint without adding a vendor-specific integration. The provider selects Anthropic Messages, OpenAI Chat Completions, or OpenAI Responses and reuses the matching transport with a configured reasoning mode.

The custom path can advertise fixed model IDs or discover models from the endpoint. Messages endpoints use Messages discovery and capability parsing. OpenAI-shaped endpoints use the shared model client. A custom reasoning mode must represent the caller's request; an exact token budget is rejected when the selected mode supports only named effort or a provider default.

References: src/…/custom.py• L95-219 src/…/custom.py• L203-216 src/…/custom.py• L393-418

Admission and failure policy

The provider layer controls how calls enter a provider and how failures affect later calls. These rules limit load, avoid duplicate recovery storms, and turn varied upstream errors into stable gateway failures.

Provider admission

Each provider has a shared capacity budget and a separate recovery episode for each model. A normal call must pass a rate limit and a concurrency limit. One logical ProviderExecution owns the physical-attempt budget, so a correction, retry, continuation, or tool repair cannot create an unbounded set of calls.

When several calls for the same model see the same transient failure, they share one recovery episode. One call performs the next probe while the others wait. Calls for different credentials can share provider capacity while retaining separate recovery state. Discovery uses the same attempt lifecycle and retains its last provider failure when a deadline cancels retry backoff.

ControlEffect
Rate windowLimits provider attempts over a configured period.
Concurrency bulkheadLimits active calls to one provider.
Attempt budgetBounds retries for one logical operation.
Shared recovery episodeCoalesces recovery for one model into one probe and one backoff schedule.
Provider capacityApplies provider-wide rate and concurrency limits across credential controllers.

Validated limits come from settings. The admission registry shares capacity by provider while keeping controller recovery state separate for each credential identity. Reconfiguration updates live controllers without resetting occupancy or recovery history.

References: src/…/admission_policy.py• L11-33 src/…/admission_registry.py• L14-74 src/…/admission.py• L109-243 src/…/admission.py• L496-595

Failure classification

The layer separates errors that can improve after a retry from errors that need a client or operator change. Rate limits, upstream server failures, timeouts, connection faults, TLS read faults, and selected malformed protocol results are normally retryable. Authentication, permission, billing, invalid request, request-too-large, and context-window failures are final when their meaning is clear.

After recovery is exhausted, the layer maps the retained upstream error to a stable failure kind, status, and user-facing message. Provider behavior can override this meaning when a vendor uses a misleading status or a provider-specific error shape. Canonical meanings include overload and context-window exhaustion. The result hides unsafe upstream details while retaining enough detail for diagnosis.

References: src/…/canonical_failures.py• L7-57 src/…/failure_reporting.py• L22-78 src/…/loop.py• L361-401

Stream recovery

An opened stream has a separate recovery policy from a failed request setup. The provider briefly holds early output so a retryable cutoff can restart without exposing partial output. After public content is committed, the layer can continue generated text or repair an incomplete tool call only when the captured source state is safe to resume. If it is not safe, the layer closes open response blocks and returns the canonical failure.

The attempt loop applies pure stream rules after it handles repairs and admission. It distinguishes a failure before an attempt, a failure on an open attempt, and a failure after delivery. A truncated provider stream is retryable only when its protocol requires a terminal marker and the attempt budget and delivery state still permit a safe next step.

This diagram shows how stream state changes the result of a transient failure.

flowchart TD
    A[Provider stream starts] --> B{Transient failure?}
    B -- No --> C[Map final failure]
    B -- Yes --> D{Output committed?}
    D -- No --> E[Retry within attempt budget]
    D -- Yes --> F{Recoverable source state?}
    F -- Yes --> G[Continue text or repair tools]
    F -- No --> C
    E --> H[Complete stream]
    G --> H

References: src/…/decision.py• L10-123 src/…/loop.py• L274-328 src/…/recovery_errors.py• L4-35 src/…/http.py• L70-133

Request correction policy

Request correction is a bounded change to one failed attempt. The layer can refresh authentication or remove a provider field only before public content is committed and only when the error identifies a safe correction. This keeps a late refresh or body downgrade from replaying content that the client already received.

Common corrections include history representation, reasoning controls, stream options, output caps, and provider-specific request fields. Each correction has a limited retry kind, and the corrected body is separate from the original request. The corrector tries a rejected history record first, then one-use provider steps, and only then moves to another history mode for a direct HTTP 400.

References: src/…/corrector.py• L25-88 src/…/loop.py• L235-272 src/…/loop.py• L416-422

Reasoning compatibility

Reasoning compatibility keeps the gateway's computation intent separate from each vendor's wire fields. The layer uses model metadata when it can, and it reshapes or removes a reasoning control only after a recognized provider response proves that the sent field is not supported.

This prevents one provider error from causing broad guesses about model identity. Focused adapters can learn accepted values, map a budget to a vendor field, preserve native signatures, or reject an intent that the account model does not advertise. The shared transport still owns the retry budget and public result.

References: src/…/client.py• L61-98 src/…/reasoning.py• L26-92 src/…/transport.py• L959-1001 src/…/transport.py• L202-225

Model listing

Model listing turns vendor catalogs into the gateway's usable model set. The layer validates the response shape, filters models that cannot serve gateway requests, keeps capability data when the vendor exposes it, and supplies the result to routing and model-list responses.

Model metadata

Each discovered model has a provider model ID and optional capability metadata. The metadata can state reasoning support, accepted input types, context-window size, and maximum output size. An unknown value preserves uncertainty when the vendor does not report the capability.

Common OpenAI-shaped lists and vendor-specific catalogs follow the same goal: advertise only usable model records. Anthropic discovery uses authenticated cursors and retains thinking and token-limit capabilities. Gemini keeps models that support GenerateContent and retains native thinking and token limits. OpenAI-compatible profiles can apply provider filters, aliases, tags, required fields, and bounded pagination. Custom endpoints either publish fixed model IDs or use the discovery format of their selected wire protocol.

References: src/…/provider.py• L142-192 src/…/client.py• L91-140 src/…/provider.py• L100-213 src/…/custom.py• L393-418 src/…/client.py• L144-174

Gateway model names

The gateway keeps raw model IDs under their provider and exposes names in the form provider_id/model_id. This prevents two vendors with the same raw model name from colliding. The API layer can encode these references into gateway-discoverable IDs and can add a no-thinking variant when a client must disable reasoning.

Discovery is limited to providers that are valid for the current settings, connected accounts, or custom endpoint definitions. Optional local endpoints do not make the whole refresh fail when they are unused. Discovery errors become safe, stable messages rather than exposing upstream URLs or raw error text. For construction, refresh, and runtime lifecycle, see Provider runtime and discovery.

References: src/…/provider.py• L144-213 src/…/client.py• L91-140 src/…/provider.py• L149-151

Vendor integration catalog

The provider catalog is the source of truth for supported provider IDs, display names, authentication style, default endpoints, local status, and provider-specific settings. Construction gives each catalog entry one owner: a shared OpenAI-compatible profile, a focused special adapter, a custom endpoint definition, or a connected-account provider.

Hosted model providers

The hosted catalog spans direct vendor APIs, model gateways, and hosted inference services. The gateway presents them through the same model and stream boundary even when their credentials, endpoint paths, model filters, or reasoning fields differ.

GroupProviders
Direct vendor APIsAnthropic, OpenAI API, NVIDIA NIM, Groq, xAI, QwenCloud Token Plan, DeepInfra, SiliconFlow, Nebius, Chutes, Featherless AI, Azure OpenAI, Gemini, Google Vertex AI, DeepSeek, Mistral, Mistral Codestral, Amazon Bedrock, Hugging Face, Cohere, MiniMax, Cerebras, SambaNova, Fireworks, Novita AI, Cloudflare, Z.ai Coding Plan, Z.ai API, Poolside AI
Connected or subscription accessOpenAI / ChatGPT, GitHub Copilot, ClinePass, Kimi Code, QwenCloud Coding, OpenCode Zen, OpenCode Go
Model gateways and hosted accessOpenRouter, Together AI, Vercel AI Gateway, TokenRouter, NaraRoute, ZenMux, Kilo.ai, Wafer, Agnes AI, W&B Inference
User-defined endpointsCustom providers using Anthropic Messages, OpenAI Chat Completions, or OpenAI Responses

Entries with separate quotas or endpoints are separate providers. Examples include Mistral and Codestral, QwenCloud Token Plan and Coding Plan, and Z.ai Coding Plan and API.

References: src/…/__init__.py• L41-59 src/…/provider.py• L68-119 src/…/custom.py• L95-153

Local model providers

Local providers connect the gateway to model servers on the user's machine rather than to a hosted vendor. LM Studio, llama.cpp, and Ollama use OpenAI-compatible Chat transport with local default endpoints. Their model visibility depends on the local service and on whether the provider is relevant to the current configuration.

References: src/…/__init__.py• L41-59

Connected account providers

Connected account providers use sign-in state and a short-lived account lease instead of a static inference key. GitHub Copilot discovers models for the current account and dispatches each request to Messages, Chat Completions, or Responses according to the account's current model data.

The lease keeps account identity, model choice, endpoint, and upstream protocol together for one request. Account loss or a changed egress becomes a stable authentication or availability failure. Cleanup waits for active leases and streams before it closes shared account resources.

References: src/…/provider.py• L74-140 src/…/provider.py• L247-407 src/…/provider.py• L50-170

Compatibility adapters

Most providers need only declarative compatibility data. Focused adapters cover stateful authentication, custom model catalogs, native stream conversion, special request fields, vendor error semantics, custom endpoint formats, and provider-owned auxiliary transports. The adapter catalog explains these differences in more detail: Vendor provider adapters.

Provider construction is checked at startup. Every catalog ID must have exactly one owner: a profile, a special factory, or an injected connected-account implementation. The direct Anthropic and OpenAI API providers are special factories because they own native protocol or public-key-specific behavior. Custom providers are created from their endpoint definition and must select one supported wire format.

References: src/…/__init__.py• L41-59 src/…/provider.py• L105-154 src/…/provider.py• L68-119 src/…/custom.py• L95-219

Change checklist

When these files or sections are changed, remember to consider these:

  • Keep provider catalog entries and construction owners in one-to-one agreement. A mismatch can make a provider visible but impossible to construct.
  • When a model-list response changes, update its filter and capability rules with the gateway model naming assumptions.
  • When a provider changes its endpoint or authentication mode, check both generation and discovery. These paths can use different URLs or credentials.
  • When retry or failure meanings change, preserve the distinction between retryable upstream faults and final client faults.
  • When stream parsing changes, check the early holdback and public commitment boundary. Emitting partial SSE too early can make a later retry duplicate output.
  • When a connected-account model can change egress during refresh, keep the model lease and endpoint snapshot tied to the same account revision.
  • When an endpoint refresh changes, keep the forced refresh pending until the next attempt resolves a new snapshot. Do not mutate credentials on a shared client.
  • When a vendor changes its reasoning fields or error shape, keep correction bounded to a recognized model or payload.
  • When a custom provider changes its reasoning format or wire protocol, keep its definition aligned with transport and model discovery.
  • When native Messages or Gemini replay changes, preserve signed reasoning, tool metadata, and source evidence for later history and continuation.
  • When OpenAI Responses request correction changes, remove only a field named by the upstream error and unchanged on the sent attempt.
  • When admission limits change, update validated settings and runtime capacity together so current occupancy and recovery state remain intact.

Important failure modes

TriggerConsequenceGuard
A catalog entry has no matching profile, special factory, or connected implementationThe provider is advertised but cannot be constructedStartup construction checks require one owner for each provider ID
A model catalog has an invalid shape, repeated pagination, or no usable model IDsThe provider's models are not published to routingModel discovery validates records, page progress, and usable IDs
Several calls for one provider model fail at the same timeIndependent retries amplify load and rate limitsAdmission coalesces them into one recovery episode and one probe
A request correction runs after public content is committedThe client can receive duplicate or inconsistent outputThe attempt loop allows refresh and correction only before commitment
A stream fails after native reasoning or hosted state was deliveredA continuation can produce a response that cannot be joined to the first oneSource recovery tracks native evidence and blocks unsafe handoff
A native Messages stream ends without its terminal eventThe client receives an incomplete responseMessages framing requires a valid stop event before success
A native Gemini stream has multiple candidates, unsupported parts, or no finish reasonThe gateway can return misleading or incomplete outputGemini stream validation requires one supported candidate and a terminal reason
A vendor rejects a reasoning field with an unrelated errorA broad retry can remove valid caller intent or hide the real failureCompatibility adapters match the recognized field and error shape
A custom endpoint uses a wire format that cannot represent its reasoning modeRequests fail late or silently change the requested computationCustom definitions select a supported transport and reject unrepresentable budgets
A connected account changes while a request lease is activeThe request can use stale credentials or the wrong upstream routeAccount leases check identity, model, endpoint, and egress before dispatch
A local endpoint is absent during an unused discovery passAn irrelevant local service can block model refreshOptional local discovery treats connection and timeout failures as non-blocking

Key files

FileWhy to read it
src/…/base.pyDefines the provider boundary for Messages, Gemini, Chat, and Responses requests.
src/…/loop.pyShows the shared order for correction, admission, retry, stream recovery, and final failure.
src/…/admission.pyCoordinates provider capacity, attempt budgets, and shared recovery episodes.
src/…/admission_registry.pyShows how provider capacity and credential-specific recovery controllers are retained.
src/…/canonical_failures.pyDefines stable overload and context-window failure meanings and provider override hooks.
src/…/failure_reporting.pyMaps exhausted provider errors to safe client-facing failures.
src/…/corrector.pyDefines the bounded order for history, reasoning, and provider request corrections.
src/…/decision.pyDefines when a failed stream can restart, recover, or fail.
src/…/endpoint.pyKeeps endpoint refresh request-scoped and pending until the next attempt resolves it.
src/…/provider.pyShows direct Anthropic credentials, discovery, capability records, and native Messages access.
src/…/transport.pyShows shared Messages translation, stream framing, correction, and recovery.
src/…/client.pyShows native Gemini discovery and conversion from all gateway request protocols.
src/…/transport.pyShows native GenerateContent execution with shared admission and recovery.
src/…/__init__.pyExposes the profile-driven OpenAI-compatible provider family.
src/…/provider.pyShows shared Chat provider ownership and model discovery.
src/…/transport.pyShows Chat request conversion, stream presentation, correction, and recovery.
src/…/transport.pyShows shared Responses execution, event presentation, correction, and recovery.
src/…/provider.pyIllustrates public-key OpenAI access and narrow sampling correction.
src/…/provider.pyIllustrates connected OpenAI account discovery and request-scoped credentials.
src/…/provider.pyShows account leases and dispatch across Copilot's three upstream egresses.
src/…/client.pyShows a focused Cloudflare adapter for account-scoped discovery and unusual Chat deltas.
src/…/custom.pyShows how user-defined endpoints reuse the shared Messages, Chat, and Responses transports.
src/…/provider.pyShows catalog-driven model routing between Chat and Responses.