Provider runtime and discovery
Provider runtime and discovery give the application one managed set of model providers. The system turns catalog metadata and validated settings into provider clients, finds the models that each usable provider offers, keeps complete model data ready for requests, and replaces or closes provider resources without interrupting work that already started. See Provider integrations for the request-facing provider contract and Provider adapter catalog for the provider-specific adapter map.
Provider catalog
The provider catalog is the shared source of truth for provider identity and availability. It links each provider ID to its display name, access kind, endpoint defaults, required settings, proxy setting, and local or remote status without depending on an adapter.
Provider identity and access boundaries
The catalog separates three access boundaries. Remote providers normally need a configured credential, local providers use fixed local identities, and connected-account providers become usable when an account manager reports a connection. The catalog order is also the canonical order for user-facing provider choices and error messages.
| Access boundary | Meaning | Current examples |
|---|---|---|
| Remote configuration | A user supplies a key and, for some providers, an endpoint or account value. | Azure OpenAI, Cloudflare, Gemini |
| Local service | The application targets a local endpoint and does not need a user key. | LM Studio, llama.cpp, Ollama |
| Connected account | A provider-specific sign-in flow supplies access outside normal API-key fields. | OpenAI / ChatGPT |
References: src/…/provider_catalog.py• L90-122 src/…/provider_catalog.py• L125-168 src/…/provider_catalog.py• L550-575 src/…/provider_catalog.py• L578-589
Configuration surface
Provider settings use the same catalog metadata to define the Admin UI fields and to decide whether a provider is configured. This keeps labels, environment-style keys, settings fields, required multi-field configuration, endpoint overrides, and proxy values aligned with runtime behavior.
A provider can be configured by a key, by a required setting such as a project or account ID, or by a fixed local credential. A provider with a missing required value remains in the catalog but is not part of the usable configured set. The Admin UI reports this state from the same boundaries instead of maintaining a separate provider list.
References: src/…/provider_manifest.py• L296-305 src/…/provider_manifest.py• L308-362 src/…/provider_manifest.py• L365-408 src/…/config.py• L11-52
Provider construction
Provider construction is the bridge from neutral catalog data to a provider adapter. It first resolves a complete shared configuration, then gives the provider its own request behavior and one admission controller for rate, concurrency, and recovery limits. The runtime package exposes this construction path together with the provider runtime used by the application.
Configuration resolution
The configuration step reads a descriptor and a validated settings snapshot. It selects the primary credential or a requested fallback credential from the configured settings, or uses a static local value when the descriptor provides one. It then chooses an explicit base URL over the catalog default, checks all required settings, and carries shared limits, timeouts, proxy data, and safe logging choices into an immutable provider configuration.
Missing credentials, including a requested fallback key, required endpoint settings, or other required provider settings raise a user-facing availability error. This fails before the provider client is created, so an unusable provider does not enter the runtime cache as if it were ready. Custom providers use their own definition for the credential and endpoint while receiving the same shared timeout and logging settings. Their proxy remains unset because the custom definition owns the endpoint directly.
References: src/…/config.py• L19-37 src/…/config.py• L40-66 src/…/config.py• L69-109 src/…/config.py• L112-130 src/…/base.py• L13-31
Adapter selection
The factory gives every catalog ID exactly one construction owner. Most compatible providers use a shared chat adapter profile, while providers that need a dedicated construction path use a named factory. The named path includes providers with distinct request, account, workspace, or service settings. The OpenAI and GitHub Copilot adapters are injected by the application composition root so their external authentication state can be shared with the related account or sign-in management.
Provider adapter modules are prepared when their provider is built rather than when the runtime package is imported. Preparation loads the provider implementation away from the provider's owner loop, then returns a constructor that creates the client on that loop. Custom provider definitions take a separate construction path. An unknown ID fails as an unknown provider. A provider that needs an injected factory fails as unavailable when the application has not supplied that factory. The factory also checks that catalog IDs have one and only one construction owner, which keeps a new provider from being silently omitted or assigned twice. Dedicated providers can carry provider-specific settings into construction, including Anthropic workspace data, NVIDIA NIM settings, Cloudflare account data, Vertex project and location data, and named OpenCode variants. Providers that promise native Messages behavior are also rejected when their selected adapter does not provide that behavior.
References: src/…/__init__.py• L1-10 src/…/factory.py• L274-314 src/…/factory.py• L317-389 src/…/bootstrap.py• L41-62
Model discovery
Model discovery turns provider model-list responses into application-owned model metadata. It uses a provider runtime to query providers in parallel, records successful results, and reports failed provider IDs without making one bad upstream response stop discovery for the rest.
Discovery scope
The system gives priority to providers named by configured chat model references. A configured chat model reference has the form provider/model; the provider prefix selects the provider and the suffix remains the provider's raw model ID.
The model cache scope includes configured catalog providers, connected providers, and custom providers selected in settings. The model-list scope uses that same available set. An unconfigured default local endpoint is treated as optional when it is not referenced by a chat model and has no explicit endpoint or proxy setting. Startup warms the providers required by configured routes, then starts a non-blocking refresh for the available set.
The discovery flow has two paths:
- Startup queries the providers required by the configured routes before normal clients perform discovery.
- Background or explicit refresh queries the available providers and updates the cache on each successful result.
This diagram shows how configured routes and provider availability determine the discovery work.
flowchart TD
S[Validated settings] --> R[Configured provider/model references]
S --> U[Configured provider set]
A[Connected account state] --> U
R --> W[Warm referenced providers]
U --> B[Background or explicit refresh]
W --> Q[Provider model-list requests]
B --> Q
Q --> C[Model metadata cache]
C --> P[Application model catalog]
References: src/…/model_refs.py• L135-181 src/…/discovery.py• L106-159 src/…/provider_manager.py• L344-373 src/…/application.py• L138-160
Model response boundary
Each provider returns ProviderModelInfo, which contains a raw model ID and optional capability data such as reasoning support and text or image input support. The model-list helpers accept provider-specific OpenAI-shaped responses, filter or include models based on provider rules, and reject malformed responses instead of adding uncertain model IDs to the catalog.
Discovery treats a malformed response, an HTTP failure, or a provider configuration failure as a per-provider failure. It walks wrapped transport and SDK errors to classify causes such as proxy, DNS, certificate, TLS, timeout, connection, HTTP status, response, or unavailable errors. Only allowlisted messages or a safe HTTP status form can be shared with callers; upstream exception text and URLs do not enter the result. An unused default local endpoint may log connection or timeout failures at debug level because its absence is normal. Other failures use a general retry message while allowing other provider queries to complete.
References: src/…/model_metadata.py• L6-19 src/…/model_listing.py• L16-32 src/…/model_listing.py• L35-172 src/…/discovery.py• L19-103 src/…/discovery.py• L139-159 src/…/provider_manager.py• L291-342
Model metadata cache
The model cache stores the latest successful metadata by provider and raw model ID. It keeps the full ProviderModelInfo record so downstream users can access provider-reported capabilities as well as model identity. It removes blank IDs, replaces one provider's prior result with its new result, and exposes user-selectable IDs with a provider/model prefix so models from different providers do not collide. A replacement generation can copy the current metadata into separate cache storage, so the new generation starts with known model data without sharing mutable state with the old one.
The cache has an explicit provider scope. A settings change or account change can remove a provider from that scope and evict its metadata. A failed refresh does not erase an existing successful result, which lets the application keep serving the last known model set until a later refresh succeeds.
References: src/…/model_cache.py• L10-36 src/…/model_cache.py• L38-59 src/…/model_cache.py• L61-96 src/…/provider_manager.py• L226-265 src/…/provider_manager.py• L515-524 src/…/provider_manager.py• L582-597
Runtime lifecycle
The runtime lifecycle gives provider clients a clear owner from creation through shutdown. A ProviderRuntime is one settings snapshot with lazy provider construction, while the manager coordinates the current runtime, older runtimes, discovery tasks, model metadata, and external catalog publication.
Lazy provider runtime
The provider runtime creates a provider only when a request or discovery operation resolves its ID and selected API key. It caches that instance by provider ID and API key number, so repeated use of one client key in a settings snapshot shares the provider's admission and client resources. Concurrent resolutions for the same uncached client key share one in-flight construction instead of creating duplicate clients. Separate runtimes never share those provider instances or their admission state.
Cleanup first marks the runtime as closing, cancels unfinished constructions, and waits for them. It then visits every constructed provider. A successful cleanup removes that provider from the runtime cache, while a failed or cancelled cleanup leaves it owned for a later retry. Once cleanup starts, new provider resolution is rejected.
References: src/…/runtime.py• L18-24 src/…/runtime.py• L26-64 src/…/runtime.py• L67-127 src/…/runtime.py• L129-151 src/…/base.py• L34-46
Generation replacement
The manager represents each immutable settings snapshot as a provider generation. A lease is a temporary claim on one generation. Request work uses that lease to resolve providers and keeps the generation alive until the work releases it.
When an Admin update changes live provider settings, the manager builds the candidate runtime and awaits the configuration commit before publishing it as current. The old generation is retired but remains open for its active leases. The new generation serves new leases, and the old one closes after its lease count reaches zero. Replacement is serialized so concurrent changes publish in call order.
If candidate construction or the asynchronous configuration commit fails, the current generation stays active. If candidate cleanup fails after a failed commit, the manager retains that unpublished runtime and retries its cleanup during a later replacement or shutdown.
References: src/…/provider_manager.py• L49-85 src/…/provider_manager.py• L216-221 src/…/provider_manager.py• L551-609 src/…/application.py• L177-212
Shutdown ownership
Shutdown first rejects new leases and refresh work. It then cancels generation discovery, waits for active leases to drain, closes every retired and current generation, retries cleanup for unpublished candidates, and clears the model cache only after all owned provider runtimes close.
Cleanup is retryable. A failed close keeps the runtime and its cached data owned by the manager, so a later close can try again without losing track of resources or silently reporting a clean shutdown.
References: src/…/provider_manager.py• L611-656 src/…/provider_manager.py• L670-757 src/…/application.py• L162-175 src/…/application.py• L436-463
Available provider set
The available provider set is the intersection of catalog support with current configuration and account state. The manager keeps this set separate from provider instances: a provider can be known by the catalog, configured for discovery, cached with model data, or constructed for a request at different times.
Settings changes
An Admin configuration update prepares and validates a new settings snapshot. If the changed fields can apply to the live process, the manager replaces the provider generation and resets the model-cache scope to match the new settings. It also drops cached custom-provider inventory when that inventory changes, so the new static model list becomes the source for that provider. The configuration commit is asynchronous, so managed persistence can finish without blocking the server event loop before the new generation becomes current. If a field needs a restart, the application reports it as pending instead of pretending that the current runtime has changed.
The model catalog publisher receives updates after cache changes and after a new generation becomes current. Publication failures are warnings only; they do not replace valid internal provider state or expose publisher error details.
References: src/…/application.py• L177-228 src/…/provider_manager.py• L247-267 src/…/provider_manager.py• L551-609 src/…/configuration.py• L1-53
Connected account changes
Connected-account state changes update only the affected provider. A new connection adds that provider to the cache scope and refreshes its models. A disconnect removes the provider and evicts its cached models without replacing unrelated provider generations.
This targeted path lets account login and logout change model availability while preserving active provider clients and the settings snapshot used by current work.
References: src/…/application.py• L258-300 src/…/provider_manager.py• L500-524 src/…/provider_manager.py• L226-231
Change checklist
When these files or sections are changed, remember to consider these:
- A new or renamed provider ID must stay aligned across the catalog order, settings model references, construction owner, and any connected-account owner.
- A new credential, base URL, proxy, or required setting must map through the catalog, validated settings, Admin field manifest, and provider configuration builder.
- A fallback API key must remain aligned across provider selection, client-cache identity, admission control, and missing-key errors.
- A custom provider setting must preserve the separate custom-definition path while keeping shared timeout and logging behavior aligned with catalog providers.
- A provider that changes its model-list shape must keep its metadata parser and discovery failure handling aligned so malformed data cannot enter the shared cache.
- A change to local, custom, or connected-account status must update both discovery scope and cache eviction rules.
- A change to optional local-provider handling must preserve the distinction between an unused default endpoint and an explicitly configured or referenced local provider.
- A change to provider preparation must preserve the split between worker-side module loading and owner-loop client construction.
- A change to generation replacement must preserve the commit-before-publish rule, await the configuration commit, and preserve the lease drain before cleanup.
- Managed configuration persistence must remain outside the server event loop while the replacement lock still protects the commit-and-publish sequence.
- A change to model metadata, capability fields, or prefixed IDs must remain compatible with the external model catalog publisher.
- A change to discovery error classification must keep shared failure messages bounded to safe categories and prevent upstream exception text from reaching callers.
- A change to custom model inventory must keep generation cache copying aligned with the rule that changed inventory replaces old custom metadata.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| Required provider key, selected fallback key, endpoint, project, or account value is missing | Provider construction fails with a user-facing unavailable error. | Catalog configuration attributes and the validated settings boundary check required values before construction. |
| A custom provider definition is missing its required endpoint | The custom provider cannot be built for the selected settings. | The custom configuration path builds its endpoint from the validated definition before client construction. |
| A provider model-list response is malformed or contains no usable IDs | That provider is marked as failed and its last successful cache entry remains available. | Strict model-list parsing and per-provider refresh results isolate the bad response. |
| A provider or network error includes unsafe upstream text | Callers receive a general safe discovery message instead of provider response text or a URL. | Discovery maps known causes to an allowlist of messages and rejects all other message values. |
| An unused default local endpoint is absent | Discovery logs the connection or timeout at debug level and does not treat the optional provider as a required failure. | Local-provider classification checks that the endpoint is default, unreferenced, and has no explicit proxy or endpoint setting. |
| A configured provider is removed from settings or a connected account is disconnected | Its cached model metadata is evicted and it is no longer considered available. | The manager synchronizes cache scope before reads and handles account changes as targeted add or remove events. |
| Two callers resolve the same uncached provider and API key at once | Duplicate client construction could waste resources or split admission state. | The runtime shares one in-flight construction task for each provider and API key number. |
| A provider generation is replaced while requests still hold leases | The old provider clients remain open until those requests release their leases. | Retired generations track active leases and close only after the count reaches zero. |
| Candidate construction or asynchronous config commit fails | The current generation remains active; a candidate may need later cleanup. | Replacement publishes only after both steps succeed and retains failed unpublished runtimes for retry. |
| A configuration commit is treated as synchronous or blocks the event loop | Managed settings I/O can stall server work or publish a runtime before persistence completes. | The replacement contract awaits an async commit before it publishes the candidate generation. |
| Provider cleanup fails during replacement or shutdown | The runtime stays owned and shutdown reports incomplete. | Cleanup keeps failed entries and retries them on a later lifecycle close. |
Key files
| File | Why to read it |
|---|---|
| src/…/provider_catalog.py | Defines the shared provider identity, access boundary, configuration metadata, defaults, and canonical order. |
| src/…/settings.py | Defines the validated settings and provider/model reference rules that feed runtime configuration. |
| src/…/provider_manifest.py | Shows how provider metadata becomes the Admin configuration surface. |
| src/…/__init__.py | Defines the runtime package facade used to access provider configuration and lifecycle entry points. |
| src/…/config.py | Explains the boundary between validated settings and shared provider configuration, including fallback keys, required values, and custom provider settings. |
| src/…/factory.py | Shows how catalog and custom providers receive one construction owner, including dedicated provider settings and injected adapters. |
| src/…/discovery.py | Shows discovery scope, optional local-provider handling, and safe failure classification for model-list work. |
| src/…/model_cache.py | Shows provider-scoped storage of complete model metadata and prefixed catalog output. |
| src/…/runtime.py | Shows lazy provider ownership, API-key-specific caching, shared in-flight construction, and retryable resource cleanup for one settings snapshot. |
| src/…/provider_manager.py | Shows generation replacement, async configuration commit, leases, discovery coordination, cache scope, publication, and shutdown. |
| src/…/application.py | Shows startup warming, Admin updates, connected-account changes, and application-level shutdown. |
| src/…/configuration.py | Shows the managed configuration boundary used during live provider replacement. |
| src/…/bootstrap.py | Shows the production wiring for connected authentication, provider construction, and model catalog publication. |