Model provider integration
Model provider integration gives the agent one stable way to use many AI services. It selects a provider route, obtains safe credentials, translates the agent's common request shape into the provider's protocol, and turns each reply back into a form that the turn loop can use. The agent runtime owns the turn; agent turn execution explains when provider calls occur; context, memory, and compression explains how the input is prepared; and agent tools and guardrails explains what happens after tool calls return.
Provider registration
Provider registration defines the identity and behavior of an AI service before a request runs. The same provider identity feeds model selection, setup screens, authentication, endpoint choice, and request policy.
Provider profiles
The provider profile is the shared description of a service. It states the provider name, aliases, endpoint defaults, authentication type, model discovery rules, and capabilities. It also offers provider-specific policy for message preparation, request fields, model limits, reasoning levels, and response use without moving client ownership or retry control out of the agent.
User-installed provider plugins can add profiles or replace bundled profiles in a profile-scoped layer. A profile can therefore extend the provider set without changing the main agent flow, while name lookup still resolves the selected profile and its aliases.
References: providers/base.py• L41-82 providers/base.py• L122-141 providers/__init__.py• L192-225
Provider catalog
The provider catalog is the common view used by model pickers and credential setup. It joins the canonical model list with provider profiles and authentication records so that a provider offered for model use also has a consistent label, credential type, endpoint hint, and setup destination.
Discovery may hide a pre-release provider until the user selects it by name. Hiding affects menus only; explicit provider resolution still works.
References: hermes_cli/provider_catalog.py• L17-44 hermes_cli/provider_catalog.py• L66-108 providers/__init__.py• L277-288
Runtime route selection
Runtime route selection turns a requested provider and model into a concrete route. The result includes the provider identity, endpoint, authentication value or token source, and wire protocol used by the agent call.
Resolution ladder
The resolver checks the most specific route first and uses a generic compatible route only when earlier choices do not apply. This keeps an explicit user choice authoritative while still allowing automatic selection when the user has not fixed a provider.
The route order is:
- Reject a provider disabled in configuration.
- Resolve special routes such as virtual, Azure, Vertex, local, and named custom providers.
- Use an explicit key or endpoint when supplied.
- Select a matching credential-pool entry.
- Resolve OAuth, external-process, native, or registry-backed API-key providers.
- Use the OpenRouter or bare-custom fallback only when no earlier route resolves.
The resolver also chooses the protocol mode for the selected endpoint. A model or endpoint can require Chat Completions, Responses, Anthropic Messages, Bedrock Converse, or an external process. A configured mode is used only when it belongs to the selected provider, so a stale mode or endpoint does not silently cross a provider boundary.
flowchart TD
A[Requested model and provider] --> B{Explicit or configured route?}
B -->|Yes| C[Validate provider and endpoint]
B -->|No| D[Resolve aliases and special routes]
C --> E[Select credentials]
D --> E
E --> F{Credential or route available?}
F -->|Yes| G[Choose API mode]
F -->|No| H[Try next resolution rung]
H --> E
G --> I[Build provider client and request]
References: hermes_cli/runtime_provider.py• L490-499 hermes_cli/runtime_provider.py• L994-1033 hermes_cli/runtime_provider.py• L1056-1098
Endpoint and protocol selection
Endpoint selection protects credentials and request shape at the same time. Built-in provider endpoints come from the provider registry or profile. Custom endpoints use explicit configuration and host checks. Local aliases fail when they have no endpoint instead of falling through to an unrelated cloud service.
Protocol selection follows the provider, endpoint, and model. Direct Anthropic endpoints use the native Messages protocol. OpenAI and compatible routes may use Chat Completions or Responses. Azure and Bedrock can select a mode from the deployment or model. External-process providers use a process route rather than an HTTP client.
References: hermes_cli/runtime_provider.py• L84-127 hermes_cli/runtime_provider.py• L142-193 hermes_cli/runtime_provider_backends.py• L205-245 hermes_cli/runtime_provider_backends.py• L248-277
Authentication and credential pools
Authentication supplies a usable credential to the selected route without exposing it to another endpoint. The system supports static keys, OAuth records, cloud identity chains, provider-owned sign-in, and local no-auth routes.
Credential sources
The resolver can use an explicit key or endpoint, profile-scoped configuration, environment-backed secrets, a stored OAuth account, a credential pool, or a provider's own external login. The source is retained with the runtime so later recovery can refresh or rotate the credential that made the failing request.
Some credentials are short-lived or callable token sources. The runtime refreshes them as part of route use rather than treating a file path or token provider as a static API key. Provider-specific authentication can also own add, refresh, status, and logout actions.
References: providers/base.py• L61-82 hermes_cli/runtime_provider_backends.py• L27-63 hermes_cli/runtime_provider_backends.py• L66-110 hermes_cli/runtime_provider.py• L770-816
Credential pool health
A credential pool lets one provider use several accounts or keys. Selection can follow fill-first, round-robin, random, or least-used policy. The pool records credential-wide failures separately from model-specific cooldowns, because a model limit does not always make the account unusable for other models.
Recovery acts on the credential that sent the failed request. Authentication failures try refresh first. Rate limits normally retry once and then rotate. Billing failures rotate without treating the same credential as healthy. Provider reset times can replace the default cooldown.
References: agent/credential_pool.py• L120-150 agent/credential_pool.py• L220-240 agent/agent_runtime_helpers.py• L879-894 agent/agent_runtime_helpers.py• L975-1014
Request and response translation
Translation separates the agent's common message and tool model from provider wire formats. The turn loop works with one internal shape; a transport adapts that shape for the selected protocol and returns a normalized result.
Transport contracts
Each transport converts messages and tool schemas, builds request fields, validates the raw response, and normalizes the result. Transports do not own credentials, client lifetime, streaming, or retry policy. Those concerns remain at the agent boundary so all providers follow the same turn and recovery rules.
The main protocol families are:
| Protocol family | System purpose |
|---|---|
| Chat Completions | Common OpenAI-shaped messages and tool calls for compatible services |
| Anthropic Messages | Native system blocks, content blocks, thinking, and tool use |
| Responses | Structured input items, reasoning items, response tools, and continuation state |
| Bedrock | AWS-native request and response forms for Converse and related routes |
References: agent/…/base.py• L1-53 agent/…/__init__.py• L16-51 agent/…/types.py• L15-73
Request translation
Request translation removes internal history fields that a destination does not accept and applies only the provider rules needed for the selected model. It can change roles, reasoning fields, output limits, cache hints, tool names reserved by a service, and multimodal content shape. Stored history stays in the agent's common form so the session can change providers safely.
Tool names can be aliased on the wire when a service reserves a name for a native tool. The transport keeps request-local alias data and restores the original name before local dispatch. This preserves the agent tool contract while avoiding provider collisions.
References: agent/…/chat_completions.py• L381-466 agent/…/chat_completions.py• L472-603 agent/…/codex.py• L126-151 agent/…/codex.py• L214-272
Response translation
Response translation maps provider output into NormalizedResponse. The common result carries visible content, tool calls, finish reason, reasoning, usage, and optional provider data. Provider data keeps protocol state that must survive a later request, such as signed thinking blocks, Responses items, or multimodal tool metadata.
The normalized result lets the turn loop apply one policy to ordinary replies, tool rounds, refusals, length limits, and content filters. Provider-specific state remains available for replay without making every downstream consumer understand every provider protocol.
References: agent/…/types.py• L60-88 agent/…/chat_completions.py• L605-686 agent/…/anthropic.py• L65-132 agent/…/codex.py• L832-888
Streaming
Streaming sends partial provider output to the user while the model is still working. It also gives the runtime a way to detect a stalled connection and recover without waiting for a full response.
Stream delivery
The agent prefers streaming when the route supports it and when the client path is not an external process or another route that cannot stream. A single stream owner forwards text and reasoning deltas to display and speech callbacks. Stateful filters remove hidden reasoning and memory markers even when a provider splits a tag across chunks.
The stream keeps the visible partial reply separate from the final normalized response. If a stream stops during an interrupt or network error, the runtime can preserve only the visible partial text and avoid replaying incomplete signed or encrypted provider state.
References: agent/turn_api_call.py• L47-65 agent/turn_api_call.py• L87-119 agent/stream_delivery.py• L47-80 agent/turn_api_call.py• L171-211
Stream health
The stream monitor emits a wait notice when no chunk arrives and treats a stale stream as a recoverable connection problem. Local model loading has its own liveness signal so a cold model is not mistaken for a dead provider.
Malformed or empty server-sent event frames can disable streaming for a later attempt. The request then uses the non-streaming path instead of repeating a known-bad stream shape.
References: agent/chat_completion_stream_monitor.py• L10-37 agent/chat_completion_stream_monitor.py• L39-89 agent/error_classifier.py• L20-30
Capabilities and model catalogs
Capabilities describe what a provider or model can safely accept. They guide model choice and request construction before the request reaches the service.
Capability declarations
Provider profiles declare broad capabilities such as vision support, tool-message image support, prompt-cache support, model context size, output limits, and supported reasoning levels. Model-specific metadata can narrow or extend those declarations. The transport uses this information to omit unsupported fields or clamp a request to the model's accepted vocabulary.
Capability checks are route-aware. An aggregator may route a vendor-prefixed model to a stricter upstream service, so the target model's profile can refine the aggregator's general rule.
References: providers/base.py• L84-109 providers/base.py• L148-150 providers/base.py• L263-312 providers/__init__.py• L229-253
Model discovery and fallback models
Model discovery uses a live provider catalog when the route supports it. A provider may define a separate catalog endpoint or return a static list when live discovery is unavailable. Static fallback models keep setup and model selection usable during catalog outages, but they should represent models that support the agent's tool use.
References: providers/base.py• L122-128 providers/base.py• L356-389 hermes_cli/provider_catalog.py• L66-108
Fallbacks and rate limits
Failure handling converts provider errors into recovery actions rather than exposing raw service behavior to the turn loop. The classification includes the provider, model, route, error kind, and reset hints needed to choose between retry, credential change, compression, fallback, or stop.
Failure classification
The classifier checks provider hooks and known response shapes before generic HTTP status and message rules. It distinguishes a bad key from a blocked upstream, a provider-wide overload from a credential rate limit, and a model entitlement failure from a transient authentication failure.
Important classes include:
- Authentication and billing failures, which affect credential health.
- Rate limits and upstream rate limits, which may affect only the current route or model.
- Overload, server, timeout, and stream failures, which usually support retry.
- Context and payload limits, which route to compression or request reduction.
- Model and format errors, which require a different model or request shape.
References: agent/error_classifier.py• L32-66 agent/error_classifier.py• L69-84 agent/error_classifier.py• L956-998 agent/error_classifier.py• L1019-1078
Recovery chain
Recovery uses the classified reason to preserve the best chance of completing the same turn. It retries transient failures with backoff, rotates a matching credential when the key is the problem, and activates the configured fallback chain when the current provider cannot recover.
An upstream aggregator limit does not bench the local credential. A rate-limited primary can fall back early when no healthy pool entry can recover it. A pinned route remains pinned, so its failure does not silently move an explicitly selected job or session to another provider.
References: agent/turn_recovery.py• L1700-1706 agent/turn_recovery.py• L1913-1965 agent/agent_runtime_helpers.py• L1006-1029
Rate-limit windows
The runtime extracts reset information from provider bodies, Retry-After, rate-limit headers, vendor reset headers, and human-readable delay text. Retry-After takes priority and is capped to prevent an unbounded wait. If no provider delay exists, the runtime uses jittered or provider-specific adaptive backoff.
The user sees a transient wait status while retries are buffered. If the window is too long or retries fail, the final result names the limit and the next available fallback rather than hiding the cause.
References: agent/agent_runtime_helpers.py• L3654-3738 agent/turn_recovery.py• L1444-1504
Change checklist
When these files or sections are changed, remember to consider these:
- Keep provider profile names, aliases, authentication records, and catalog descriptors aligned. A mismatch can make a provider selectable but not resolvable.
- Keep each
api_modevalidator and transport registration aligned. A registered dialect that is rejected during route resolution loses its request and response translation. - Keep request-local tool aliases paired with response de-aliasing. A missing reverse map sends a provider name into local tool dispatch.
- Keep capability declarations aligned with model catalog entries. A wrong vision or reasoning capability can produce a request that the service rejects.
- Keep credential-pool provider identity and endpoint matching intact. A fallback provider must not mutate the primary provider's pool or endpoint.
- Keep reset-time extraction and cooldown state consistent. Losing a provider reset hint can cause repeated calls during the same limit window.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| A configured local alias has no endpoint | The request could otherwise fall through to an unrelated cloud route | Fail before generic fallback and require an endpoint |
| A provider mode remains from a previous provider | The service receives the wrong wire protocol | Honor a saved mode only when it belongs to the selected provider |
| An aggregator returns an upstream 429 | A healthy local credential could be benched | Classify it as an upstream rate limit and use a model fallback |
| One pooled credential receives a model-specific limit | Other models on that credential become unavailable | Store model cooldowns separately from credential-wide status |
| A streaming endpoint sends empty or non-JSON frames | Repeating streaming requests can stall the turn | Disable streaming for the next attempt |
| A reserved provider tool name is sent unchanged | The provider rejects the request or invokes its native tool | Alias the name per request and restore it on response |
Key files
| File | Why to read it |
|---|---|
| hermes_cli/runtime_provider.py | Main provider, endpoint, protocol, and credential resolution flow |
| providers/base.py | Provider profile contract for authentication, endpoints, and capabilities |
| providers/__init__.py | Provider registration, aliases, plugin layers, and profile lookup |
| hermes_cli/provider_catalog.py | Shared provider view used by model and credential setup surfaces |
| agent/…/base.py | Common request translation and response normalization contract |
| agent/…/chat_completions.py | OpenAI-compatible request and response path |
| agent/…/anthropic.py | Anthropic Messages translation and signed content handling |
| agent/…/codex.py | Responses API translation, reasoning, and tool-name handling |
| agent/error_classifier.py | Provider error taxonomy and recovery classification |
| agent/credential_pool.py | Credential state, selection, cooldowns, and rotation data |
| agent/turn_recovery.py | Retry, backoff, credential recovery, and provider fallback decisions |
| agent/stream_delivery.py | Stream ownership and safe partial-output delivery |