LLM providers and model selection
The provider layer gives the LLM runtime one common way to use hosted and local language models. It selects a provider from explicit input or saved runtime settings, then lets that provider choose the requested model and translate the common request into its service format. See LLM runtime and provider routing for the parent runtime view and LLM prompts and tool execution for prompt and tool behavior.
Provider abstraction
An LLM provider is an adapter that connects the runtime to one model service. Every adapter exposes generation, model information, configuration validation, a default model, and capability flags through the shared LLMProvider contract.
The runtime exchanges common LLMInput and LLMOutput values with the adapter. This keeps provider choice separate from the rest of the request flow. A ModelInfo record describes whether a model supports tools or vision and can state its token limits.
References: src/…/interface.py• L11-30 src/…/types.py• L97-167
Supported providers
The runtime supports managed APIs, OpenAI-compatible gateways, and a local model service. The provider name is the stable value used by the resolver and configuration.
| Provider name | Service role | Default model or model source | Main configuration need |
|---|---|---|---|
claude | Anthropic Messages API | claude-sonnet-4-6 | ANTHROPIC_API_KEY |
openai | OpenAI chat completions | gpt-4o-mini | OPENAI_API_KEY |
ollama | Local Ollama service | OLLAMA_MODEL or llama3.2 | Reachable OLLAMA_BASE_URL |
astraflow | UModelVerse global OpenAI-compatible endpoint | ASTRAFLOW_MODEL, or gpt-4o-mini | ASTRAFLOW_API_KEY |
astraflow_cn | UModelVerse China OpenAI-compatible endpoint | ASTRAFLOW_CN_MODEL, then ASTRAFLOW_MODEL, or gpt-4o-mini | ASTRAFLOW_CN_API_KEY |
atlas | Atlas Cloud OpenAI-compatible endpoint | ATLAS_MODEL or deepseek-ai/deepseek-v4-pro | ATLAS_API_KEY or ATLASCLOUD_API_KEY |
Claude and OpenAI publish fixed model catalogs in this layer. The gateway adapters expose the configured default model as their available model entry, so the configured model name remains the routing choice.
References: src/…/types.py• L17-24 src/…/claude.py• L18-56 src/…/openai.py• L27-69 src/…/ollama.py• L23-58 src/…/astraflow.py• L20-23 src/…/astraflow.py• L133-151 src/…/atlas.py• L20-24 src/…/atlas.py• L40-80
Provider resolution
The provider resolver turns a provider name into a configured adapter. It accepts an explicit provider, reads the process environment when no provider is given, and then reads .llm.env; if none of these sources names a provider, it uses Claude.
The resolution order is therefore:
- Explicit provider passed to
get_provider. LLM_PROVIDERin the process environment.LLM_PROVIDERin the current working directory's .llm.env file.claudeas the built-in fallback.
Provider values from environment sources are trimmed and converted to lower case. An unknown value fails before a request is sent. The resolver also permits registration of another adapter under a ProviderType, which extends the routing table without changing the caller contract.
The following diagram shows how provider input becomes an adapter instance.
flowchart TD
A[Provider request] --> B{Explicit provider?}
B -->|Yes| D[Normalize provider type]
B -->|No| C{LLM_PROVIDER set?}
C -->|Yes| D
C -->|No| E[Read .llm.env]
E --> F{Saved provider found?}
F -->|Yes| D
F -->|No| G[Use claude]
G --> H[Resolve registered adapter]
D --> H
H --> I[Create provider]
References: src/…/resolver.py• L16-23 src/…/resolver.py• L35-75 src/…/resolver.py• L78-79 tests/test_resolver.py• L50-102
Model selection
Model routing has two levels. The provider is chosen by the resolver, while the model is chosen for each generation request or falls back to the provider's configured default.
Request model precedence
When a request contains LLMInput.model, adapters use that model for the call. When it is absent, the adapter uses its own default. This means a request can route to a different model without changing the selected provider, while provider-specific environment settings control the normal path.
The interactive selector is a separate convenience path. It presents provider and model choices, writes both values to .llm.env, and can also place them in the current process when persistence is requested. The resolver consumes the saved provider, but provider adapters do not read the generic LLM_MODEL value as their default; model defaults come from the adapter-specific settings described below.
References: src/…/types.py• L97-104 src/…/openai.py• L71-75 src/…/claude.py• L58-60 src/…/ollama.py• L60-67 src/…/astraflow.py• L69-73 src/…/atlas.py• L82-86 src/…/selector.py• L94-106 src/…/selector.py• L109-158
Interactive model choices
The command-line selector offers guided choices for Claude, OpenAI, and Ollama. Its lists describe a capability trade-off such as power, speed, cost, or code focus. Astraflow and Atlas remain available through direct provider configuration even though they are not in the selector's default menu.
References: src/…/selector.py• L109-141 src/…/selector.py• L143-158
Provider configuration
Configuration supplies credentials, service endpoints, and default models to each adapter. Constructor arguments take priority over environment values where the adapter supports both. The values below are the provider-specific controls that affect routing.
| Provider | Credential | Endpoint override | Model override |
|---|---|---|---|
| Claude | ANTHROPIC_API_KEY | Optional constructor base_url | No provider-specific model environment value; use the request model or adapter default |
| OpenAI | OPENAI_API_KEY | Optional constructor base_url | No provider-specific model environment value; use the request model or adapter default |
| Ollama | None required | OLLAMA_BASE_URL; default http://localhost:11434 | OLLAMA_MODEL; default llama3.2 |
| Astraflow | ASTRAFLOW_API_KEY | ASTRAFLOW_BASE_URL; global UModelVerse endpoint by default | ASTRAFLOW_MODEL; default gpt-4o-mini |
| Astraflow China | ASTRAFLOW_CN_API_KEY | ASTRAFLOW_CN_BASE_URL; China ModelVerse endpoint by default | ASTRAFLOW_CN_MODEL, then ASTRAFLOW_MODEL; default gpt-4o-mini |
| Atlas | ATLAS_API_KEY or ATLASCLOUD_API_KEY | ATLAS_BASE_URL; Atlas Cloud endpoint by default | ATLAS_MODEL; default deepseek-ai/deepseek-v4-pro |
Validation checks whether the adapter has the minimum value it needs to attempt a request. For managed providers this is normally an API key. For Ollama it is a non-empty service URL, so validation does not prove that a local service is running.
References: src/…/claude.py• L26-30 src/…/claude.py• L135-142 src/…/openai.py• L27-35 src/…/openai.py• L124-131 src/…/ollama.py• L26-32 src/…/ollama.py• L111-118 src/…/astraflow.py• L39-59 src/…/astraflow.py• L123-130 src/…/atlas.py• L48-72 src/…/atlas.py• L140-147
Capability routing
Capability metadata helps the runtime and callers decide whether a selected model can support a requested interaction. The current catalogs mark Claude, OpenAI, Astraflow, and Atlas models as tool-capable, while the listed Ollama models are not marked as tool-capable; Claude and OpenAI models also expose vision support.
OpenAI-compatible adapters share the same broad request shape, but each gateway still owns its endpoint and model defaults. Claude uses its own message service and handles system content and adaptive thinking according to the selected Claude model. Atlas applies a minimum output-token floor for reasoning models. These differences affect which model is safe for a request even when the common runtime interface is the same.
References: src/…/interface.py• L23-30 src/…/claude.py• L60-85 src/…/openai.py• L36-68 src/…/ollama.py• L33-57 src/…/astraflow.py• L60-67 src/…/atlas.py• L73-80 src/…/atlas.py• L90-97
Error boundaries
Adapters map common request failures into typed LLM errors where the service response makes the cause clear. Authentication failures, rate limits, and context-length failures retain the provider identity so the caller can report or handle the problem without guessing which backend failed.
An empty or filtered response is treated as a failed generation for the OpenAI-compatible adapters that check response choices. Malformed gateway tool arguments are preserved as raw data rather than silently discarded, which lets the higher-level tool path decide how to handle them.
References: src/…/interface.py• L33-60 src/…/openai.py• L83-122 src/…/claude.py• L86-133 src/…/astraflow.py• L82-121 src/…/atlas.py• L99-138
Change checklist
When these files or sections are changed, remember to consider these:
- Keep the
ProviderTypevalues, resolver map, exported provider names, and configuration names aligned. A mismatch can make a provider selectable but not constructible. - Keep provider defaults consistent with the selector's model labels when the selector presents that provider. The selector writes the generic
LLM_MODEL, but adapters use their own model settings. - Update both the global and China Astraflow endpoint or model fallback rules together. The China adapter intentionally falls back to the global model setting.
- Preserve the
ModelInfocapability flags when changing a model catalog. Callers may use them before sending tools or vision input. - Keep provider error types tied to the correct
ProviderType. The runtime uses that identity to report backend-specific failures.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
LLM_PROVIDER names a value outside the registered provider types | Provider creation stops with an unknown-provider error | Use one of the resolver's registered names or register a matching adapter |
| A managed provider has no API key | Configuration validation fails and a request cannot authenticate | Set the provider key or pass it to the adapter |
| Ollama has a valid-looking URL but no running service | Validation can pass, then generation fails on connection | Check the local service at the configured base URL |
| A request sends tools to a model marked as not tool-capable | The backend may reject the request or return no usable tool call | Check ModelInfo.supports_tools before routing tool work |
The selector saves a model only as LLM_MODEL | The selected provider may still use its adapter default | Set the provider-specific model variable or pass LLMInput.model |
| Atlas receives too little output-token space for reasoning | The answer can be truncated or empty | Allow the adapter's minimum token floor or provide a larger limit |
Key files
| File | Why to read it |
|---|---|
| src/…/interface.py | Defines the provider contract and typed error boundary. |
| src/…/types.py | Defines provider names, model metadata, and the request field that selects a model. |
| src/…/resolver.py | Explains provider precedence, default selection, and adapter registration. |
| src/…/selector.py | Shows the user-facing provider and model choice flow and its saved configuration. |
| src/…/claude.py | Shows the Anthropic adapter and Claude model-specific behavior. |
| src/…/openai.py | Shows the OpenAI adapter and its fixed model catalog. |
| src/…/ollama.py | Shows local service routing and local model defaults. |
| src/…/astraflow.py | Shows the global and China OpenAI-compatible gateway variants. |
| src/…/atlas.py | Shows Atlas Cloud routing and reasoning-model token protection. |