LLM runtime and provider routing
The Python LLM runtime gives callers one way to send messages to several language-model backends. It covers command-line setup, provider and model choice, shared request and response data, prompt shaping, and bounded callable-tool work. Provider-specific behavior is described in LLM providers and model selection, while prompt and tool-call behavior is described in LLM prompts and tool execution.
Runtime boundary
The runtime exposes a small public surface through the llm package. A caller can select a provider, build an LLMInput, call the provider, and receive an LLMOutput without using a provider SDK directly. The package also exposes the tool registry and executor so an application can add actions around a model call. References: src/…/__init__.py• L1-34
Command-line operation
The command-line entry point starts an interactive setup flow rather than a chat session. It presents a provider list, presents models for the chosen provider, and saves the choice in .llm.env; the saved values can also be loaded into the current shell process. The installed llm-select command and python -m llm use this path. References: src/…/__main__.py• L1-7, src/…/selector.py• L52-69, src/…/selector.py• L72-106, pyproject.toml• L41-42
The command-line list is a user-facing starting point. It includes Claude, OpenAI, and Ollama with common model choices. The runtime resolver also knows about Astraflow, Astraflow China, and Atlas, so applications can route to those providers even though the default interactive list does not show them. References: src/…/selector.py• L109-158, src/…/types.py• L17-24, src/…/resolver.py• L16-23
Provider resolution
Provider resolution turns a requested provider name into an initialized object that follows the common provider contract. The precedence order is explicit input, the LLM_PROVIDER process setting, the local .llm.env file, and then Claude as the default. Names from the environment are trimmed and normalized to lower case. References: src/…/resolver.py• L28-75
The routing path is therefore:
flowchart TD
A[Caller or CLI] --> B{Explicit provider?}
B -- Yes --> E[Resolve provider]
B -- No --> C{LLM_PROVIDER set?}
C -- Yes --> E
C -- No --> D{.llm.env has provider?}
D -- Yes --> E
D -- No --> F[Use Claude]
F --> E
E --> G[Initialize provider adapter]
An unknown name fails during resolution instead of silently selecting another backend. Applications can add a provider to the resolver registry when it implements the shared contract. References: src/…/resolver.py• L62-79, src/…/interface.py• L11-30
Core execution model
The core model keeps application messages and model results independent from any one vendor format. Provider adapters receive the same request shape and return the same result shape. This lets higher-level code reason about content, model use, token use, stop state, and tool calls through one set of concepts. References: src/…/types.py• L10-24, src/…/types.py• L26-45, src/…/types.py• L96-147
Request and response types
Message is a conversation item with one of four roles: system, user, assistant, or tool. LLMInput groups the message history with an optional model, generation controls, tool definitions, streaming intent, and caller metadata. LLMOutput returns text plus optional tool calls, model identity, usage data, and a stop reason. References: src/…/types.py• L10-15, src/…/types.py• L26-45, src/…/types.py• L82-147
Tool data uses a shared three-part model. A ToolDefinition describes an action and its parameter schema. A ToolCall is the model's request to use that action. A ToolResult carries the action result or an error back into the conversation. The shared definitions can be serialized for provider APIs, which keeps the boundary between the runtime and an adapter clear. References: src/…/types.py• L48-94
Provider contract
LLMProvider defines the runtime's main provider boundary. Each adapter must generate a result, list its known models, and report whether its configuration is usable. It also reports tool and vision support and a default model. Common error classes identify authentication, rate limits, context limits, missing models, and tool execution problems for callers that need to react by category. References: src/…/interface.py• L11-60
The runtime maps provider-specific SDK responses into LLMOutput and maps common request data into the provider's request format. This is the main reason an application can change its route without changing its conversation model. See LLM providers and model selection for the provider choices and their capability differences. References: src/…/claude.py• L58-124, src/…/openai.py• L71-113, src/…/ollama.py• L60-100
Prompt normalization
Prompt normalization prepares a common message history for provider families with different prompt rules. PromptBuilder applies optional prompt settings to the message list and can merge a system instruction with an existing system message. It can also place tool descriptions in the system message when a provider needs a text-based form. Provider builders choose the appropriate policy before the request reaches an adapter.
References: src/…/builder.py• L11-79 src/…/builder.py• L104-132
Provider prompt adaptation
Claude and OpenAI use native tool descriptions, while Ollama's configured prompt path includes tool descriptions in the system message. A provider name without a special prompt policy uses the general defaults. The builder accepts either a complete prompt configuration or individual prompt options, but not both at the same time. This adaptation changes the presentation of the same intent; it does not change the application's Message or ToolDefinition concepts.
References: src/…/builder.py• L11-17 src/…/builder.py• L19-79 src/…/builder.py• L104-132
Named templates provide a separate shared store for reusable prompt text. Callers can register, replace, read, or remove a named template. The registry does not select a provider by itself. References: src/…/__init__.py• L5-41
Callable tool execution
Callable tools let a model request a named application action during a run. The runtime keeps the public tool description next to the callable that fulfills it, then turns each call into a result message that the model can use in its next step. See LLM prompts and tool execution for the detailed prompt and tool lifecycle. References: src/…/executor.py• L18-40, src/…/types.py• L48-94
Tool registry
ToolRegistry is the name-based bridge between model-facing tool definitions and Python callables. Registration replaces an existing entry with the same name, and the registry can list definitions for request construction. It does not run code or decide when a model should call a tool. References: src/…/executor.py• L21-40
ReAct execution
ReActAgent coordinates a bounded request-and-action loop. It sends the current history to the provider, returns when the model produces no tool calls, or executes every requested call and appends the assistant request and tool results before asking again. The loop returns a synthetic max_iterations result when it reaches its limit. References: src/…/executor.py• L71-122
The flow keeps model reasoning and application actions separate:
sequenceDiagram
participant A as Agent
participant P as Provider
participant X as Tool executor
A->>P: Send message history and tool definitions
P-->>A: Text result or tool calls
alt Text result
A-->>A: Return final output
else Tool calls
A->>X: Execute named callables
X-->>A: Return tool results
A->>P: Send updated history
end
An unknown tool or a callable exception becomes an error ToolResult linked to the original call. The agent can pass that result back to the model, so a failed action remains part of the conversation instead of escaping as an unstructured runtime failure. References: src/…/executor.py• L43-68, src/…/executor.py• L97-117
Change checklist
When these files or sections are changed, remember to consider these:
- When adding or renaming a provider, update the provider type, resolver map, public provider exports, and any user-facing selection list that should expose it.
- When changing a request or result field, update its provider serialization and response mapping so all adapters preserve the shared contract.
- When changing tool schemas or call shapes, keep the shared definition, provider-specific serialization, registry lookup, and tool-result message aligned.
- When changing provider prompt policy, check whether tool descriptions must remain native or move into the system message.
- When changing saved configuration names or precedence, keep the CLI writer and resolver reader consistent.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| Provider name is not in the resolver registry | Provider creation raises ValueError and no model request starts | Use a registered ProviderType or register the adapter before routing to it |
| Required provider credentials are missing or rejected | The adapter raises an authentication error during validation or generation | Validate the selected provider configuration before sending work |
| Request exceeds a provider context limit | The adapter raises a context-length error and returns no usable output | Keep message history and requested output within the selected model's limits |
| Model requests a name absent from the tool registry | The executor returns an error tool result | Register every advertised tool with the exact model-facing name |
| A callable raises an exception | The executor converts the exception to an error tool result | Treat tool results with is_error as failed actions in the next model step |
| Agent reaches its iteration limit | The agent returns Max iterations reached with stop reason max_iterations | Set a suitable limit and handle this stop reason as incomplete work |
Key files
| File | Why to read it |
|---|---|
| src/…/__init__.py | Public runtime entry points and package-level abstractions |
| src/…/selector.py | Interactive provider and model setup flow |
| src/…/resolver.py | Provider precedence, lookup, and registration |
| src/…/interface.py | Shared provider boundary and error categories |
| src/…/types.py | Common message, request, result, model, and tool concepts |
| src/…/builder.py | Provider-aware prompt normalization |
| src/…/executor.py | Callable registration and bounded tool-call loop |
| src/…/claude.py | Example of a native provider adapter |
| src/…/openai.py | Example of a chat-completions provider adapter |
| src/…/ollama.py | Example of a local provider route |