Introducing a new way to review your PRs.

Read the announcement

affaan-m / ecc

The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.

View on GitHub

30 documents · Updated Nov 20, 2026

docs / llm-runtime-and-provider-routing

LLM runtime and provider routing

The Python LLM runtime gives callers one way to send messages to several language-model backends. It covers command-line setup, provider and model choice, shared request and response data, prompt shaping, and bounded callable-tool work. Provider-specific behavior is described in LLM providers and model selection, while prompt and tool-call behavior is described in LLM prompts and tool execution.

Runtime boundary

The runtime exposes a small public surface through the llm package. A caller can select a provider, build an LLMInput, call the provider, and receive an LLMOutput without using a provider SDK directly. The package also exposes the tool registry and executor so an application can add actions around a model call. References: src/…/__init__.py• L1-34

Command-line operation

The command-line entry point starts an interactive setup flow rather than a chat session. It presents a provider list, presents models for the chosen provider, and saves the choice in .llm.env; the saved values can also be loaded into the current shell process. The installed llm-select command and python -m llm use this path. References: src/…/__main__.py• L1-7, src/…/selector.py• L52-69, src/…/selector.py• L72-106, pyproject.toml• L41-42

The command-line list is a user-facing starting point. It includes Claude, OpenAI, and Ollama with common model choices. The runtime resolver also knows about Astraflow, Astraflow China, and Atlas, so applications can route to those providers even though the default interactive list does not show them. References: src/…/selector.py• L109-158, src/…/types.py• L17-24, src/…/resolver.py• L16-23

Provider resolution

Provider resolution turns a requested provider name into an initialized object that follows the common provider contract. The precedence order is explicit input, the LLM_PROVIDER process setting, the local .llm.env file, and then Claude as the default. Names from the environment are trimmed and normalized to lower case. References: src/…/resolver.py• L28-75

The routing path is therefore:

flowchart TD
    A[Caller or CLI] --> B{Explicit provider?}
    B -- Yes --> E[Resolve provider]
    B -- No --> C{LLM_PROVIDER set?}
    C -- Yes --> E
    C -- No --> D{.llm.env has provider?}
    D -- Yes --> E
    D -- No --> F[Use Claude]
    F --> E
    E --> G[Initialize provider adapter]

An unknown name fails during resolution instead of silently selecting another backend. Applications can add a provider to the resolver registry when it implements the shared contract. References: src/…/resolver.py• L62-79, src/…/interface.py• L11-30

Core execution model

The core model keeps application messages and model results independent from any one vendor format. Provider adapters receive the same request shape and return the same result shape. This lets higher-level code reason about content, model use, token use, stop state, and tool calls through one set of concepts. References: src/…/types.py• L10-24, src/…/types.py• L26-45, src/…/types.py• L96-147

Request and response types

Message is a conversation item with one of four roles: system, user, assistant, or tool. LLMInput groups the message history with an optional model, generation controls, tool definitions, streaming intent, and caller metadata. LLMOutput returns text plus optional tool calls, model identity, usage data, and a stop reason. References: src/…/types.py• L10-15, src/…/types.py• L26-45, src/…/types.py• L82-147

Tool data uses a shared three-part model. A ToolDefinition describes an action and its parameter schema. A ToolCall is the model's request to use that action. A ToolResult carries the action result or an error back into the conversation. The shared definitions can be serialized for provider APIs, which keeps the boundary between the runtime and an adapter clear. References: src/…/types.py• L48-94

Provider contract

LLMProvider defines the runtime's main provider boundary. Each adapter must generate a result, list its known models, and report whether its configuration is usable. It also reports tool and vision support and a default model. Common error classes identify authentication, rate limits, context limits, missing models, and tool execution problems for callers that need to react by category. References: src/…/interface.py• L11-60

The runtime maps provider-specific SDK responses into LLMOutput and maps common request data into the provider's request format. This is the main reason an application can change its route without changing its conversation model. See LLM providers and model selection for the provider choices and their capability differences. References: src/…/claude.py• L58-124, src/…/openai.py• L71-113, src/…/ollama.py• L60-100

Prompt normalization

Prompt normalization prepares a common message history for provider families with different prompt rules. PromptBuilder applies optional prompt settings to the message list and can merge a system instruction with an existing system message. It can also place tool descriptions in the system message when a provider needs a text-based form. Provider builders choose the appropriate policy before the request reaches an adapter.

References: src/…/builder.py• L11-79 src/…/builder.py• L104-132

Provider prompt adaptation

Claude and OpenAI use native tool descriptions, while Ollama's configured prompt path includes tool descriptions in the system message. A provider name without a special prompt policy uses the general defaults. The builder accepts either a complete prompt configuration or individual prompt options, but not both at the same time. This adaptation changes the presentation of the same intent; it does not change the application's Message or ToolDefinition concepts.

References: src/…/builder.py• L11-17 src/…/builder.py• L19-79 src/…/builder.py• L104-132

Named templates provide a separate shared store for reusable prompt text. Callers can register, replace, read, or remove a named template. The registry does not select a provider by itself. References: src/…/__init__.py• L5-41

Callable tool execution

Callable tools let a model request a named application action during a run. The runtime keeps the public tool description next to the callable that fulfills it, then turns each call into a result message that the model can use in its next step. See LLM prompts and tool execution for the detailed prompt and tool lifecycle. References: src/…/executor.py• L18-40, src/…/types.py• L48-94

Tool registry

ToolRegistry is the name-based bridge between model-facing tool definitions and Python callables. Registration replaces an existing entry with the same name, and the registry can list definitions for request construction. It does not run code or decide when a model should call a tool. References: src/…/executor.py• L21-40

ReAct execution

ReActAgent coordinates a bounded request-and-action loop. It sends the current history to the provider, returns when the model produces no tool calls, or executes every requested call and appends the assistant request and tool results before asking again. The loop returns a synthetic max_iterations result when it reaches its limit. References: src/…/executor.py• L71-122

The flow keeps model reasoning and application actions separate:

sequenceDiagram
    participant A as Agent
    participant P as Provider
    participant X as Tool executor
    A->>P: Send message history and tool definitions
    P-->>A: Text result or tool calls
    alt Text result
        A-->>A: Return final output
    else Tool calls
        A->>X: Execute named callables
        X-->>A: Return tool results
        A->>P: Send updated history
    end

An unknown tool or a callable exception becomes an error ToolResult linked to the original call. The agent can pass that result back to the model, so a failed action remains part of the conversation instead of escaping as an unstructured runtime failure. References: src/…/executor.py• L43-68, src/…/executor.py• L97-117

Change checklist

When these files or sections are changed, remember to consider these:

  • When adding or renaming a provider, update the provider type, resolver map, public provider exports, and any user-facing selection list that should expose it.
  • When changing a request or result field, update its provider serialization and response mapping so all adapters preserve the shared contract.
  • When changing tool schemas or call shapes, keep the shared definition, provider-specific serialization, registry lookup, and tool-result message aligned.
  • When changing provider prompt policy, check whether tool descriptions must remain native or move into the system message.
  • When changing saved configuration names or precedence, keep the CLI writer and resolver reader consistent.

Important failure modes

TriggerConsequenceGuard
Provider name is not in the resolver registryProvider creation raises ValueError and no model request startsUse a registered ProviderType or register the adapter before routing to it
Required provider credentials are missing or rejectedThe adapter raises an authentication error during validation or generationValidate the selected provider configuration before sending work
Request exceeds a provider context limitThe adapter raises a context-length error and returns no usable outputKeep message history and requested output within the selected model's limits
Model requests a name absent from the tool registryThe executor returns an error tool resultRegister every advertised tool with the exact model-facing name
A callable raises an exceptionThe executor converts the exception to an error tool resultTreat tool results with is_error as failed actions in the next model step
Agent reaches its iteration limitThe agent returns Max iterations reached with stop reason max_iterationsSet a suitable limit and handle this stop reason as incomplete work

Key files

FileWhy to read it
src/…/__init__.pyPublic runtime entry points and package-level abstractions
src/…/selector.pyInteractive provider and model setup flow
src/…/resolver.pyProvider precedence, lookup, and registration
src/…/interface.pyShared provider boundary and error categories
src/…/types.pyCommon message, request, result, model, and tool concepts
src/…/builder.pyProvider-aware prompt normalization
src/…/executor.pyCallable registration and bounded tool-call loop
src/…/claude.pyExample of a native provider adapter
src/…/openai.pyExample of a chat-completions provider adapter
src/…/ollama.pyExample of a local provider route