LLM request observability
LLM request observability records each model-provider attempt so scan operators can connect a scan action to its provider result. It preserves route, model, timing, usage, outcome, provider identifiers, and bounded diagnostic data without storing prompt or completion bodies. This page is a child of Model context management; Context budgets and compaction explains the context controls that shape the requests being observed.
Request event model
An LlmRequestEvent is the common record for one HTTP attempt. It covers both successful and failed calls across the LiteLLM route and native OpenAI-compatible routes. A retry is a new event, while a streamed call produces one event when the stream ends or is abandoned.
The event joins three kinds of information:
- Identity: route, provider, model, API host, provider request ID, response ID, scan agent, and retry attempt.
- Behavior: success or error outcome, status, streaming state, duration, time to first token, finish reason, and serialized request and response sizes.
- Usage and diagnosis: input, output, cached, and total tokens; cost when available; response headers; bounded request, response, LiteLLM, upstream, and error details.
The provider request ID is the key for external support or gateway investigation. Strix accepts provider-specific header names and can recover an ID from an error body when a gateway removes the response header. The API host keeps only the hostname, so endpoint paths and query values do not enter the event.
References: strix/…/request_log.py• L1-21 strix/…/request_log.py• L70-122 strix/…/request_log.py• L247-295 strix/…/request_log.py• L427-435
Request capture paths
The capture layer observes the two model routes at the boundary where each route exposes the most reliable request result. LiteLLM sends success and failure callbacks. The native model wrapper records the full attempt and reads status and headers from the shared HTTP client response hook.
The request lifecycle is:
flowchart TD
A[Agent turn] --> B[Bind agent and retry context]
B --> C{Model route}
C -->|LiteLLM| D[Receive LiteLLM callback]
C -->|Native OpenAI-compatible| E[Wrap model attempt]
E --> F[Read shared HTTP reply]
D --> G[Build bounded event]
F --> G
G --> H[Emit to log line and registered sinks]
H --> I[Scan log and operator-visible failure text]
The capture is installed when model defaults are configured. Installation is idempotent and also attaches the HTTP response hook once to the SDK shared client. For the LiteLLM route, the wrapper adds only abandoned attempts because normal outcomes already arrive through LiteLLM callbacks. This closes the gap for cancellation, shutdown, idle timeout, or a caller that closes a stream early.
References: strix/…/models.py• L659-710 strix/…/request_log.py• L189-220 strix/…/request_log.py• L852-917 strix/…/request_log.py• L991-1005 strix/…/request_log.py• L1165-1333
Correlation context
Correlation context connects a provider attempt to the scan agent and the replay of its turn. The agent execution loop binds the agent ID and name in the task that owns the run. Each event then receives that context, while the retry counter marks the first request as attempt zero and later replays with increasing values.
The context uses task-local state. This keeps concurrent agent tasks and concurrent provider calls from borrowing one another's labels. The HTTP reply holder follows the task awaiting the attempt, so a response header is attached to the correct event even when model calls run at the same time.
Operators can therefore read a sequence such as agent_id plus retry_attempt plus provider_request_id as one diagnostic chain:
- Find the agent and turn replay in the scan log.
- Compare the event outcome, status, timing, and upstream fields.
- Use the provider request ID with the provider or gateway when the failure needs external investigation.
When a model failure reaches scan status handling, failure_text adds a missing provider request ID to the agent's failure text. This gives the operator a useful external correlation key even when the exception exposes the ID only in headers.
References: strix/…/execution.py• L230-265 strix/…/execution.py• L769-772 strix/…/execution.py• L886-924 strix/…/request_log.py• L70-77 strix/…/request_log.py• L132-149 strix/…/request_log.py• L166-213 strix/…/request_log.py• L1413-1427
Diagnostic data boundary
Request observability is designed to explain provider behavior without becoming a prompt or completion store. It records serialized sizes instead of raw request and response bodies. It retains selected headers and structured summaries so operators can inspect limits, retries, routing, usage, and error shape.
All free-form data is bounded before it reaches an event. Strings, nesting, item counts, header counts, and total detail size have limits. If the detail object remains too large, it names the dropped top-level entries. Error messages are also capped. This keeps local logs and custom sinks usable during provider failures.
The boundary does not mean every diagnostic value is secret-free. Response headers and request parameters can contain deployment-specific data. The safe interpretation is that prompts and completions are excluded, endpoint query strings are reduced to host names, and diagnostic fields are bounded rather than fully redacted. Operators should treat the per-scan log as sensitive local output.
References: strix/…/request_log.py• L236-240 strix/…/request_log.py• L302-392 strix/…/request_log.py• L516-536 strix/…/request_log.py• L735-768 strix/…/request_log.py• L1387-1400
Scan log delivery
The built-in sink turns each event into one structured log line. Successful requests use debug level and failed requests use warning level. The line exposes the route, provider, upstream route when known, model, host, outcome, status, IDs, stream timing, sizes, token counts, cost, agent, retry attempt, and a short error description.
The scan logging setup writes debug records to the per-scan strix.log file in append mode. Each record also carries scan and agent fields from logging context. Console output is quieter by default and becomes more detailed when debug mode is enabled. Additional sinks can receive the typed event for another local destination, but sink failures are logged and do not fail the model request.
References: strix/…/request_log.py• L216-233 strix/…/request_log.py• L920-952 strix/…/logging.py• L20-47 strix/…/logging.py• L151-212
Streaming and failure behavior
Streaming calls are measured as one attempt from stream start until completion or interruption. The event records time to first token when available and preserves a finish reason for a completed or incomplete response. If the caller cancels the stream or closes it early, the event records an error outcome with the abandonment reason and any wire status or provider ID already seen.
Provider failures keep the status and headers when a response reached the client. A timeout or connection failure with no provider response has no fabricated status or request ID. This distinction lets operators separate an upstream rejection from a failure that occurred before the provider could identify the request.
Retries remain visible as separate events. The execution loop can replay transient model errors with backoff, while context overflow and image rejection use their own recovery paths. The request log therefore shows whether a failure was a one-off attempt or part of a repeated turn.
References: strix/…/request_log.py• L619-645 strix/…/request_log.py• L1048-1122 strix/…/request_log.py• L1239-1333 strix/…/execution.py• L851-924
Change checklist
When these files or sections are changed, remember to consider these:
- Keep LiteLLM callbacks and the native model wrapper complementary. A route change must not create duplicate normal events or lose abandoned attempts.
- Keep provider request-ID precedence aligned with the headers and error bodies exposed by each provider and gateway.
- Keep task-local agent and HTTP reply context bound in the owning task. Moving binding into SDK callback tasks can misattribute concurrent requests.
- Keep event size limits and the no-raw-body boundary intact when adding diagnostic fields.
- Keep the scan log level and format aligned with the operator workflow. Success detail belongs in debug output; failed requests must remain visible.
- Keep failure text correlation aligned with event correlation. A provider ID visible in an event should remain available to the operator handling the agent failure.
- Keep event outcome timing correct for streams. An early close must not look like a successful completed response.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| Native HTTP response hook is not attached | Status, headers, and provider request ID can be missing from native-route events | Install the hook on the SDK shared client and make installation idempotent |
| LiteLLM and the wrapper both emit normal outcomes | One provider attempt appears twice and usage can be overstated | Use LiteLLM callbacks for normal LiteLLM outcomes and the wrapper only for abandoned attempts |
| A provider ID is read from the wrong header or gateway marker | An operator cannot match the local failure to provider support records | Use the ordered provider header set and body fallback; do not treat CDN tracing IDs as provider request IDs |
| Task-local context is bound in an SDK callback task | Concurrent requests can receive the wrong agent or retry label | Bind context in the agent run task and keep reply state in the awaiting attempt |
| Raw prompt, completion, or endpoint query data is added to event details | Sensitive scan data can enter local logs or custom sinks | Record sizes and bounded summaries; keep only the hostname from endpoint URLs |
| A stream closes before its terminal event | The attempt can be shown as successful or disappear from diagnostics | Emit an abandonment error with the observed timing, status, and request ID |
| A connection or timeout error receives a synthetic HTTP status | Operators can mistake a pre-response failure for a provider rejection | Leave status and provider ID empty when no provider reply was received |
Key files
| File | Why to read it |
|---|---|
| strix/…/request_log.py | Defines the event contract, provider-ID correlation, bounded diagnostics, route capture, stream handling, and sinks. |
| strix/…/execution.py | Shows how agent identity and retry attempts enter request events and how provider failures affect scan execution. |
| strix/…/models.py | Shows when observability is installed and how model routes determine the capture path. |
| strix/…/logging.py | Defines per-scan file and console delivery, log context, and debug visibility. |
| strix/…/state.py | Shows the separate provider-attribution path used for streamed OpenRouter usage and cost. |
| tests/test_llm_request_log.py | Demonstrates the supported correlation, streaming, failure, size, privacy, and concurrency guarantees. |
| docs/model-context-management.md | Places request observability within the wider context budget, compaction, and warmup lifecycle. |
| docs/context-budget-and-compaction.md | Explains the request-size controls and recovery behavior that request events help diagnose. |