Introducing a new way to review your PRs.

Read the announcement

thedotmack / claude-mem

Persistent Context Across Sessions for Every Agent – Captures everything your agent does during sessions, compresses it with AI, and injects relevant context back into future sessions. Works with Claude Code, OpenClaw, Codex, Gemini, Hermes, Copilot, OpenCode + More

View on GitHub

47 documents · Updated Nov 20, 2026

docs / observation-generation-and-provider-runtime

Observation generation and provider runtime

Observation generation turns captured agent activity into durable observations and session summaries. The repository supports a local worker runtime that keeps a live provider conversation and a server runtime that processes queued generation jobs. Both runtimes protect private input, require a structured XML response, and preserve captured work when a provider cannot complete a request.

This page gives the generation view of the local runtime API. Read HTTP API and authentication for request and access boundaries, server jobs and process supervision for server job ownership, and transcript observation processing for captured activity and stored results. The interactive viewer client covers live observation delivery, while the smart file read gate, capture and memory, and media artifact lifecycle pages cover source controls and stored media.

Runtime roles

Local worker runtime

The local worker observes an active session as the agent runs. It places tool activity in a per-session buffer, starts a provider only when work exists, and sends one observation or summary request at a time. Provider choice can follow the session's platform source, so different sources can use different configured providers. Claude runs through a fresh SDK process for each observer generation, while OpenAI-compatible providers send the bounded conversation history with each request. A successful response is stored before the worker confirms the buffered messages.

This design keeps the source activity safe across provider pauses. A failed or incomplete turn leaves the messages pending so a later generator can process them. A missing Claude executable or an unserved Claude model is treated as setup trouble rather than as work that can succeed through repeated retries.

References: src/…/SessionManager.ts• L439-493 src/…/SessionManager.ts• L726-755 src/…/ClaudeProvider.ts• L393-522 src/…/OpenAICompatibleProvider.ts• L385-481 src/…/ResponseProcessor.ts• L984-1000

Server generation runtime

The server runtime handles observation generation as an asynchronous outbox job. It reloads the job and its source events from Postgres, checks that the queued scope and initiating key are still valid, selects a provider, persists the parsed result, and advances the job state. The queue payload is a dispatch hint; the database row remains the source of truth for scope and lifecycle.

The server path is separate from the local worker session store. It is suited to event-level generation and complete session summaries that can run after capture has ended. A stale-lock reclaimer returns work from a dead worker to the queue or marks it failed when no attempt remains.

References: src/…/ProviderObservationGenerator.ts• L58-73 src/…/ProviderObservationGenerator.ts• L177-259 src/…/reclaimStaleProcessingJobs.ts• L11-17 src/…/reclaimStaleProcessingJobs.ts• L36-67

Observation generation flow

Captured activity moves from a session buffer or a server generation job to a provider prompt, then through XML parsing and durable storage. The two runtimes use different conversation models, but they share the same purpose: produce useful memory without losing the source when generation fails.

The following diagram shows the common flow and the two execution paths.

flowchart LR
    A[Captured tool activity] --> B{Runtime}
    B -->|Local worker| C[Session buffer]
    B -->|Server| D[Generation outbox job]
    C --> E[Provider dispatch]
    D --> E
    E --> F[Prompt with private data removed]
    F --> G[Provider and model]
    G --> H[XML response]
    H --> I[Parse and classify]
    I -->|Valid| J[Store observations or summary]
    I -->|Skip| K[Confirm with no observation]
    I -->|Failure| L[Keep pending or schedule retry]
    J --> M[Search, events, alerts, and viewer updates]

Activity capture and batching

The local worker treats each tool event as pending work and also accepts an explicit summarize request. It suppresses duplicate queue entries, claims messages when a generator reads them, and confirms them only after storage succeeds. A session with no pending work does not open a paid provider turn.

The worker can add compatible observation events to one request. It bounds the combined prompt by the active model window and sends oversized tool fields through a separate condensation pass. The first event remains the anchor for the batch, so all claimed events can be linked to the response that stores them.

Server jobs use either one agent event or the events attached to a server session. A session-summary job bounds input by the prompt footprint of each event, then keeps the start and end of the activity when the size limit is reached. The opening gives the goal and the end gives the result or pending work.

References: src/…/SessionManager.ts• L475-540 src/…/SessionManager.ts• L839-917 src/…/OpenAICompatProvider.ts• L535-565 src/…/ProviderObservationGenerator.ts• L91-157 src/…/ProviderObservationGenerator.ts• L651-685

Provider dispatch

Dispatch chooses a configured provider that can serve the request. The local runtime can use Claude, Gemini, OpenRouter, Codex, or an OpenAI-compatible endpoint. A session-specific source mapping takes priority over the global provider when the mapped provider is configured. The server generation interface exposes Claude, Gemini, and OpenRouter adapters.

The selected provider can change when a provider is held by a quota or availability guard. A quota fallback defaults to Claude and can move local capture to another available provider; an explicit off setting disables it. A Claude run can use its other credential before switching providers. A cmem gateway rejection can also move local memory generation to the Claude plan for a limited window. After that window, one caller probes the gateway while other work stays on Claude until the probe resolves.

OpenRouter can send an ordered primary model and fallback model list when the request reaches OpenRouter itself. A custom OpenAI-compatible endpoint receives only the primary model because it may reject OpenRouter-only fields. The runtime resolves the endpoint, model, and credentials as one coherent tuple, so a gateway key is not sent to a personal endpoint or the reverse. Gateway error codes, unavailable models, mandatory-reasoning refusals, and input moderation refusals are classified before the shared retry rules act.

References: src/…/provider-dispatch.ts• L54-73 src/…/provider-dispatch.ts• L117-155 src/…/provider-dispatch.ts• L187-240 src/…/provider-dispatch.ts• L397-505 src/…/provider-dispatch.ts• L507-605 src/…/OpenRouterProvider.ts• L537-568 src/…/OpenRouterProvider.ts• L764-881

Prompt and response contract

Prompt construction

Prompts tell a model to extract durable information instead of copying the whole activity stream. They define the XML contract and distinguish an observation request from a complete session summary.

The local worker uses a multi-turn conversation. It starts with an initialization or continuation prompt, then sends observation prompts for tool activity and summary prompts for explicit session summaries. It waits for each answer before claiming more work, which prevents several unanswered requests from growing the conversation at once.

Each generation resolves a context window and scales its conversation and field budgets to that window. When a tool field is too large, the worker asks for a bounded condensation before it sends the observation prompt. OpenRouter can apply a reasoning effort to native OpenRouter observation turns, while wrap-ups and field condensation favor direct text output. Claude applies its configured thinking mode to the SDK session; if an Anthropic-compatible gateway rejects the disabled-thinking setting, the worker strips that setting for the next generation and retries the preserved batch.

The OpenAI-shaped providers place the observer instructions in a stable system anchor. They remove empty turns, merge adjacent turns with the same role, and always send a usable user turn. OpenRouter and compatible endpoints stream when supported, but callers receive one normalized completion result.

References: src/…/ClaudeProvider.ts• L494-522 src/…/ClaudeProvider.ts• L885-923 src/…/OpenRouterProvider.ts• L511-530 src/…/OpenRouterProvider.ts• L537-568 src/…/OpenRouterProvider.ts• L623-727 src/…/OpenRouterProvider.ts• L1000-1087 src/…/OpenRouterProvider.ts• L1286-1339 src/…/OpenAICompatibleProvider.ts• L285-357 src/…/OpenAICompatibleProvider.ts• L381-472 src/…/OpenAICompatibleProvider.ts• L534-650 src/…/context-window.ts• L263-308 src/…/context-window.ts• L363-410 src/…/streamed-chat-completion.ts• L1-23 src/…/prompt-builder.ts• L11-44 src/…/prompt-builder.ts• L46-137

Media input

Media input is an optional side channel for an observation request. The worker sends image derivatives only when the setting, endpoint, model, and request parameters all qualify; otherwise it sends the same turn as text and leaves the images uninspected.

The capability check is strict for OpenRouter and uses the gateway capability result for the cmem endpoint. The request has bounded image count, size, dimensions, and body size. If an image request is rejected before inference, the worker retries the turn once without images. It records which images the accepted request carried so later storage can link only inspected media to observations.

References: src/…/media-capability.ts• L11-24 src/…/media-capability.ts• L306-334 src/…/OpenRouterProvider.ts• L915-927 src/…/OpenRouterProvider.ts• L1046-1094 src/…/ResponseProcessor.ts• L348-389

Structured response processing

The parser accepts observation XML, summary XML, and the self-closing skip marker. A valid skip is a successful decision that the activity has no durable value; it is not the same as an empty provider response or invalid XML.

The local worker classifies provider refusal prose before adding it to the conversation. Context overflow recycles the conversation. Quota, authentication, transport, and setup failures return the claimed batch to pending. A non-XML answer to queued work earns one fresh-generation retry; a second unusable answer routes to a usable fallback when possible or drops the batch with a recorded failure. A schema-drifted XML reply can be salvaged and stored, then replaces the drifted history turn with canonical XML and reminds the next prompt of the schema; three consecutive drifted replies end the generation so the next one starts clean. An init reply does not create an observation.

A parseable summary that has only a request title or future plan is treated as answered but not as progress. It stores nothing, confirms the batch, and does not retry. Progress means text in investigated, learned, completed, or notes; the skip_summary marker remains a separate explicit outcome. A valid skip also releases the provider quota breaker, including the early confirmation path used before a memory session ID is known, because the provider has proved that it can serve the request even though it stored no memory.

Before storage, the worker adds file evidence from successful read, write, and patch tool calls to the model's file lists. It ignores calls marked as failed, denied, or timed out, so an attempted path is not treated as proof that a file was read or changed. The same evidence is added to summary records when a response has no observation blocks. When enabled for Claude sessions, optional line anchors add source-line context for supported successful file operations; anchor capture is best effort and never blocks storage.

Complete XML blocks are stored when a response hits its output cap, while an unfinished tail is reported as lost. A valid skip or a dropped unusable turn records its claimed media events before the worker confirms the batch. For stored turns, observation rows, media event results, and accepted media links share the storage boundary; a link failure degrades the media reference instead of replaying provider work. The worker confirms the batch only after that durable write. It then performs search sync, viewer broadcasts, alerts, and integrations as downstream effects that must not replay the stored work. When storage reuses an existing observation through deduplication, later delivery uses only newly inserted rows so the canonical row is not overwritten or announced again.

The server processor strips private tags again before persistence. It writes observations, source links, job state, lifecycle events, and audit records in one transaction. Generation keys and source links make a repeated job safe to run again. A summary job writes one summary-kind observation and can use observation blocks as a guarded fallback if a provider returns the wrong block type. Content made only of concepts is also retained as observation content instead of being silently dropped.

References: src/…/ResponseProcessor.ts• L54-100 src/…/ResponseProcessor.ts• L115-138 src/…/ResponseProcessor.ts• L147-299 src/…/ResponseProcessor.ts• L393-489 src/…/ResponseProcessor.ts• L521-747 src/…/ResponseProcessor.ts• L764-924 src/…/ResponseProcessor.ts• L928-1123 src/…/ResponseProcessor.ts• L1140-1201 src/…/summary-content.ts• L1-11 src/…/processGeneratedResponse.ts• L78-131 src/…/processGeneratedResponse.ts• L197-263 src/…/processGeneratedResponse.ts• L285-495 src/…/processGeneratedResponse.ts• L563-580

Downstream observation delivery

After local storage, the worker updates optional vector search and emits observation or summary events for the viewer. The delivery contract includes observation metadata and validated media references. Storage results distinguish new rows from deduplicated rows, so downstream sync and live events describe only data that this turn actually inserted. SQLite remains the durable read source, while Chroma adds semantic matches and falls back to SQLite keyword search when it is unavailable or returns too few usable results.

Alerts and external integrations use only newly stored rows, so a deduped result does not send a second notification. These side effects are not allowed to turn a stored observation back into pending work. Observation feeds add validated media references and hide raw media metadata; project file paths are shortened before they leave the worker.

The server path records provider and model identity with the observation and source link. Its committed result is then available to the server APIs and other consumers of Postgres.

References: src/…/DatabaseManager.ts• L34-112 src/…/SearchManager.ts• L447-662 src/…/SearchManager.ts• L989-1029 src/…/PaginationHelper.ts• L48-120 src/…/types.ts• L11-64 src/…/ResponseProcessor.ts• L1001-1123 src/…/ResponseProcessor.ts• L1215-1403 src/…/processGeneratedResponse.ts• L341-429

Provider failure and retry behavior

Failure classes

Providers report failures by meaning rather than by raw status code. The runtime separates temporary upstream or network failures, rate limits, spent quotas, invalid credentials, setup faults, context overflow, invalid provider data, and other permanent errors.

The server also records the likely result of the paid call. A rate limit means the provider refused work before generation. A network or timeout failure is ambiguous because the provider may have completed and billed the request. An unusable response is an output failure, so resending it can pay twice. This distinction controls whether the same work is retried, paused, routed to another provider, or marked failed.

Claude adds setup and model checks around the SDK process. A missing or unspawnable executable, an unserved model, and a permanent request error do not benefit from repeated session retries. A gateway rejection of the thinking setting is narrower: the worker removes that setting and preserves the batch for a fresh generation.

OpenRouter's gateway can provide a structured error code. The worker maps allowance exhaustion, invalid keys, inactive subscriptions, rate limits, upstream outages, and bad requests into the shared classes. A retired or unknown model and a moderation refusal are request-specific failures. They do not mean that all future observation work should pause.

Failure classRuntime response
Transient network or upstream faultRetry within the paid-send bound or preserve the batch for a later generator.
Rate limit with a wait hintWait for the provider hint, then make a bounded resume attempt.
Quota exhaustedHold requests behind a cooldown or use a configured fallback.
Invalid auth or setupPause until credentials or local setup changes.
Context overflowRecycle the local conversation or fail the server job as non-retryable.
Invalid JSON or XMLTreat the response as invalid output and use the runtime's recovery path.
Provider output failure after a billed responseDo not resend the same server call. Record a terminal failure.

References: src/…/ClaudeProvider.ts• L131-287 src/…/ClaudeProvider.ts• L991-1044 src/…/OpenAICompatibleProvider.ts• L737-769 src/…/provider-errors.ts• L1-31 src/…/OpenRouterProvider.ts• L218-397 src/…/error-classification.ts• L10-38 src/…/error-classification.ts• L40-75 src/…/error-classification.ts• L111-201

Local retry and pause behavior

Local HTTP providers retry temporary and rate-limit failures within one request, with a small maximum retry count, exponential delay, jitter, and provider-supplied wait hints. Key pools can rotate to another key when the failure means that the current key is spent or blocked.

Streaming changes the paid-send boundary. It uses a configurable idle-silence window within the normal request cap. A network or idle failure before any output may be retried once. A failure after output starts is treated as an output failure and is not resent. An endpoint that rejects streaming fields is remembered and receives plain requests for the rest of the process. The cmem gateway remains non-streamed and uses its normal deadline.

An HTTP provider response that was accepted and billed but cannot be read is classified as an output failure. The runtime does not resend it. The current batch follows the output-failure path while later buffered work can continue. Context overflow recycles the conversation and retries the batch in a fresh generation.

OpenRouter also supports a bounded reasoning-budget retry for an empty answer that ends at the output cap. The retry forces reasoning off and counts against the same paid-send budget. An already reasoning-disabled request gives up instead of sending the same request again. If the endpoint rejects the reasoning-off field because reasoning is mandatory, one resend omits that field. Image-request rejection has a separate pre-inference path that resends the turn once as text-only.

When those attempts end, the session runner keeps the claimed activity for a later generation when the failure is transport-related. It schedules bounded resumes for a rate limit, response stall, conversation recycle, schema retry, or gateway fallback. Quota, auth, and setup pauses wait for a cooldown or user repair instead of creating a request storm. A non-XML answer to queued work gets one fresh-generation retry; a second such answer drops that batch with an error rather than looping forever.

Requests to the cmem gateway carry a bounded, ID-only correlation value. It identifies the logical request, retry attempt, and a small set of source event keys without sending prompt text, paths, session IDs, or credentials.

References: src/…/retry.ts• L162-176 src/…/retry.ts• L179-277 src/…/ClaudeProvider.ts• L1073-1107 src/…/OpenAICompatibleProvider.ts• L746-769 src/…/OpenAICompatibleProvider.ts• L791-895 src/…/OpenRouterProvider.ts• L1000-1087 src/…/OpenRouterProvider.ts• L1120-1138 src/…/OpenRouterProvider.ts• L1286-1339 src/…/streamed-chat-completion.ts• L1-23 src/…/streamed-chat-completion.ts• L36-61 src/…/streamed-chat-completion.ts• L293-353 src/…/streamed-chat-completion.ts• L366-495 src/…/cmem-request-correlation.ts• L1-47 src/…/GeneratorRunner.ts• L241-345 src/…/GeneratorRunner.ts• L432-491 src/…/ResponseProcessor.ts• L649-917

Server job retry behavior

The server bounds each provider call with a deadline. A timeout becomes a transient provider error, while an empty response and unparseable XML become terminal outcomes because rerunning the same invalid result does not repair it.

Retryable server failures move the outbox job back to queued until its attempt limit is reached. The delay grows from seconds to minutes and then caps. A rate limit is not treated as a paid generation and keeps the job's normal attempt allowance. An ambiguous failure can be retried only within the server's small paid-send bound. A recognized output failure is terminal because sending the same work again can charge twice. Non-retryable errors move the job to failed, and every failure records its classification and reason in the job lifecycle.

If a worker dies after taking a job into processing, the stale-lock sweep moves it back to queued and republishes the BullMQ job when attempts remain. It marks the row failed when the attempt limit is spent or when republishing fails. The sweep locks and rechecks rows so it does not reclaim live work.

Provider adapters apply provider-specific rules before the shared mapping. Gemini distinguishes a temporary 429 from an exhausted period quota and carries either an HTTP wait hint or the provider's structured delay. It treats invalid keys and unsupported requests as non-retryable. OpenRouter maps network, HTTP, and body-level errors into the same server classes and preserves empty or malformed output for the job processor to handle.

References: src/…/ProviderObservationGenerator.ts• L84-88 src/…/ProviderObservationGenerator.ts• L290-325 src/…/ProviderObservationGenerator.ts• L342-405 src/…/processGeneratedResponse.ts• L145-194 src/…/processGeneratedResponse.ts• L583-587 src/…/reclaimStaleProcessingJobs.ts• L36-67 src/…/reclaimStaleProcessingJobs.ts• L93-136 src/…/GeminiObservationProvider.ts• L113-167 src/…/OpenRouterObservationProvider.ts• L107-165 src/…/error-classification.ts• L20-38

Worker session execution

Generator lifecycle

The session route starts a generator only when a session has buffered work. A per-session start lock prevents concurrent starts from claiming the same queue, while provider selection and cooldown checks decide which provider may send the next request.

The generator opens with an init or continuation turn, then drains observation and summarize messages. It records which provider and model produced each answer, and it ends when the queue is empty, the provider pauses, the conversation must be recycled, or the session is deleted. Provider selection is resolved when the generator starts, including source-specific routing and any active fallback. A provider switch occurs at a message boundary so one request is not split across provider histories. The Claude feed also waits for the current answer before it pulls more work, and a stream that ends without a result releases the claimed batch for transport recovery.

References: src/…/SessionRoutes.ts• L179-235 src/…/SessionRoutes.ts• L238-273 src/…/provider-dispatch.ts• L397-505 src/…/ClaudeProvider.ts• L569-715 src/…/ClaudeProvider.ts• L971-1044 src/…/OpenAICompatibleProvider.ts• L381-472 src/…/OpenAICompatibleProvider.ts• L483-505 src/…/OpenAICompatibleProvider.ts• L543-665 src/…/SessionManager.ts• L839-917

Session completion

Session completion is separate from observation storage. The worker marks the local session complete and broadcasts that state after the session has ended, while observation persistence happens when each provider answer is accepted.

This separation lets the runtime retain partial observations and pending work even when the session process stops. Session deletion also aborts the generator, waits for its subprocess within a bound, and disposes of the in-memory buffer. A spent paid-send budget parks its batch instead of resending it, while later work can continue when possible.

References: src/…/SessionCompletionHandler.ts• L7-35 src/…/SessionManager.ts• L79-143 src/…/SessionManager.ts• L585-741

Change checklist

When these files or sections are changed, remember to consider these:

  • Keep the prompt block type and the matching response processor aligned. Event jobs need observation blocks, while session-summary jobs need a summary block.
  • Keep the summary progress fields and the parser's summary schema aligned. A title or future plan alone must remain distinct from progress and must not enter a retry loop.
  • Keep provider dispatch, source-specific mappings, cooldown state, fallback routing, gateway probe claims, and provider-health reporting aligned. A new provider must not be selected without an availability and failure policy.
  • Preserve the claim, store, and confirm order in the local worker. Confirming before durable storage can silently lose captured activity.
  • Preserve the server outbox state transition, idempotency keys, paid-send outcome, and stale-lock recovery when changing response persistence. A retry must not create duplicate observations or source links, and an ambiguous or billed output failure must not cause an unsafe resend.
  • Keep privacy stripping at both prompt creation and response persistence boundaries. A change to one boundary must not make private content visible downstream.
  • Keep tool-name and tool-result error handling aligned with file-evidence extraction. Failed or denied calls must not add attempted paths to observation or summary file lists.
  • Keep optional line-anchor capture gated by the user setting and Claude source check. Anchor lookup must remain best effort so a source-read failure cannot replay or block a stored batch.
  • Keep file-evidence extraction, deduplication flags, and observation ordering aligned with storage result IDs. Position changes can attach a tool-use link, media link, or viewer event to the wrong observation.
  • Keep media capability checks aligned with the request body. If a new request field is sent, image qualification must require it before images are delivered.
  • Keep stream liveness, pre-output retry, output-failure classification, reasoning-budget retry, mandatory-reasoning recovery, and paid-send accounting aligned. A stream or reasoning retry must not resend a billed result outside its allowed boundary.
  • Keep Claude executable and model classification aligned with session-start health reporting. A permanent setup or model error must not enter an unbounded transport retry loop.
  • Keep OpenRouter model fallbacks, endpoint-specific fields, and key/base pairing aligned. Native OpenRouter fields must not reach strict custom gateways, and a fallback model must not repeat the failed primary.
  • Keep OpenRouter and OpenAI-compatible endpoint correlation ID-only. Source event keys must remain bounded and must not be replaced with prompt or session content.
  • Keep context-window resolution, field condensation limits, output caps, and recycle behavior aligned. A smaller model window must not receive the old fixed-size prompt budget.
  • Keep optional Chroma sync and SQLite fallback independent from observation confirmation. A search index failure must not replay or discard a stored observation.
  • Keep media-event result writes in the same durability boundary as observation storage. A failed linkage write must degrade the media link, not ask the provider to inspect the event again.
  • Keep schema-drift correction aligned with parser output and conversation history. Salvaged observations must use canonical XML before the next prompt, and repeated drift must end the generation instead of growing a bad history.
  • Keep output-cap handling aligned with tool-use commitment. Complete blocks may be stored, but an unfinished response must not be replayed as if the provider had not answered.
  • Keep quota-breaker release aligned with the definition of a successful provider turn. A valid skip_summary response must release the breaker even when it stores no memory or completes before a memory session ID exists.

Important failure modes

TriggerConsequenceGuard
A local provider returns quota or auth prose as if it were a normal answerThe batch can be confirmed or the worker can keep sending doomed requestsOutput classification resets the batch to pending and arms the matching pause or cooldown.
Claude cannot launch its executable or the selected model is not servedThe session can retry a permanent setup error instead of giving useful repair adviceClaude classifies spawn and model refusal errors as setup or unrecoverable failures.
A Claude gateway rejects the configured thinking parameterThe batch can be dropped as an ordinary empty responseThe worker strips the rejected parameter once, preserves the batch, and retries the next generation.
A response exceeds the conversation or model context windowThe next request can fail again because it reuses the same oversized historyThe local runtime recycles the conversation and seeds the next generation from stored observations.
A streamed provider fails after output has startedResending can charge for a generation that already ranStream processing marks the result as an output failure and does not retry it.
An accepted provider response is billed but its body is unusableThe same batch can be charged twice if the request is resentThe local provider classifies the unreadable body as an output failure and prevents a resend.
An OpenRouter reasoning turn ends empty at the output capThe model can spend the answer budget on reasoning and leave no observation textThe runtime allows one same-budget retry with reasoning disabled, then gives up.
An endpoint rejects the default reasoning-off fieldA valid endpoint can be marked unusable and the queued batch can be droppedOpenRouter resends once without the reasoning field, then applies the normal failure path.
A configured OpenRouter model is retired or unavailableEvery request can fail even though the key and endpoint are validModel-unavailable responses are classified as non-retryable and direct the operator to choose a served model.
A server summary contains large event payloadsThe provider can reject the request for context overflow before it can write a summaryThe server measures prompt-sized event blocks and keeps the head and tail within a byte budget.
A server provider returns empty text or invalid XMLThe job has no usable observation and repeated queue attempts only repeat the bad resultThe processor marks the job failed as a terminal empty-response or parse outcome.
A summary contains only a request title or future planThe provider answer is accepted but no summary is stored, so expected progress can be absent without a retryThe local progress gate requires text in investigated, learned, completed, or notes.
A provider returns a billed body-level output failureA retry can charge for the same generation twiceThe server error model marks the outcome as output failure and stops the resend.
A Gemini 429 is treated as quota exhaustion without reading its quota windowA temporary throttle loses its wait hint or a spent allowance is retried as if it were temporaryThe Gemini adapter checks the structured quota details before applying shared 429 handling.
A queue message is delivered to two server workersTwo paid provider calls can be made for one generation jobThe outbox lock skips rows already in processing, and the persistence transaction is idempotent.
A worker dies while a job remains in processingBullMQ redelivery and operator retry can both leave the row stuck without a new provider callThe stale-lock sweep atomically reclaims old locks, republishes available jobs, and fails jobs that cannot be republished.
A queue payload has a different tenant scope or its API key is revokedThe worker could generate data under the wrong tenant or after access was withdrawnThe server reloads the canonical row, compares scope, checks key status, audits the violation, and refuses generation.
A provider fails after local messages are claimedCaptured activity can disappear from the next generationThe failure path resets or preserves claimed messages until storage confirms them.
A provider returns schema-drifted XML repeatedlyThe model can copy the wrong tags in later turns or keep producing unusable outputThe processor stores salvageable rows, replaces the history turn with canonical XML, and starts a fresh generation after three consecutive drifts.
A response reaches the output cap with an unfinished observation blockThe complete part can be lost or the same billed tool use can be replayedThe processor stores complete blocks, reports the lost tail, and commits the answered tool uses.
A failed or denied tool call includes a file path in its inputThe observation can claim that the agent read or changed a file when the tool did not completeFile evidence ignores tool calls marked as errors before it adds paths to observations or summaries.
Line-anchor capture cannot read a source file or its setting is malformedStored observations lose optional source-line context, but the batch must still completeAnchor capture is opt-in, limited to Claude sessions, and catches lookup errors without retrying storage.
Deduplication reuses an existing observation rowSync or SSE can overwrite or repeat the canonical observationStorage reports inserted rows separately, and downstream delivery skips merged rows.
A skipped or dropped turn has media eventsImages can remain pending or be sent for inspection again after the source work is confirmedThe processor records skipped event results and capture-only links before it confirms the batch.
An image request is rejected after capability checksThe turn can fail or the same media can be sent repeatedlyThe worker sends the turn once without images and marks the images uninspected.
A search index or media side effect fails after storageA valid observation can appear missing from one read path or trigger a replaySQLite storage and batch confirmation finish first; indexing and notifications remain non-critical.
The selected cmem gateway stops serving the accountMemory capture can remain stuck between a gateway and an unavailable providerDispatch records a bounded fallback to Claude and admits only one later gateway probe.
A provider answers a probe with valid skip_summary but the quota breaker stays armedThe worker keeps probing instead of returning to the provider's normal pathResponse processing releases the breaker for a valid skip on both the early and stored-turn paths.

Key files

FileWhy to read it
src/…/SessionManager.tsShows how local sessions buffer, claim, confirm, pause, resume, park, and delete captured work.
src/…/SessionRoutes.tsShows the gate that starts generators and serializes starts for one session.
src/…/GeneratorRunner.tsShows provider selection at execution time and the session-level failure and resume policy.
src/…/OpenAICompatibleProvider.tsShows shared multi-turn execution, context limits, summary routing, and preservation of queued work.
src/…/OpenAICompatProvider.tsShows custom OpenAI-shaped endpoint choice, response use, and endpoint-specific failure handling.
src/…/ClaudeProvider.tsShows the local Claude prompt feed, SDK process lifecycle, error classification, and the one-answer-at-a-time session loop.
src/…/OpenRouterProvider.tsShows endpoint-safe model fallback dispatch, reasoning controls, media turns, streaming, error classification, and usage reporting.
src/…/provider-dispatch.tsShows global and source-specific provider selection, quota fallback, gateway recovery, and single-probe behavior.
src/…/streamed-chat-completion.tsShows stream liveness and the boundary between retryable transport failure and billed output failure.
src/…/ResponseProcessor.tsShows response classification, summary progress gating, schema-drift correction, output-cap handling, file-evidence extraction, optional line-anchor capture, durable local storage, media linkage, deduplication handling, quota-breaker release, and downstream delivery.
src/…/summary-content.tsDefines which summary fields count as progress before the worker stores a summary.
src/…/types.tsDefines storage results and the live observation or summary event contracts used after processing.
src/…/retry.tsShows request retry rules, deadlines, backoff, and rate-limit wait handling.
src/…/context-window.tsShows model-window lookup and the prompt, field, and output budgets derived from it.
src/…/media-capability.tsShows when an observation request may carry images and the bounds applied to them.
src/…/cmem-request-correlation.tsShows the bounded ID-only link between gateway attempts and source media events.
src/…/DatabaseManager.tsShows initialization of the shared local store, optional vector search, and media services used after generation.
src/…/SearchManager.tsShows semantic search, SQLite fallback, and retrieval of stored observations.
src/…/PaginationHelper.tsShows the paged observation feed and the path sanitization applied to returned file evidence.
src/…/ProviderObservationGenerator.tsShows canonical job loading, scope checks, provider deadlines, source loading, and server persistence dispatch.
src/…/prompt-builder.tsShows the server prompt contract, privacy filtering, event size bounds, and summary split.
src/…/ClaudeObservationProvider.tsShows the server provider adapter and its Anthropic response and error boundary.
src/…/GeminiObservationProvider.tsShows the Gemini adapter, model endpoint, response extraction, and quota, region, and rate-limit classification.
src/…/OpenRouterObservationProvider.tsShows OpenRouter and OpenAI-compatible endpoint handling, request controls, and response extraction.
src/…/error-classification.tsShows shared provider failure classes, retry hints, and the paid-send safety boundary.
src/…/processGeneratedResponse.tsShows parse outcomes, transactional persistence, idempotency, and server retry state changes.
src/…/reclaimStaleProcessingJobs.tsShows how dead worker locks return jobs to BullMQ or become terminal failures.
src/…/generation-jobs.tsShows the job state rules and the lock-safe stale processing reclaim used by the server runtime.