Server jobs and process supervision
Server jobs turn accepted events and ended sessions into durable work that can run outside an HTTP request. Process supervision keeps the local server, its generation sessions, and their child processes safe to start, observe, pause, and stop. For the wider service boundary, see Local runtime and API service; for request and access rules, see HTTP API and authentication; for generation and provider behavior, see Observation generation and provider runtime; for the live client, see Interactive viewer client; for protected file reads, see Smart file read gate.
Job system
Job identity
The generation job is the durable link between a source event or ended session and the observations that the server later creates. Its identity includes tenant scope, source kind, and source record, so retries and process restarts refer to the same work rather than create new work.
Queue messages carry tracing and audit context such as the job ID, actor, source adapter, and request ID. The durable job record remains the authority for scope, permission, and current state.
The system uses separate event and summary job kinds. Deterministic queue IDs prevent duplicate delivery for the same source while keeping different tenants and source records separate.
References: src/…/types.ts• L9-52 src/…/job-id.ts• L6-29
Durable queueing
The server records a generation job in Postgres before it publishes work to BullMQ. Postgres is the system of record. BullMQ or Valkey delivers and runs the work. If publishing fails, the durable record still shows that work was accepted but not yet delivered.
Event and session services use the same record-then-publish boundary. A queue outage therefore returns a queued-only result instead of claiming generation completed. A bounded wait can ask for a terminal result, but reaching the wait limit does not cancel the job.
This diagram shows how accepted input becomes durable work and then generated memory.
flowchart LR
I[Event or ended session] --> T[Postgres transaction]
T --> J[Durable generation job]
T --> S[Source record]
J --> Q[Event or summary queue]
Q --> W[Generation worker]
W --> C[Reload and scope check]
C --> P[Provider and result storage]
P --> J
P --> M[Generated observations]
References: src/…/IngestEventsService.ts• L89-153 src/…/EndSessionService.ts• L70-146 src/…/ServerJobQueue.ts• L21-28 src/…/ServerV1PostgresRoutes.ts• L78-115
Job lifecycle
Jobs move from queued to processing and then to completed, failed, or cancelled. Attempts and lock data show ownership of the current run. Lifecycle and audit records make the path from acceptance to outcome visible.
Before provider work starts, a worker reloads the canonical job and checks its payload, scope, key status, and current state. Retryable provider failures can return work to the queue within the attempt limit. Bad payloads, scope errors, and revoked keys do not retry.
Operators can inspect jobs and can retry or cancel them. Retry is idempotent. Cancellation blocks later side effects even when an old queue delivery arrives.
The active generation worker serves both event and summary lanes. It also reclaims processing rows that remain stale after a worker failure and records a scoped audit event when BullMQ marks a job stalled. These recovery actions are best effort and do not make a stalled job look completed.
References: src/…/generation-jobs.ts• L15-24 src/…/generation-jobs.ts• L164-223 src/…/ProviderObservationGenerator.ts• L69-207 src/…/ActiveServerGenerationWorkerManager.ts• L68-110 src/…/ActiveServerGenerationWorkerManager.ts• L120-149 src/…/ActiveServerGenerationWorkerManager.ts• L151-193 src/…/ServerV1PostgresRoutes.ts• L603-718
Server runtime services
Service roles
The server can run as an HTTP service, a generation worker, or both. The HTTP role accepts and reads memory data. The generation role consumes event and summary queues and writes generated observations. Both roles use the same Postgres records and queue names, so they can scale and restart independently.
Startup builds the database, queue, and generation boundaries. Missing provider credentials or a disabled queue create an explicit disabled generation state. They do not make the server report that generation is active. A split deployment can keep generation out of the HTTP process and run it in a separate worker process.
The active generation manager connects the queue lanes to the observation generator only when the queue and provider are available. Its health state reports whether this generation boundary is disabled, active, or closed.
References: src/…/create-server-service.ts• L177-239 src/…/types.ts• L41-82 src/…/ActiveServerGenerationWorkerManager.ts• L42-110 src/…/ActiveServerGenerationWorkerManager.ts• L195-218 src/…/ServerService.ts• L286-305 src/…/ServerService.ts• L767-805
Worker initialization
The local worker opens its HTTP listener before all background services finish loading. Health and readiness routes remain available during startup, while most work returns 503 until the database and search path are ready. This lets a launcher distinguish a live process from a usable service.
Background setup loads settings, checks local dependencies, opens SQLite, and starts search and corpus services. It also runs log retention as non-blocking maintenance. Core readiness does not wait for every optional service. Sync, transcript capture, telemetry, and diagnostic checks can continue after the worker can serve core memory requests.
References: src/…/worker-service.ts• L455-514 src/…/worker-service.ts• L516-565 src/…/worker-service.ts• L631-663 src/…/log-retention.ts• L18-34
Database boundary
The worker uses one shared SQLite connection as the local source for sessions, observations, media, and search. DatabaseManager creates the session store, media store, and search view together so request handling and generation see the same local state.
Optional vector search attaches to that boundary only when enabled. Cloud sync is active only when its token, user identity, and hub URL are complete. Inactive installs do not create an outbox that no service can drain. A local device identity is established before cloud setup so later sync uses the same local origin. Conflict recovery and media replication use that shared boundary, while media cleanup retries durable cleanup work in the background. Media cloud deletion can run when sync credentials exist; media upload also requires its capture setting. Optional backups start only when enabled.
Without vector search or cloud sync, local storage remains usable with SQLite-only search. Search can combine semantic results with local keyword results and can fall back to SQLite when the semantic service is unavailable. Closing the database stops optional media sync, text sync, backup, conflict recovery, and cleanup services before the shared connection closes.
References: src/…/DatabaseManager.ts• L25-70 src/…/DatabaseManager.ts• L71-139 src/…/DatabaseManager.ts• L142-168 src/…/SearchManager.ts• L42-65 src/…/SearchManager.ts• L489-662 src/…/PaginationHelper.ts• L56-121
Provider boundary
The worker presents Claude and OpenAI-compatible providers through one session flow. Each provider receives the same session context and pending observations, returns normalized content and usage data, and reports errors through shared outcome classes.
The provider boundary separates account or endpoint setup from session work. OpenRouter can use its managed service or a custom compatible gateway. The separate OpenAI-compatible path can use a hosted endpoint or a local server, with authentication required only when the selected endpoint needs it. Its model list and request fields stay specific to the endpoint, while OpenRouter fallback models remain limited to OpenRouter requests.
Provider calls use classified errors to separate retryable transport and rate-limit failures from quota, authentication, setup, and permanent request failures. Streamed endpoints use token or ping activity as their liveness signal. A failure before output can be retried within the batch budget; a failure after output is treated as already-paid work and is not resent. The shared retry path honors a provider Retry-After value when present and applies a bounded per-attempt deadline. When a retryable failure still reaches the session boundary, the session preserves its pending work instead of treating the failure as completed or discarding the batch.
The provider sends a bounded, ID-only correlation value to the managed gateway when needed. It identifies the logical request, attempt, and a small set of media source keys without sending prompts, paths, session IDs, or credentials.
Provider generation uses images only after the endpoint and model capability checks succeed. If the check fails, the observation continues as text-only and the image remains uninspected.
References: src/…/OpenAICompatibleProvider.ts• L123-194 src/…/OpenAICompatProvider.ts• L465-510 src/…/OpenAICompatProvider.ts• L703-765 src/…/OpenRouterProvider.ts• L631-711 src/…/OpenRouterProvider.ts• L749-866 src/…/streamed-chat-completion.ts• L1-23 src/…/streamed-chat-completion.ts• L293-350 src/…/cmem-request-correlation.ts• L1-47 src/…/media-capability.ts• L11-24 src/…/media-capability.ts• L306-334
Session generation
Session work units
An active session is the in-memory work unit for one observed IDE session. It holds the session identity, pending messages, provider state, conversation history, and exit state while the database holds the durable session and memory records.
The session message buffer is process-local work state rather than a durable queue. It deduplicates tool-use IDs for the life of the worker and claims one message at a time. It waits for new input or an idle timeout. A batch whose paid-send budget is spent is parked and is not sent again. The transcript and stored prompt history remain the recovery source after a worker restart, so the buffer does not attempt to resurrect in-flight claims across processes.
Session initialization refreshes durable project and provider metadata when a cached session is reused. A new process clears the old provider memory-session ID and rebuilds prompt context from local records, including the latest stored user prompt when a new hook does not provide one. At each message boundary, the session rechecks whether observer capture is enabled. Disabling capture discards the remaining in-memory buffer while keeping delivery identities deduplicated. Bounded media references travel with observation messages, while media identity remains in durable media records.
References: src/…/worker-types.ts• L9-89 src/…/SessionManager.ts• L257-405 src/…/SessionManager.ts• L496-563 src/…/SessionManager.ts• L565-623 src/…/SessionManager.ts• L917-997 src/…/SessionMessageBuffer.ts• L21-41 src/…/SessionMessageBuffer.ts• L42-71 src/…/SessionMessageBuffer.ts• L169-207
Generator coordination
Each session has at most one active generator. Session routes serialize concurrent start requests for the same session, select the provider at start time, and release provider probes when a run does not start. Automatic transport recovery also uses one shared probe at a time, with ready sessions kept in FIFO order. If a hook-driven generator starts first, the automatic path waits for that run to settle and rearms only when the preserved work still needs recovery.
Generator exit uses one abort category as the coordination contract. Quota and authentication exits discard buffered input and wait for fresh activity. Rate-limit, overflow, provider-switch, transport, output-retry, and schema-drift exits preserve the session and its buffered work. Idle and unclassified failures normally finalize the session. A small number of bounded rescue paths protect work that arrived while an idle generator was stopping or preserve one final summary after an unclassified failure.
Provider changes wait for an active response to finish. A generator still parked for a concurrency slot can be aborted and cleaned up before the new provider starts. A provider switch at a message boundary preserves the buffer and starts a new run with the selected provider. The same boundary ends a Claude simple-tier run when a later message needs the full model, or ends a fallback run when the primary credential can serve again. Setup failures pause the session behind a dependency gate rather than repeatedly finalizing it. Generator exits release their claims and can schedule a bounded resume for conversation recycling, transport recovery, rate limits, response stalls, or repeated response errors.
The exit reason has two uses. Its category selects whether work survives, while a closed normalized value records safe telemetry without exposing free-form detail. This keeps pause behavior and monitoring aligned when a reason carries a provider-specific suffix.
References: src/…/SessionRoutes.ts• L74-88 src/…/SessionRoutes.ts• L125-326 src/…/SessionRoutes.ts• L328-583 src/…/SessionManager.ts• L108-250 src/…/SessionManager.ts• L917-997 src/…/GeneratorRunner.ts• L138-222 src/…/GeneratorRunner.ts• L400-630 src/…/GeneratorExitHandler.ts• L28-123 src/…/abort-reason.ts• L3-82
Generation continuity
Each provider starts with session context that comes from stored memory. The session can therefore continue after a worker restart, provider switch, quota pause, or conversation recycle without relying on one provider process remaining alive forever.
The provider feed admits one unanswered prompt at a time. A finished SDK turn opens the gate for the next message. A response stall fences late frames, returns the claimed batch to pending, and allows only a bounded number of delayed restarts. Transport failures use paced resumes with a capped backoff and jitter. A batch whose paid-send budget is spent is parked rather than sent again. Unattended resumes on the managed gateway share a bounded budget, so a failing gateway cannot cause endless paid retries. SessionManager runs at most one automatic transport probe at a time, keeps ready sessions in order, and does not let a declined start or a hook-driven run consume the slot without settling the resume state.
Subscription quota events are kept as process-local snapshots by window. Rejected or near-exhausted subscription windows stop background generation, while API-key users continue because their calls use per-call billing. The provider records a new exhaustion event once per reset window rather than once per request.
The generator retires a conversation before it exceeds its character budget. The budget and field limits scale with the selected model's context window. Known OpenRouter models use cached catalogue data; unknown or offline endpoints use a conservative fallback. The generator also reacts to provider context-overflow errors. The pending batch returns to the session buffer, and a fresh generator starts with prior stored context. Repeated failed recycles pause the session instead of creating an automatic restart loop.
Oversized tool fields receive one bounded compression attempt before they enter the main conversation. Image payloads are removed before the size check. If compression fails, times out, or still exceeds the limit, the normal bounded truncation path keeps processing.
References: src/…/response-pacer.ts• L19-119 src/…/response-pacer.ts• L121-227 src/…/SessionManager.ts• L84-250 src/…/SessionManager.ts• L820-902 src/…/SessionManager.ts• L917-997 src/…/GeneratorRunner.ts• L530-630 src/…/OpenAICompatibleProvider.ts• L359-425 src/…/OpenAICompatibleProvider.ts• L546-580 src/…/ClaudeProvider.ts• L265-320 src/…/ClaudeProvider.ts• L334-385 src/…/ClaudeProvider.ts• L565-618 src/…/ClaudeProvider.ts• L724-909 src/…/RateLimitStore.ts• L100-140 src/…/RateLimitStore.ts• L192-211 src/…/RateLimitStore.ts• L221-281 src/…/recycle-conversation.ts• L68-145 src/…/field-optimizer.ts• L76-131 src/…/context-window.ts• L242-297 src/…/context-window.ts• L299-360
Response admission
ResponseProcessor decides whether provider output is safe to admit into durable memory. It treats quota, rate-limit, authentication, transport, and context-overflow responses as failures to preserve rather than as conversation turns. Ordinary non-XML idle output is a confirmed no-op, so low-signal replies do not create a respawn loop.
For valid output, the processor derives file evidence from the claimed tool batch and stores observations or summaries before it confirms those messages. It then triggers optional vector and cloud sync and live broadcasts. A storage failure leaves the claim unconfirmed so the batch can be retried.
This diagram shows the admission order that protects pending work and durable memory.
flowchart LR
B[Claimed session batch] --> R[Provider response]
R --> D{Response class}
D -- quota, auth, transport, overflow --> P[Return batch to pending]
D -- idle or plain text --> N[Confirm no-op]
D -- valid structured output --> S[Store observations or summary]
S --> C[Confirm claimed messages]
S --> X[Optional sync and broadcasts]
References: src/…/ResponseProcessor.ts• L283-447 src/…/ResponseProcessor.ts• L450-639 src/…/ResponseProcessor.ts• L750-916 src/…/SessionManager.ts• L547-600
Session shutdown
Session shutdown fences delayed resumes, aborts the generator, waits for it within a bounded period, waits for its provider child when possible, and asks the supervisor to reap session-owned processes. It then disposes the pending buffer and removes the active session.
Shutdown also fences delayed transport resumes and scheduled restart timers. If teardown fails before removal, the session and its buffered work stay available. A pre-existing transport pause can be restored with its backoff position, while quota, authentication, overflow, and provider-switch pauses remain under their own guards. Session-end telemetry is flushed while the session still exists.
Session-end wrap-up is separate from ordinary stop behavior. A session-end request marks a short grace period so a summary can land first. The wrap-up then runs in the background and remains visible as active work for worker idle checks. If shutdown removes the session first, the wrap-up still uses the durable session record and does not block teardown.
References: src/…/SessionManager.ts• L418-489 src/…/SessionManager.ts• L626-741 src/…/SessionManager.ts• L743-813
Process supervision
Process ownership
The process registry is the shared record for managed workers, SDK children, and MCP processes. It records process type, session ownership, start time, process group, and live child state. This lets shutdown target a session or process class without treating every child as one undifferentiated tree.
The registry persists enough identity to remove dead entries after an interrupted run. It reserves SDK slots before spawn, which keeps concurrent sessions inside the configured pool and hard process limit.
References: src/…/process-registry.ts• L20-34 src/…/process-registry.ts• L104-204 src/…/process-registry.ts• L464-570
Worker spawning
The worker spawner first checks the PID record, health endpoint, readiness endpoint, and port. It starts a worker only when no usable worker owns the port. A cross-launcher spawn lock makes concurrent start requests wait for one worker instead of racing to create two.
The start result distinguishes a ready worker, a live worker that is still warming, and a dead worker. If a dead worker left a known sidecar listening on the port, safe ghost-port recovery can remove that listener before the next start. A failed boot can return bounded diagnostic text to the caller.
References: src/…/worker-spawner.ts• L72-121 src/…/worker-spawner.ts• L123-167 src/…/worker-spawner.ts• L175-233 src/…/worker-spawn-gate.ts• L7-27
Worker startup
Worker startup records process ownership and begins listening before background readiness. It detects whether the prior run stopped cleanly by combining a stale PID record with a clean-shutdown marker. The marker is consumed on the next start so one old marker cannot mislabel a later crash.
The worker also checks configured local tools before generation. Missing Claude or vector-search support produces a focused degraded dependency state instead of making the whole process appear dead.
References: src/…/worker-service.ts• L151-199 src/…/worker-service.ts• L422-453 src/…/worker-service.ts• L468-505 src/…/worker-service.ts• L516-544
Worker shutdown
Shutdown is a guarded sequence shared by signals and administrative commands. The first request marks shutdown in progress, stops background loops, records the clean-run marker, performs a bounded graceful drain, and closes managed resources. Later requests do not repeat the sequence.
Restart has a separate handoff contract. The old worker waits for its port to become free, removes only its own or a confirmed-dead PID record, and starts the successor as its last action. Restart verification requires a new process identity and readiness. A health response from the dying worker is not enough.
References: src/…/worker-service.ts• L864-927 src/…/worker-service.ts• L880-926 src/…/worker-service.ts• L960-1025
Shutdown cascade
The supervisor snapshots each managed process and its descendants before graceful shutdown. It sends a soft stop, waits for a bounded time, then force-stops survivors and reaps descendants that outlive their root. Platform-specific process-tree rules preserve the same stop contract on Unix and Windows.
Process start identity is checked again before forceful signals. This prevents a reused PID from authorizing a kill against an unrelated process. Once the cascade starts, new child spawns are rejected, which closes the race between shutdown completion and a late SDK or MCP spawn.
References: src/…/index.ts• L104-139 src/…/shutdown.ts• L15-137 src/…/shutdown.ts• L200-301 src/…/kill-process-tree.ts• L101-112 src/…/process-identity.ts• L143-187
Health and environment boundaries
Health states
Health checks answer different questions: whether the listener responds, whether core work is ready, and whether supporting dependencies are active. Worker startup uses health and readiness probes so a live process that is still warming is not reported as ready.
Runtime status adds active sessions and queue information when the queue can be sampled. The supervisor periodically removes dead process records. Provider quota snapshots also expose the latest state for each subscription window. Queue depth, quota pressure, and queue sampling errors remain separate signals so pressure is not confused with an outage.
The generation boundary reports its own state separately from listener health. A closed manager is an error state, while a manager that has not started is disabled rather than ready for generation.
References: src/…/HealthMonitor.ts• L86-124 src/…/health-checker.ts• L5-34 src/…/ServerService.ts• L53-99 src/…/ActiveServerGenerationWorkerManager.ts• L195-209 src/…/ServerJobQueue.ts• L38-52 src/…/RateLimitStore.ts• L78-98
Environment isolation
Managed child processes receive a filtered environment. The filter blocks host Claude Code session settings, session markers, and credentials that could change SDK or helper behavior. It preserves provider and proxy settings that the child needs.
Provider generation creates its intended credentials before applying the filter. The SDK therefore receives the worker's authentication context rather than inheriting the host CLI context. The same boundary applies to server daemons, SDK processes, MCP checks, and process identity probes.
References: src/…/env-sanitizer.ts• L1-60 src/…/worker-service.ts• L768-790 src/…/worker-service.ts• L1000-1025 src/…/ClaudeProvider.ts• L282-320 src/…/ClaudeProvider.ts• L640-665
Settings boundary
The worker settings boundary controls which local runtime choices can change through the settings API. It exposes masked values for known secrets, accepts only an allowlist of operational settings, and validates provider, bind address, port, and path values before saving them. Browser writes must come from the same loopback host and port. This protects the unauthenticated local settings path without blocking trusted CLI and hook clients that do not send an origin.
The boundary keeps executable spawn paths and most service tokens in file or environment configuration. It also exposes dependency health and MCP state so a local operator can see whether supporting services are available and enable or disable the MCP search service.
References: src/…/SettingsRoutes.ts• L21-67 src/…/SettingsRoutes.ts• L81-127 src/…/SettingsRoutes.ts• L129-245 src/…/SettingsRoutes.ts• L247-257 src/…/SettingsRoutes.ts• L259-391
PID ownership
PID state identifies a process with its port, start time, and platform start token when available. Start and stop commands check liveness and ownership before acting. This prevents a reused PID from being treated as the old worker and prevents an old process from deleting a successor's state.
Repeated status checks may use a cached identity observation. Kill authorization reads the operating system again before signaling. State files are removed only by the owning process or after the recorded owner is confirmed dead.
References: src/…/process-registry.ts• L67-140 src/…/process-identity.ts• L143-187 src/…/index.ts• L172-215 src/…/worker-service.ts• L912-925
Change checklist
When these files or sections are changed, remember to consider these:
- Keep the durable job row, payload, deterministic queue ID, and event and summary enqueue paths aligned. A mismatch can strand work or create duplicate generation.
- Keep worker authorization based on the canonical job record. Queue payload identity is for comparison and audit.
- Keep HTTP-only and generation-worker roles consistent with the queue setting. A disabled generation boundary must not report active provider work.
- Keep readiness separate from liveness. A worker can answer health checks while its database and search services are still warming.
- Keep the process-local message buffer and transcript replay boundary explicit. Do not add a restart path that treats lost in-memory claims as durable queued work.
- Keep capture-setting checks at message boundaries. Disabling capture must not allow an already-running generator to drain the remaining RAM-only buffer.
- Keep one unanswered-prompt pacing, stall fencing, claimed-message reset, paid-send budget, and bounded resume as one contract. Changing one can duplicate a late response or strand a pending batch.
- Keep automatic transport resumes single-flight and ordered. A new resume path must not compete with a hook-driven start, rearm after an explicit pause, or consume more unattended gateway budget than intended.
- Keep abort categories, normalized telemetry values, preserved-work behavior, and exit finalization aligned. A new category must not preserve work in one path and discard it in another.
- Keep setup-required provider exits behind their dependency gate. They must release provider probes and must not schedule a retry loop.
- Keep provider error classes, retry rules, quota snapshots, and provider availability checks shared by status and session startup. Otherwise status can claim one provider while the next session uses another or can retry a failure that should pause work.
- Keep subscription quota thresholds and API-key exemption aligned with rate-limit event extraction. A wrong event shape can disable the guard while leaving health data stale.
- Keep the LLM timeout and stream idle settings bounded and resolved at request time. A slow local backend must not cause an unbounded wait or an in-loop retry storm.
- Keep conversation recycle limits, context-window fallbacks, and field compression limits bounded. A new provider path must not create an unbounded restart loop.
- Keep image capability checks and text-only fallback aligned with endpoint and model changes. Unsupported media must remain uninspected.
- Keep correlation headers ID-only, bounded, and limited to the managed gateway. They must not become a second channel for prompts, paths, or credentials.
- Keep image removal, one compression attempt, and truncation fallback in the field-size path. Otherwise the compressor can receive data that the final prompt would not carry or can become a second retry loop.
- Keep spawn locks and supervisor spawn guards on every worker, SDK, MCP, and helper launch path.
- Keep the event and summary queue lanes attached to the same generation boundary. Stale-job recovery and stalled-job audit must remain non-fatal to the worker.
- Keep environment sanitization after provider credentials are assembled. Reversing that order can leak host credentials or remove the intended worker identity.
- Keep port-free confirmation, PID ownership checks, and successor spawn in the restart order. Reversing them can block a successor or erase its state.
- Revalidate process identity before forceful signals and use the shared process-tree teardown. A direct kill path can target a reused PID or leave descendants alive.
- Keep provider endpoint and credential sources coherent. A personal key must not be sent to the managed gateway, and custom gateways must not receive OpenRouter-only request fields.
- Keep file evidence and summary evidence derived from the claimed batch. Storing a result without its batch context can mislabel durable memory.
- Keep the settings API allowlist, secret masking, origin check, and value validation aligned with the worker's spawn and provider settings. A new setting can otherwise become an unsafe unauthenticated write or reach a child process with the wrong type or identity.
- Keep executable paths and protected service tokens outside the HTTP write contract. Moving them into the settings allowlist changes the trust boundary for process launch and service access.
- Keep log retention limited to old, inactive daily logger files. A broad cleanup rule can remove active, legacy, or unrelated files.
- Keep semantic search fallback and pagination on the shared database boundary. A search-service outage must not make local memory reads unavailable.
- Keep cloud sync activation, device identity, media sync, conflict recovery, backup, and database close order aligned. A partial sync configuration must not create undrainable local work or outlive the shared database.
- Keep session wrap-up work separate from session removal. The end marker must not make ordinary stop requests send a notification, and background delivery must remain visible to worker idle checks.
- Keep session initialization's durable prompt refresh and provider-boundary resets aligned. A reused session must not send stale prompt metadata, memory-session identity, or model-tier work to the next generator.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| Postgres commit succeeds but queue publish fails | Work stays queued and no result is claimed. | The durable job row is canonical and the enqueue result reports queued-only. |
| A queue payload has the wrong tenant or project | Generation stops without a retry. | The worker reloads the durable job and checks scope before provider work. |
| A worker sees a job already processing | A second worker does not make another provider call. | Processing ownership is checked before generation. |
A worker crash leaves a generation row in processing | The job can remain unavailable after the queue has declared it stalled. | The active generation manager runs a bounded stale-row sweep based on queue lock age and provider-call time. |
| A BullMQ job becomes stalled but its outbox row is missing | No misleading audit record is created. | Stalled-job auditing looks up the durable row and treats missing or failed audit data as best effort. |
| Worker restarts with messages still in the in-memory buffer | Buffered work is lost from that process. | The transcript is the recovery source; the buffer does not resurrect claims across restarts. |
| Capture is disabled while a generator is draining messages | Remaining RAM-only work is processed after the setting says capture is off. | The session rechecks capture at each message boundary and clears the buffer when disabled. |
| Provider quota, authentication, rate-limit, or transport failure occurs | The batch stays pending or is discarded according to the failure class, and the session pauses after allowed retries. | Classified errors select bounded retry or preservation behavior. |
| A provider setup gate remains active | Repeated starts waste provider calls and can mask the real dependency fault. | Setup-required exits pause the session, release probes, and wait for repair. |
| A subscription rate-limit event reports rejection or near-exhaustion | Background generation consumes no more of that subscription window. | Window snapshots apply thresholds and provider rejection rules; API-key auth is exempt. |
| A provider request exceeds its per-attempt deadline or stream idle window | The current request stops without multiplying load against a slow backend. | The timeout becomes a transient classified error and the session preserves buffered work. |
| A provider prompt remains unanswered | The current batch returns to pending and late frames are ignored. | The response pacer fences the stream and allows only bounded delayed resumes. |
| Two automatic transport resumes become ready together | Sessions can compete for provider admission or spend budget out of order. | SessionManager runs one unattended probe at a time and keeps ready work in FIFO order. |
| A hook-driven generator starts while an automatic transport resume is ready | The automatic path can occupy a second probe or lose its preserved-work retry. | The resume waits for the running generator and rearms only after the generator settles while the same session and buffer remain current. |
| Response-stall resumes are exhausted | Buffered work is discarded instead of restarting forever or waiting for an event that may never come. | The stall counter is bounded; the next captured event can start fresh work. |
| Unattended managed-gateway resumes keep failing | The worker stops spending gateway retries and waits for the next hook. | Gateway resumes share a bounded per-session budget. |
| Provider context remains over budget after recycle | New input can repeatedly start work that cannot succeed. | Context-aware budgets and consecutive overflow tracking pause the session after bounded recycle attempts. |
| An oversized field cannot be compressed | The observation loses detail through the normal truncation path but does not block generation. | Compression has one timeout-bounded attempt and falls back to truncation. |
| Provider returns ordinary non-XML idle output | The claimed batch is discarded as a confirmed no-op. | The response path confirms idle output and does not add respawn debt. |
| Storage fails after a batch is claimed | The same work remains available for a later generator. | Messages are confirmed only after durable storage succeeds. |
| Two HTTP requests start one session at once | Two generators can race for the same session or provider probe. | Per-session start calls are serialized and provider claims are released on non-start paths. |
| Two launchers request a worker at once | Workers can race for the same port. | The cross-launcher spawn lock gives one launcher ownership. |
| A dead worker leaves a safe sidecar on its port | New starts can report a dead worker forever. | Ghost-port reclaim acts only after owner death and listener checks. |
| Shutdown begins while a child spawn is pending | A child can survive the shutdown cascade. | The supervisor rejects new spawns once shutdown starts. |
| A stale PID names a reused process | Start or stop can affect an unrelated process. | PID checks include process start identity and forceful teardown re-reads it. |
| Provider credentials or host session flags leak to a child | The child can use the wrong account or request settings. | Credential setup and shared environment sanitization define the child boundary. |
| A browser sends a settings write from another loopback port | A local page can change worker behavior through the unauthenticated settings path. | Browser writes require a matching loopback host and port. |
| A settings response exposes a stored secret | A caller can read a provider or service credential from the local API. | Known secret keys are masked before the response leaves the process. |
| A settings client posts an unchanged secret mask | The stored secret could be replaced by the mask text. | The save path recognizes only the exact mask of the stored value as unchanged. |
| An HTTP client tries to set an executable path or protected token | A settings write could redirect process launch or change a service identity. | Executable paths and protected tokens remain file- or environment-only. |
| Custom gateway receives OpenRouter-only fields | The gateway can reject a valid generation request. | Endpoint classification gates fallback and usage fields to OpenRouter. |
| Worker starts without a required local tool | Generation fails later with a broad error. | Dependency preflight records the missing tool as focused degraded health. |
| Successor starts before the old PID state is safely removed | The old worker can make the live successor appear stopped. | PID cleanup is owner- or dead-process-guarded during restart handoff. |
| A stream fails after output starts | The same paid request may be sent twice and duplicate work. | Output-started failures are marked as output failures and are not resent. |
| An endpoint cannot accept image input or its capability check is unavailable | The observation must not inspect or send its image data. | Capability failures use a text-only request and leave images uninspected. |
| A semantic search service is unavailable | Search can lose semantic ranking but local reads remain possible. | The worker falls back to SQLite keyword search and keeps pagination on the local database. |
| Log retention sees an old daily filename that is still active | The worker can lose a file that the logger still writes. | Retention checks both UTC age and file modification time and skips non-daily names. |
| Cloud sync settings are only partly configured | Local mutations can accumulate with no service able to acknowledge them. | DatabaseManager enables sync work only for a complete configuration and purges undrainable outbox state when sync is off. |
| Media cleanup or optional sync continues while the database closes | Background work can use a closed connection or leave local media state inconsistent. | DatabaseManager stops cleanup, media sync, text sync, conflict recovery, and backup before closing SQLite. |
| Teardown fails while a session has preserved work | A later event can lose the session or its pending buffer if cleanup removes it too early. | The deletion fence keeps the session alive and restores only an eligible prior transport pause. |
| A session-end wrap-up is requested during teardown | The summary notification can be lost or an ordinary stop can send one by mistake. | Only the session-end marker starts the grace timer; delivery uses the durable session and runs in the background. |
| A cached session keeps old prompt or provider state after durable session data changes | The next generator can use stale context or remain on a provider or model tier that no longer matches the session. | SessionManager refreshes durable metadata on reuse and ends incompatible provider or tier runs at a message boundary. |
Key files
| File | Why to read it |
|---|---|
| src/…/types.ts and src/…/job-id.ts | Define the durable job payload and stable queue identity. |
| src/…/ServerJobQueue.ts | Defines queue delivery, worker lifecycle, counts, and queue fault signals. |
| src/…/IngestEventsService.ts | Shows how accepted events become durable generation work. |
| src/…/EndSessionService.ts | Shows how ended sessions create summary work. |
| src/…/ProviderObservationGenerator.ts | Shows the worker-side job trust check and generation outcome. |
| src/…/ActiveServerGenerationWorkerManager.ts | Connects event and summary queues to generation, reports generation health, and recovers stale or stalled jobs. |
| src/…/create-server-service.ts and src/…/ServerService.ts | Show runtime role selection, split worker mode, lifecycle, and status. |
| src/…/worker-service.ts | Shows worker startup, readiness, environment isolation, crash detection, and shutdown handoff. |
| src/…/DatabaseManager.ts | Shows the shared local database boundary, sync activation, media services, backup, conflict recovery, cleanup, and close order. |
| src/…/SearchManager.ts and src/…/PaginationHelper.ts | Show local search fallback, scope-aware reads, timelines, and paginated memory access. |
| src/…/SessionManager.ts and src/…/SessionMessageBuffer.ts | Define session initialization, durable prompt refresh, buffering, claims, capture gates, transport resumes, provider-boundary changes, paid-send parking, activity signals, wrap-up work, and child cleanup. |
| src/…/GeneratorRunner.ts and src/…/GeneratorExitHandler.ts | Define provider-run exit handling, setup gates, preserved work, bounded resumes, and session finalization. |
| src/…/abort-reason.ts and src/…/response-pacer.ts | Define abort categories, telemetry mapping, one-at-a-time prompt admission, and stall limits. |
| src/…/SessionRoutes.ts | Coordinates session HTTP input, generator admission, provider switching, and bounded resumes. |
| src/…/OpenAICompatibleProvider.ts | Shows the shared multi-turn provider lifecycle, response preservation, context limits, and continuation behavior. |
| src/…/OpenAICompatProvider.ts | Shows custom and local OpenAI-shaped endpoint selection, error classes, and request handling. |
| src/…/OpenRouterProvider.ts | Shows managed gateway and OpenRouter routing, model fallbacks, media input, and usage handling. |
| src/…/streamed-chat-completion.ts | Shows stream liveness, output-paid failure handling, and compatibility fallbacks. |
| src/…/cmem-request-correlation.ts | Defines the bounded gateway request correlation contract. |
| src/…/context-window.ts | Defines model-window lookup, safe fallbacks, and prompt-size budgets. |
| src/…/media-capability.ts | Defines when observation images may be sent and the text-only fallback. |
| src/…/retry.ts | Shows classified provider retries, per-attempt deadlines, backoff, and rate-limit delays. |
| src/…/ClaudeProvider.ts | Shows isolated SDK work, process slots, quota checks, paced prompts, and conversation input. |
| src/…/RateLimitStore.ts | Defines provider quota snapshots, health-window views, and subscription abort rules. |
| src/…/recycle-conversation.ts | Defines conversation retirement, context reseeding, and overflow pause limits. |
| src/…/field-optimizer.ts | Defines bounded field condensation and the truncation fallback. |
| src/…/ResponseProcessor.ts | Shows response admission, durable storage, file evidence, and message confirmation. |
| src/…/log-retention.ts | Shows safe, non-blocking pruning of inactive daily worker logs. |
| src/…/worker-spawner.ts | Shows liveness checks, spawn locking, warming results, and ghost-port recovery. |
| src/…/process-registry.ts | Shows process ownership, session grouping, slot control, and child tracking. |
| src/…/index.ts and src/…/shutdown.ts | Show spawn refusal, signal handling, descendant shutdown, and cleanup. |
| src/…/process-identity.ts and src/…/kill-process-tree.ts | Show PID ownership checks and cross-platform tree termination. |
| src/…/env-sanitizer.ts | Defines the environment boundary for managed children. |
| src/…/HealthMonitor.ts | Defines worker health, readiness, port, and shutdown probes. |
| src/…/SettingsRoutes.ts | Defines the local settings, dependency-health, and MCP control boundary. |