Remote worker capture
Remote worker capture keeps hook events available when the local processing path is unavailable. Deferred events use a durable spool and a remote acceptance boundary, while some live observer events use a bounded direct handoff. The worker accepts deferred events before it starts normal observation or session work, then applies bounded live processing with recovery rules for provider outages. This keeps capture separate from the hook's small time budget and lets later processing use the same ingest path as local capture. See Capture and memory lifecycle, CLI hook event ingestion, Transcript observation processing, Memory ingestion and reinforcement, and Native memory bridge.
Capture boundary
The capture boundary is the point where a hook stops depending on immediate worker availability. Deferred capture accepts observations, file edits, summaries, session ends, and advisor calls as one durable event stream. The event keeps its session identity, source, payload, and capture time so delayed processing can still attach it to the right work. Live observer events can instead use a bounded direct handoff when the hook can reach the worker within its time budget.
Deferred hook-event spool
The hook spool is a file-backed hand-off for short host hooks. Each event has its own atomic file, and its stable name makes repeated delivery replace or identify the same event instead of creating an unbounded duplicate. The spool sorts events by their enqueue time and keeps newer work safe when a drain stops part way through.
The spool is not searchable memory. It is an outbox that protects capture until a worker accepts the event. Accepted events then enter the normal ingest path, which can queue observation generation or session work.
References: src/…/hook-spool.ts• L9-32 src/…/hook-spool.ts• L35-92 src/…/hook-spool.ts• L252-327
Capture-time ordering
The spool carries the time at which the hook saw an event. Worker ingestion uses that time when it links a delayed observation to the prompt that was active during capture. Direct live handoffs carry the same time value. This prevents an outage from moving an old tool action under a later prompt.
Summary and session-end events wait behind retained observations for the same session. An observation can create the session that those closing events need, so this order prevents a temporary outage from stranding the session close.
References: src/…/hook-spool-drain.ts• L17-54 src/…/hook-spool.ts• L362-393 src/…/shared.ts• L1-112 src/…/live-observer-event.ts• L62-73
Remote transport
Remote transport moves the durable outbox from a client process to a worker over HTTP. The client sends one versioned envelope to POST /api/spool/ingest, and the worker acknowledges only after it has written the event to its own spool.
Transport selection
The transport uses the filesystem for loopback workers and HTTP for workers on another host. The setting CLAUDE_MEM_HOOK_SPOOL_TRANSPORT can force either choice. In automatic mode, loopback addresses stay local while other hosts use the remote path. A token is required on both sides before remote capture is enabled. Live observer events select the same local or remote worker boundary, but they do not create a disk outbox.
References: src/…/hook-spool-remote.ts• L6-8 src/…/hook-spool-remote.ts• L56-74 src/…/SettingsDefaultsManager.ts• L168-184 src/…/SettingsDefaultsManager.ts• L400-414
Live observer handoff
The live observer handoff is a short direct request for an event that is already in the hook process. Remote delivery uses the versioned spool endpoint, while loopback delivery uses the session routes for observations and summaries. The hook waits only for a bounded handoff and does not keep a disk copy or retry an ambiguous result, so deferred spooling remains the reliable path when delivery cannot be confirmed.
References: src/…/live-observer-event.ts• L9-18 src/…/live-observer-event.ts• L33-46 src/…/live-observer-event.ts• L48-79
Delivery acknowledgement
The worker validates the protocol version, event shape, and bearer token before it writes anything. It returns the event's deterministic receipt only after the server spool write and its receipt marker succeed. The client removes its copy only when the acknowledgement names that exact receipt, so a lost response or a wrong response leaves the event available for retry.
This boundary gives remote delivery at-least-once transport with duplicate suppression. The worker's receipt marker makes a repeated request harmless even when the client did not receive the first acknowledgement.
References: src/…/HookSpoolRoutes.ts• L17-47 src/…/hook-spool.ts• L121-123 src/…/hook-spool.ts• L166-172 src/…/hook-spool.ts• L319-327
Remote delivery flow
The diagram shows how a captured event crosses the client and worker boundaries before normal memory processing begins.
flowchart LR
H[Short-lived hook] --> S[Client durable spool]
S --> U[Detached uploader]
U -->|versioned HTTPS request| R[Worker ingest route]
R --> D[Worker durable spool]
D --> I[Shared worker ingest]
I --> M[Queued observation or session work]
U -. retry on timeout or bad acknowledgement .-> S
References: src/…/spool-hook-event.ts• L23-75 src/…/HookSpoolRoutes.ts• L21-46 src/…/hook-spool-drain.ts• L17-54
Upload scheduling
Upload scheduling keeps deferred remote work outside the hook's main action. A hook records the event first, then requests a detached uploader. The hook only waits for the short scheduling attempt; it does not wait for remote ingestion or model work. The separate live handoff path waits for a bounded request instead of starting an uploader.
Burst coalescing
Many events can arrive during one agent turn. The client shares one uploader across a burst and records that a follow-up is needed when another event arrives during delivery. This limits process creation while still giving late events a later upload pass.
Each upload pass has a short transport budget and a bounded batch size. Older entries go first, and entries that do not fit remain in the outbox for the next hook or later retry.
References: src/…/spool-hook-event.ts• L14-21 src/…/spool-hook-event.ts• L30-75 src/…/spool-hook-event.ts• L77-143 src/…/hook-spool-remote.ts• L6-9
Retry scheduling
An unavailable worker, missing credential, failed process launch, timeout, or invalid acknowledgement keeps the client event in place. The next hook can trigger another upload, so the capture path does not need a long-running client daemon. The worker also has its own drain triggers after an event arrives.
References: src/…/spool-hook-event.ts• L23-29 src/…/spool-hook-event.ts• L49-73 src/…/spool-hook-event.ts• L101-143 src/…/spool-hook-event.ts• L145-193
Handoff timeout
The live handoff timeout is capped below the shortest host hook deadline. An invalid or oversized value falls back to the default or to the safe maximum. A timeout or ambiguous result is logged and skipped rather than resent because the worker may already have accepted the event.
References: src/…/live-observer-event.ts• L9-30 src/…/live-observer-event.ts• L33-46 src/…/live-observer-event.ts• L48-78
Worker ingestion
Worker ingestion turns both local spool files and remote spool files into the same capture actions. The worker does not need a separate remote processing model; remote delivery ends at durable acceptance, after which the normal worker ingest functions decide whether to queue an observation, summary, advisor call, or session close. Queue acceptance is not the same as durable memory storage: the live processing buffer may still need a provider run before the event becomes an observation or summary.
Drain triggers
The worker drains at boot, after a remote ingest request, after a local nudge, when the spool directory changes, and during a periodic safety sweep. A single-flight drain prevents concurrent passes. A new request during a pass schedules one more pass instead of running two drains at once.
References: src/…/worker-service.ts• L469-484 src/…/worker-service.ts• L542-555 src/…/hook-spool-drain.ts• L75-105 src/…/hook-spool-drain.ts• L108-157
Ingest hand-off
The drain marks an event when the shared ingest path has accepted it. The marker is written before later generator work begins. If the worker stops after acceptance but before file removal, the next worker removes the marked event without sending it again. If the worker stops before acceptance, the event remains available for a full retry.
References: src/…/hook-spool.ts• L329-347 src/…/hook-spool.ts• L395-466 src/…/SessionStore.ts• L2042-2069
Live processing buffer
The live processing buffer holds accepted work only while the worker can process it. It uses per-session and global bounds, removes expired unsent work, and suppresses repeated tool-use identities during the worker lifetime. The buffer is not a restart-safe backlog; the transcript remains the durable source for recovery workflows.
References: src/…/SessionMessageBuffer.ts• L47-65 src/…/SessionMessageBuffer.ts• L75-105 src/…/SessionMessageBuffer.ts• L267-317 src/…/SessionMessageBuffer.ts• L349-361
Processing recovery
Processing recovery keeps accepted work from blocking newer events when a provider pauses. The session manager preserves pending batches, schedules paced transport resumes, and runs one unattended probe at a time. Provider dispatch can route work to a configured fallback while the selected provider is held. A streamed request is retryable only when it fails before output starts; a failure after output starts is not resent because the provider may have charged for it.
References: src/…/SessionManager.ts• L108-149 src/…/SessionManager.ts• L158-235 src/…/provider-dispatch.ts• L193-223 src/…/provider-dispatch.ts• L487-559 src/…/streamed-chat-completion.ts• L1-23 src/…/streamed-chat-completion.ts• L367-495
Worker memory access
After ingestion stores memory, worker read paths expose observations, summaries, and prompts with project, source, and session context. Search can use semantic results with keyword fallback, while timeline and paged reads keep results bounded and preserve the matched record's project scope. These read paths let delayed capture become part of the same memory view as local capture.
References: src/…/SearchManager.ts• L124-193 src/…/SearchManager.ts• L501-723 src/…/ProgressiveMemorySearch.ts• L26-63 src/…/ProgressiveMemorySearch.ts• L65-127 src/…/PaginationHelper.ts• L58-123
Retention and quarantine
The worker keeps temporary failures for retry and moves malformed or permanently rejected events to corrupt/ instead of deleting them. Events that never become acceptable move to expired/ after the retry window. A session end for a session that never registered has a shorter grace period because it cannot succeed after the worker has confirmed that the session is absent.
References: src/…/hook-spool.ts• L131-146 src/…/hook-spool.ts• L427-455 src/…/hook-spool.ts• L524-547 src/…/hook-spool-drain.ts• L35-50
Change checklist
When these files or sections are changed, remember to consider these:
- Keep the remote envelope schema, protocol version, receipt value, and worker acknowledgement in sync.
- Preserve the distinction between the client spool receipt marker and the worker's ingest hand-off marker.
- Keep transport selection aligned with worker host settings so loopback delivery does not require HTTP credentials and remote delivery does not use an unshared filesystem.
- Preserve capture timestamps and the ordering rule that keeps summaries and session ends behind retained observations.
- Keep the live handoff timeout below the host hook deadline and do not turn an ambiguous direct handoff into an automatic resend.
- Preserve the live buffer limits and its distinction from the durable spool. Do not make accepted in-memory work look restart-safe.
- Keep provider fallback and stream retry outcomes aligned with paid-send behavior. Do not resend a request after provider output has started.
- If a new spool kind is added, update its payload validation, receipt behavior, client upload path, worker drain path, and expiry outcome together.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| Client or worker has no valid ingest token | The event stays in the client spool and remote delivery does not start | Configure a safe shared CLAUDE_MEM_WORKER_INGEST_TOKEN on both sides |
| Network timeout or missing acknowledgement | The client keeps the event and retries it later | Use the exact receipt check and bounded upload timeout |
| Direct live handoff times out after an ambiguous worker result | The live event is skipped because it has no durable client outbox | Use deferred spool capture when the event must survive an uncertain handoff |
| Remote payload fails validation or exceeds the accepted size | The event is not ingested and is moved to the client's corrupt/ area | Validate the versioned envelope before acknowledging it |
| Worker stops after ingest accepts an event but before file removal | A retry could duplicate capture without a hand-off marker | Write the durable consumed marker before generator work and remove marked files without re-ingest |
| Live buffer reaches its session or global limit | New live input is skipped instead of creating an unbounded catch-up queue | Keep the bounded buffer policy and use the transcript for recovery |
| Provider fails after streamed output starts | A resend could charge twice or create a duplicate result | Treat post-output stream failure as non-retryable |
| Session end arrives before session creation | The end waits briefly or is later retired while the session remains unregistered | Keep closing events behind observations and apply the unknown-session grace window |
Key files
| File | Why to read it |
|---|---|
| src/…/hook-spool.ts | Defines durable event files, ordering, receipts, retry retention, and exactly-once hand-off rules. |
| src/…/hook-spool-remote.ts | Defines the remote protocol, payload limits, token checks, and transport selection. |
| src/…/spool-hook-event.ts | Shows how hooks schedule local nudges or detached remote uploads without waiting for processing. |
| src/…/live-observer-event.ts | Shows the bounded direct handoff used for live observer events and its local-versus-remote route choice. |
| src/…/HookSpoolRoutes.ts | Shows the authenticated remote acceptance boundary and acknowledgement contract. |
| src/…/hook-spool-drain.ts | Shows how the worker schedules drains and maps spool entries to shared ingest actions. |
| src/…/SessionMessageBuffer.ts | Shows the bounded, non-durable buffer used after accepted live work reaches the worker. |
| src/…/SessionManager.ts | Shows queue admission, provider-pause recovery, and paced transport resumes. |
| src/…/provider-dispatch.ts | Shows provider selection and fallback when a provider cannot serve capture. |
| src/…/streamed-chat-completion.ts | Shows the boundary between retryable pre-output failures and non-retryable post-output failures. |
| src/…/worker-service.ts | Shows worker startup, nudge routing, remote spool route registration, and the uploader command. |
| src/…/SessionStore.ts | Defines the durable hand-off markers that protect against duplicate processing across restarts. |
| src/…/SearchManager.ts | Shows how stored capture is read through semantic and keyword search with bounded fallbacks. |
| src/…/ProgressiveMemorySearch.ts | Shows how observations, summaries, and prompts become one bounded memory view. |
| src/…/PaginationHelper.ts | Shows bounded project, source, and session reads for stored capture. |