Deep-search execution
Deep-search execution provides bounded, targeted access to stored memory and raw tool-use evidence without slowing the normal context reader. It combines a local child process for isolated raw searches with a progressive memory lookup that returns an index first and details only for selected records. For the wider delivery path, see Context delivery; related retrieval behavior is described in Search and memory retrieval and Context construction.
Execution boundary
The deep-search boundary separates expensive or sensitive retrieval from ordinary context work. The worker admits one raw search at a time, gives it a fixed time and size budget, and returns an explicit status when the search cannot complete.
Single-flight client
The DeepSearchClient is the local entry point for raw tool-use search. It rejects work when the client is closed, when another child is still active, or when the runtime cannot support a standalone child. A timeout completes the request for the caller, but the client keeps its slot until the operating system confirms that the child has exited. This prevents a late child from overlapping a new query.
The client shares child preparation across requests without keeping timed-out callers attached to the build. It passes only the query and supported search scope to the child. The worker gives this executor to SearchManager, which invokes it only for an eligible local observation search.
References: src/…/DeepSearchClient.ts• L25-39 src/…/DeepSearchClient.ts• L119-145 src/…/SearchManager.ts• L463-498 src/…/worker-service.ts• L1275-1285
Search eligibility
Raw search supplements an ordinary observation search; it does not replace it. The request must be a first-page observation search with a valid query and no filter that the raw tier cannot apply. A successful ordinary search with observations does not run the raw tier, while an ordinary error or an unsupported scope produces a clear non-success status.
The result keeps raw evidence separate from observations. This lets callers distinguish “the raw search was not applicable” or “it timed out” from “the raw search completed and found no matches.” Server search rejects the deep option before sending a local or server search request because this raw tier is local-only.
References: src/…/SearchManager.ts• L463-498 tests/…/mcp-tool-schemas.test.ts• L153-185
Child-process protocol
The client and child use one request and one reply over standard input and output. The protocol carries a version, request identity, parent identity, absolute deadline, database path, query, and supported search options. The reply carries the same version and identity plus a bounded DeepSearchResult.
The parent derives the child deadline from the remaining request budget. The child reads the wall-clock deadline once, then uses a monotonic clock for its own work. This prevents a clock change during startup from giving the child extra time or ending the search too early.
The child opens the database as read-only and does not create a missing file. It checks the raw index before searching. The search applies project, platform, date, and result-order scope, then returns bounded tool-use metadata and snippets rather than full raw bodies. A caller can use the numeric IDs to request selected tool-use bodies later.
The transport has separate request and reply byte limits. The client validates the reply, checks the exit result, reassembles split UTF-8 output, and treats malformed, oversized, mismatched, or abnormal replies as incomplete errors. This keeps untrusted child output from becoming an apparently complete search result.
References: src/…/protocol.ts• L17-40 src/…/child.ts• L6-40 src/…/DeepSearchClient.ts• L65-105 src/…/tool-use-search.ts• L179-228
Process guards
The child has an independent guard that watches both the parent process and the deadline. If the parent dies or the deadline passes while native database work is blocked, the guard kills the child. The parent also requests termination when a search times out or fails, then waits for confirmed process exit before releasing the single-flight slot.
The child keeps this guard armed until its reply has drained from standard output. This lets the transport finish without leaving a live child behind, including on hosts where pending pipe writes need time to complete.
This two-sided cleanup protects the normal reader and the database's read snapshot. It also makes host shutdown wait for the deep-search child instead of leaving a reader behind.
References: src/…/guard.ts• L1-29 src/…/child.ts• L33-40 src/…/DeepSearchClient.ts• L111-117 src/…/DeepSearchClient.ts• L136-145 src/…/worker-service.ts• L1610-1671
Runtime support
Deep search needs a runtime that can start the standalone child through the Bun launcher contract. Source runs build the child program when first needed. Release builds embed the same child program so the installed binary does not need a source tree, package directory, or writable temporary file.
Compiled binaries are checked separately from source execution. An embedded runtime below the supported Bun version is rejected before preparation or spawn because it cannot safely start the child. An unsupported runtime returns unsupported rather than attempting an inline fallback.
References: src/…/child-program.ts• L1-13 scripts/deep-search-build.js• L1-17 src/…/runtime-support.ts• L1-12 src/…/bun-be-bun-support.ts• L1-25
The following sequence shows how a raw query moves from the worker to isolated retrieval and back to the caller.
sequenceDiagram
participant Caller
participant Worker as SearchManager
participant Client as DeepSearchClient
participant Child as Read-only child
participant Guard as Child guard
participant DB as Memory database
Caller->>Worker: Search with deep enabled
Worker->>Worker: Run ordinary observation search
Worker->>Client: Admit eligible raw query
Client->>Child: Send versioned request over stdin
Child->>Guard: Start parent/deadline guard
Child->>DB: Open read-only and search raw index
DB-->>Child: Bounded raw hits and snippets
Child-->>Client: Versioned reply over stdout
Child->>Guard: Drain output, then terminate guard
Client-->>Worker: Complete result or explicit status
Worker-->>Caller: Ordinary results plus formatted raw tier
Progressive memory retrieval
Progressive retrieval is the shallow memory path used when a caller needs fast targeted recall. It separates discovery from disclosure: the first step returns bounded index rows, the second adds nearby records around selected anchors, and the third fetches details only for relevant non-prompt records.
Retrieval stages
The ProgressiveSearch flow validates the query and bounds the project, result count, context depth, and detail count before it calls the backend. It ranks titles by query-word overlap. If no title is selected, it may make one normalized query refinement, but it does not perform hidden reasoning or unbounded retries.
When anchors exist, the flow reads nearby timeline context within the anchor project and ranks that context again. It then fetches only selected observation or summary details. Prompt rows can remain in the index, but they are not fetched as memory details.
References: src/…/progressive-search.ts• L15-63 src/…/progressive-search.ts• L193-242
Result shaping
Progressive results report the completed step, the query, the selected project, the operation trace, and a reason when retrieval stops early. The response has a fixed byte budget. The system first reduces the index and then shortens the largest detail, marking shortened content so the caller knows that the evidence is incomplete.
The formatter labels retrieved content as data rather than instructions. It shows the index when no details are selected and gives the caller the next Smart search workflow when shallow title matching is not enough. Backend errors are reduced to safe contract messages instead of exposing database or credential details.
References: src/…/progressive-search.ts• L162-191 src/…/progressive-search.ts• L245-370
Native-memory bridge
The native-memory hook supplements an agent's pending memory or code-search action without replacing the native tool result. It recognizes supported memory tools and memory paths, maps the lookup to the active project and source, and sends only a bounded index to the model. If the worker is unavailable, the native operation continues without supplemental context.
This bridge preserves the same staged recall rule: the model receives index fields first, then chooses timeline context and selected details. It also treats stored memory as evidence, not as instructions to execute.
References: src/…/memory-search.ts• L102-136 src/…/memory-search.ts• L138-183
Result formatting
Deep results are projected into the same text surfaces used by normal search and MCP. Raw hits use a distinct #T identity, show a short tool and project label with a bounded snippet, and state whether more matches exist. The formatter keeps incomplete statuses visible because a timeout or unsupported search is not proof that no raw evidence exists.
For completed raw results, the text points to get_tool_uses with numeric IDs and preserves the project and platform scope. It warns callers not to pass raw IDs to observation or timeline operations. MCP reserves space for this raw block even when ordinary summaries are large, so targeted raw evidence is not silently lost to response-size limits.
References: src/…/deep-search-format.ts• L1-18 src/…/mcp-text-format.ts• L139-184 src/…/mcp-text-format.ts• L213-230
Change checklist
When these files or sections are changed, remember to consider these:
- Keep the request and reply version, identity checks, byte limits, and result completeness rules aligned between the client and child.
- Keep the raw-search eligibility rules aligned with the filters that
searchToolUsescan actually scope. - Keep embedded release child content and source-built child behavior equivalent when the child protocol changes.
- Keep runtime support checks aligned with the Bun self-launch contract used by compiled binaries.
- Keep raw
#TIDs and theget_tool_usesfollow-up separate from observation and timeline IDs in every text and JSON projection. - Keep progressive index, timeline, and detail budgets aligned with the backend's owner and project scope checks.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| Raw index is missing, building, disabled, or not ready | Raw search returns an explicit incomplete status instead of raw hits | Check index readiness before the child query and preserve the status in the formatter |
| Query exceeds the request or search limits | The raw tier returns invalid_query or the transport rejects the request | Bound query terms and request bytes before opening or searching the child |
| Child exceeds its deadline or the parent dies | The child is killed and the request is incomplete; a live child keeps the slot busy until exit | Use the child guard and confirmed-exit cleanup |
| Compiled Bun runtime cannot start a standalone child | Deep search returns unsupported without building or spawning | Reject unsupported embedded runtimes before preparation |
| Ordinary MCP text consumes the response budget | Raw evidence could disappear from the caller's view | Reserve space for the formatted raw tier and cap both projections |
| Progressive backend returns invalid, cross-project, or oversized details | Unsafe or excessive memory content reaches the model | Validate identities, keep timeline context in the anchor project, and truncate within the response budget |
Key files
| File | Why to read it |
|---|---|
| src/…/DeepSearchClient.ts | Defines admission, deadlines, child lifecycle, reply validation, and shutdown behavior. |
| src/…/child.ts | Shows the child-side request validation, read-only database use, and reply lifecycle. |
| src/…/guard.ts | Defines parent-death and deadline cleanup that protects the host and database reader. |
| src/…/protocol.ts | Defines the transport contract and shared size and time limits. |
| src/…/tool-use-search.ts | Explains raw index readiness, scope-aware search, bounded snippets, and raw hit shape. |
| src/…/SearchManager.ts | Connects ordinary memory search to eligible local raw search and result projection. |
| src/…/progressive-search.ts | Defines staged memory retrieval, validation, ranking, truncation, and safe text output. |
| src/…/deep-search-format.ts | Defines the caller-facing raw evidence format and follow-up ID contract. |
| src/…/mcp-text-format.ts | Keeps deep evidence visible and budgeted in MCP text responses. |
| src/…/memory-search.ts | Shows how native memory actions receive bounded supplemental index context. |
| scripts/deep-search-build.js | Shows how release artifacts embed the child program for isolated execution. |