Introducing a new way to review your PRs.

Read the announcement

thedotmack / claude-mem

Persistent Context Across Sessions for Every Agent – Captures everything your agent does during sessions, compresses it with AI, and injects relevant context back into future sessions. Works with Claude Code, OpenClaw, Codex, Gemini, Hermes, Copilot, OpenCode + More

View on GitHub

47 documents · Updated Nov 20, 2026

docs / sync-application-protocol

Sync application protocol

The sync application protocol keeps shared memory content aligned between local installations and the cloud. It uses durable local records for recovery and treats Chroma as a rebuildable search view, so search delays do not change sync progress or local data.

Protocol model

The protocol has two durable lanes. The push lane sends local changes to the cloud. The pull lane reads ordered cloud history and applies remote changes to the local database. Cloud sequence order controls replay, while local SQLite remains the usable record store. Pulled rows and the pull cursor commit in one transaction, so a restart replays a whole batch or none of it.

Synchronization path

The path turns a local write into a canonical operation, stores it until the cloud confirms it, and applies that history on other installations. Search indexing starts after the durable local transaction and does not decide whether synchronization succeeds.

flowchart LR
    A[Local record or change] --> B[Durable outbox]
    B --> C[Canonical operation]
    C --> D[Cloud sync hub]
    D --> E[Ordered user history]
    E --> F[Pull client]
    F --> G[Local SQLite apply]
    G -. committed content .-> H[Chroma search view]

References: src/…/CloudSync.ts• L1-41 src/…/SyncClient.ts• L1-57 src/…/SyncApply.ts• L590-738

Device identity

One resolved device identity names the installation in both lanes. Remote rows keep their source identity, so the push lane does not send them again and the apply lane can discard echoes of the installation's own operations.

The apply path receives the identity resolved by the cloud sync service. This single source prevents a local row from being attributed to different devices by push and pull. Frozen outbox snapshots made under an old identity are set aside and native rows are snapshotted again under the current identity.

References: src/…/CloudSync.ts• L641-681 src/…/CloudSync.ts• L1525-1678 src/…/SyncApply.ts• L362-371 src/…/SyncApply.ts• L461-471 src/…/SyncApply.ts• L634-637

Canonical content

Canonical content is the shared wire form for records and logical changes. It gives an operation a stable identity, a positive entity revision, a canonical JSON body, and hashes that expose altered data.

Content records

The protocol carries observations, summaries, prompts, and work_state. A content identity comes from its kind, origin device, and origin local id. The receiving installation maps that identity to its own local row instead of reusing the sender's SQLite row id.

Live records carry portable application fields. Local sync markers, row ids, session row ids, and other device-only state stay local. Deletes use tombstones that retain identity and revision but carry no live payload. Work-state records remain an append-only local list log and have no search copy.

Media metadata uses a bounded manifest inside the permitted metadata namespace. The protocol carries media identity and state, not paths, object keys, data URLs, or file bytes.

References: src/…/CanonicalContent.ts• L5-48 src/…/CanonicalContent.ts• L165-202 src/…/CanonicalContent.ts• L270-390 src/…/CanonicalContent.ts• L409-424 src/…/SyncApply.ts• L1006-1051

Operation integrity

Canonical JSON sorts object keys and keeps array order. Large sequence and revision values use decimal strings so they do not lose precision. The operation hash covers the exact serialized body, while the payload hash covers the exact canonical payload.

The shared validator rejects unknown fields, invalid identities, bad hashes, malformed timestamps, unsafe sizes, and unsupported payload shapes. It also checks mutation predicates and work-state fields. Both cloud push and local apply therefore use the same contract instead of repairing different versions of one operation.

References: src/…/CanonicalContent.ts• L69-145 src/…/CanonicalContent.ts• L231-268 src/…/CanonicalContent.ts• L294-407

Mutation operations

Mutation operations carry logical changes that are not independent content rows. The protocol supports custom session titles, prompt-to-session repairs, and bounded project remaps. They use the same canonical envelope and cloud order as content records.

Their order is part of their meaning. A title can arrive before its session, a prompt repair can add a missing session link later, and a project remap can update matching observations and summaries as one bounded change.

References: src/…/CanonicalContent.ts• L13-18 src/…/CanonicalContent.ts• L204-229 src/…/CanonicalContent.ts• L426-475 src/…/SyncApply.ts• L1550-1739

Cloud synchronization

Cloud synchronization uses local SQLite as the durable work queue. A local write signals the service, which later groups pending work into bounded requests for the authenticated per-user sync hub.

Content push

The push path freezes a row into an immutable canonical snapshot before sending it. A later local edit receives a higher revision and a new snapshot. This keeps retries stable while allowing newer local state to remain pending.

Native rows are eligible for push only when their sync marker is empty. Replica rows are excluded because another device owns their origin. Work-state waits until the hub advertises support for that kind, then the service checks again later if it remains pending.

Deletes send higher-revision tombstones. Invalid rows and repeatedly rejected operations move to durable quarantine so one bad item does not block later work. Credential failures pause the lane for a later status check instead of consuming the normal retry cycle.

References: src/…/CloudSync.ts• L70-109 src/…/CloudSync.ts• L711-825 src/…/CloudSync.ts• L965-1017 src/…/CloudSync.ts• L1118-1175

Outbox remapping

Project remapping keeps local adoption and cloud convergence in one logical operation. The local update, its revision stamp, and the bounded mutation outbox entry commit together.

The mutation lane preserves relationships by sending frozen content first, then mutations, then newly materialized row snapshots. Retries reuse one mutation identity. Native rows become pending again for corrected snapshots, while replica rows keep their remote origin.

References: src/…/CloudSync.ts• L756-803 src/…/CloudSync.ts• L1196-1324 src/…/CanonicalContent.ts• L459-475 src/…/SyncApply.ts• L1685-1739

Acknowledgement proof

The local service treats a successful push as complete only when every sent operation has one matching acknowledgement. The acknowledgement must preserve operation identity, revision, hash, and server sequence. Each acknowledged sequence must be covered by the cloud's projected checkpoint. The cloud head may be further ahead.

Only after this proof does the service mark the current native row as synced or remove a mutation from its queue. If the row changed during the request, the current revision no longer matches and the newer state remains pending for another push.

References: src/…/CloudSync.ts• L1413-1522 src/…/CloudSync.ts• L1766-1935

Local synchronization

The local synchronization path reads the ordered cloud history and applies it to SQLite. The pull cursor and cloud-history epoch live in local sync state, so a restart can resume without a separate cursor file.

Pull client

The pull client reads short HTTP pages after the stored cursor and sends each page to the local applier. Active sessions use a faster poll tier. Idle installations poll less often and can suspend the loop after extended inactivity.

Supabase Realtime is an advisory speed lane. It sends advance notices only, which trigger the same HTTP pull. It never carries records or writes the cursor. Malformed frames, stale epochs, join failures, missing heartbeats, and unavailable Realtime fall back to HTTP polling. A push response can also expose a newer head sequence and trigger an immediate pull.

References: src/…/SyncClient.ts• L1-58 src/…/SyncClient.ts• L408-467 src/…/SyncClient.ts• L630-722

Cursor and epoch

The cursor is the last cloud sequence applied locally. The epoch identifies the cloud history that owns those sequences. A page advances the cursor in the same transaction as its row changes.

When the epoch changes, the applier stores the new epoch, resets the cursor, and discards the current page. It requeues eligible native revisions so the rebuilt history can receive them again, while replica rows and launch-baseline exclusions stay out of the queue. The client then pulls from the start. Idempotent entity revisions make replay safe.

Older installations may have set aside work-state changes that they could not decode. Startup makes one controlled replay attempt for those records. Changes that still cannot apply remain quarantined.

References: src/…/SyncApply.ts• L505-584 src/…/SyncApply.ts• L605-708 src/…/SyncApply.ts• L824-868 src/…/SyncClient.ts• L408-417

Row application

Each pulled page runs in one SQLite transaction. A remote row is inserted or updated by origin identity, and an entity head remembers the highest accepted revision even if the local row is later deleted. This prevents stale data from replacing newer data or resurrecting a tombstone.

Each operation also has a savepoint. A malformed or permanently conflicting operation is recorded in pull quarantine and does not block later history. A constraint error that may depend on another row is retried at the end of the batch and in later batches. Other transient database errors fail the page so the whole page can retry. Remote rows are pre-marked as synced, which keeps them out of the local push queue.

Inbound text and JSON fields follow the local redaction policy before storage. Shared work lists also apply their local inbound privacy rule before they are stored. Observation file indexes and replica media links update with the row revision. Base-table writes also feed the normal SQLite search triggers. Chroma forwarding starts only after commit and does not control the durable cursor.

References: src/…/SyncApply.ts• L590-738 src/…/SyncApply.ts• L735-813 src/…/SyncApply.ts• L928-1100 src/…/SyncApply.ts• L1219-1568

Session association

Sessions are local records and are not themselves synchronized. If a remote observation, summary, or prompt needs a missing session, the applier creates a minimal sync stub and later folds or links stubs when the content and memory identities become known.

Titles use the same delayed association model. If a target session is absent, the title is parked in sync state. When the session or its content-to-memory link appears, the parked title is claimed only when no local title already exists.

References: src/…/SyncApply.ts• L1103-1267 src/…/SyncApply.ts• L1269-1568 src/…/SyncApply.ts• L1615-1695

Chroma synchronization

Chroma is a search projection of local SQLite content. The sync path forwards inserted and revised observations, summaries, and prompts after their durable SQLite transaction. A Chroma outage can delay semantic search but cannot roll back local data or cloud progress.

Search projection

Rows expand into deterministic Chroma documents with project and row metadata. Observations can produce narrative, text, fact, or title documents. Summaries produce documents for their searchable fields. Prompts produce one searchable document. An observation with no body or facts can still produce a title document, so searchable title content is not lost.

The writer confirms the number of documents that landed before it marks the row complete. Duplicate ids are reconciled in place. A changed row removes fragments that no longer exist. Live writes, deletes, and backfill use the same per-row order, so an older in-flight write cannot follow a delete or a newer row update. A failed delete is recorded for retry because backfill only sees rows that still exist. Text with NUL bytes is replaced before it reaches the search index. Project remaps update only the merge metadata, so they do not replace newer row content.

Search reads can scope results by project, merged project, platform, or document type. The client reduces duplicate fragments to one result per content row and uses bounded over-fetch or row-id narrowing when a broad Chroma filter would be costly. Each semantic search shares a 20-second Chroma wait budget across its widening reads. If that budget ends, the caller uses the SQLite/FTS5 keyword path while the slow Chroma read remains managed by the connection layer.

References: src/…/ChromaSync.ts• L405-475 src/…/ChromaSync.ts• L923-1265 src/…/ChromaSync.ts• L1267-1609 src/…/ChromaSync.ts• L2322-2627 src/…/chroma-search-budget.ts• L1-47 src/…/ChromaSearchStrategy.ts• L104-170 src/…/SearchManager.ts• L133-197

Watermark state

A watermark records the highest local row id fully represented in Chroma for one project and content kind. Pending ids record holes below that watermark. Fragment-reconciliation state records rows whose old document fragments must be removed before the new row is complete. Pending row deletes record Chroma documents for rows that no longer exist in SQLite.

This state is kept in an atomic local checkpoint outside SQLite. Progress can be grouped into short durable write windows, but a failed checkpoint write stays dirty for retry. An unreadable checkpoint is treated as empty so the next backfill can compare SQLite with Chroma instead of stopping the sync system. A watermark advances only after every document for a row is confirmed. A partial write leaves the row pending, including rows with no searchable text. Empty rows are drained as complete rows rather than treated as missing.

Delete cleanup has a separate durable retry path in the same checkpoint. The delete is recorded before it enters the shared cleanup lane. A failed or interrupted cleanup stays in the checkpoint and retries before backfill writes or when Chroma recovers. If the same row returns before retry, the stale cleanup is discarded so it cannot remove the restored row's documents.

References: src/…/ChromaSyncState.ts• L7-29 src/…/ChromaSyncState.ts• L31-121 src/…/ChromaSyncState.ts• L213-293 src/…/ChromaSyncState.ts• L315-404 src/…/ChromaSyncState.ts• L419-564 src/…/chroma-row-cleanup.ts• L17-102 src/…/ChromaSync.ts• L1312-1415

Backfill and recovery

Startup backfill considers every project represented by observations, summaries, or session-linked prompts. When the checkpoint is missing, it compares SQLite rows with Chroma and counts a row as present only when all of its fragments exist. Missing rows become pending. A project with source rows but no indexed rows is scheduled for a rebuild from zero.

Backfill runs in bounded project groups and shares a paced write turn across projects. It reads the current row inside its per-row turn, so a live update cannot leave an older snapshot in the search view. Isolated write failures leave rows pending while other content kinds and projects continue. Empty rows are marked complete, and older title-only observations can be requeued for a one-time repair.

A confirmed corrupt collection is marked for rebuild before deletion. A generation change stops writers targeting the dropped collection. Every project then rebuilds from SQLite. The next run retries incomplete projects rather than treating the sweep as complete. Post-delete cleanup runs before these writes and also drains when the Chroma connection recovers.

References: src/…/ChromaSync.ts• L1516-1717 src/…/ChromaSync.ts• L1790-2105 src/…/ChromaSync.ts• L2630-2816 src/…/ChromaSyncState.ts• L213-249 src/…/ChromaSyncState.ts• L419-454

Chroma process lifecycle

The Chroma manager owns the local search process and its connection. It prevents two workers from writing the same persistent store, waits for old subprocess cleanup before reconnecting, and serializes local mutations so an in-flight write is not interrupted by a read timeout or shutdown.

Connection recovery notifies the sync layer so pending row deletes drain as soon as Chroma returns. Failed environment prewarms use a circuit breaker and bounded scratch cleanup instead of spawning endlessly. A persistent store records its writer compatibility epoch, and a newer incompatible writer is refused before it can open the store.

Search reads have a shorter caller budget than the manager's transport deadline. When a read reaches the caller budget, the search falls back without killing a slow healthy process. When the transport itself times out, the manager avoids killing the process while a mutation may still be committing, then restarts it only after the write settles.

References: src/…/ChromaMcpManager.ts• L362-412 src/…/ChromaMcpManager.ts• L446-668 src/…/ChromaMcpManager.ts• L786-1069 src/…/ChromaMcpManager.ts• L1277-1453 src/…/ChromaMcpManager.ts• L1455-1632 src/…/ChromaSync.ts• L617-685

Change checklist

When these files or sections are changed, remember to consider these:

  • Changes to canonical fields, identity rules, revision encoding, media manifests, or work_state support require matching changes in local producers, cloud validation, local apply, and compatibility replay.
  • Changes to push ordering, request limits, frozen snapshots, dead-letter rules, or acknowledgement checks must preserve forward progress and must stamp only the current revision.
  • Changes to device identity resolution must account for frozen operations created under an older identity and must not add a second identity source to local apply.
  • Changes to cursor or epoch handling must preserve page contiguity, full replay after a rebuilt history, and the exclusion of replica and launch-baseline rows from requeue.
  • Changes to entity heads, tombstones, or equal-revision hash checks must preserve convergence after local deletion and must not accept two bodies for one entity revision.
  • Changes to quarantine, savepoint, and head-ledger rules must keep permanent bad operations from blocking the cursor, dependent row errors retryable, and stale revisions unable to replace or resurrect newer rows.
  • Changes to session stubs, title parking, prompt repair, or project remaps must preserve delayed association and bounded mutation scope.
  • Changes to Realtime notices, head-sequence hints, poll suspension, or socket recovery must keep HTTP pulls as a complete durable recovery path.
  • Changes to Chroma document expansion, duplicate reconciliation, fragment deletion, row ordering, or watermark rules must keep row completion tied to every generated document.
  • Changes to search over-fetch, row-id narrowing, filter predicates, or the shared search budget must preserve exact scope and leave keyword fallback time within the caller deadline.
  • Changes to checkpoint persistence, watermark coalescing, pending-id repair, or rebuild flags must preserve restart recovery when the state file or Chroma is unavailable.
  • Changes to post-delete cleanup reservation, checkpoint persistence, retry draining, or row restoration checks must keep deleted-row fragments recoverable without deleting documents for a restored row.
  • Changes to Chroma collection recovery must persist rebuild intent before deletion and must stop old-generation writers from repopulating the dropped collection.
  • Changes to Chroma process ownership, mutation serialization, timeout handling, writer compatibility, or prewarm recovery must not kill an active commit or allow two writers on one store.
  • Changes to inbound redaction, file indexes, or replica media links must keep local privacy and derived indexes aligned with the applied row revision.

Important failure modes

TriggerConsequenceGuard
A canonical body or hash is invalidThe cloud rejects it or local apply sets it asideShared validation, operation savepoints, and pull quarantine
A pulled page has a sequence gap or repeats an earlier sequence out of orderThe page cannot safely advance the pull cursorContiguous-page checks and one transaction for the page
A row changes while a push is in flightThe old acknowledgement does not hide newer local stateCurrent-revision and canonical-body checks before stamping
A frozen operation carries an old device identityThe hub rejects it and it can block later queue workStale-identity dead-lettering and native-row resnapshot
The hub does not advertise work_stateWork-state remains pendingCapability probing and later re-probing
The cloud epoch changesThe current page is discarded and native work is made eligible againEpoch reset, launch exclusions, and idempotent replay
A push acknowledgement is incomplete or alteredSent work remains pendingExact acknowledgement tuple and per-ack projection-checkpoint proof
A Realtime socket fails or is unavailableLive delivery slows or stopsHTTP polling remains the recovery path
Two operations claim one entity revision with different bodiesThe first accepted body remains and the later operation is set asideEntity-head hash check and pull quarantine
A pulled row depends on a local row that has not arrived yetThe row is set aside without blocking later historyRetryable constraint quarantine and end-of-batch retry
A remote row needs a missing sessionForeign-key application can fail or content can be orphanedSync stubs, delayed prompt repair, and title parking
Chroma writes only part of a row's documentsSearch is incomplete while SQLite remains correctPending ids and watermark advancement only after full confirmation
A Chroma search uses its full caller budgetSemantic results stop for that requestShared 20-second budget and SQLite/FTS5 keyword fallback
A Chroma checkpoint write failsProgress may be replayed, but SQLite remains the source of truthAtomic checkpoint writes, dirty-state retry, and backfill comparison
A Chroma row delete fails after SQLite commitsOld search fragments remain with no later source-row scan to remove themDurable pending row-delete records and retry on backfill or connection recovery
A restored row reuses an id before a stale delete retryThe restored row's search documents could be removedRetry checks SQLite and drops the stale cleanup
A Chroma collection has repeated corrupt-segment failuresSearch writes stop until rebuildPersisted rebuild flags, collection generation, and SQLite rebuild
A revised or deleted row leaves old fragmentsSearch returns stale contentPer-row writer ordering and fragment reconciliation
A scoped Chroma query uses a broad metadata filterSearch becomes slow or misses the intended result windowBounded over-fetch, row-id narrowing, and safe client-side filtering
A Chroma process dies during reconnect or shutdownA replacement process can race with old descendants or lose cleanup workGeneration checks, captured process identity, tree cleanup, and one writer lock
A prewarm host fails repeatedlyReconnect attempts waste CPU, disk, and build scratchExponential breaker cooldown, bounded scratch sweep, and restart latch
A Chroma read times out while a write is activeKilling the process can interrupt a persistent index commitLeave the process running until mutations settle, then restart

Key files

FileWhy to read it
src/…/CanonicalContent.tsDefines operation identity, canonical JSON, hashes, revisions, tombstones, content payloads, media manifest limits, and mutation shapes.
src/…/CloudSync.tsShows how local rows and mutations become durable push requests, how identity changes, capabilities, retries, acknowledgements, and queues work.
src/…/SyncClient.tsShows how cloud history is pulled, how HTTP owns recovery, and how advisory Realtime delivery affects timing only.
src/…/SyncApply.tsShows how remote changes converge into rows, entity heads, sessions, titles, indexes, media links, quarantine state, and post-commit Chroma jobs.
src/…/ChromaSync.tsShows the search projection, deterministic document writes, query scope handling, search deadlines, row ordering, fragment cleanup, durable cleanup recovery, backfill, and collection rebuild.
src/…/chroma-search-budget.tsDefines the shared time limit that lets semantic search fall back to SQLite/FTS5 without cancelling a slow Chroma read.
src/…/ChromaSyncState.tsDefines persisted project watermarks, pending ids, atomic checkpoint progress, fragment repair, rebuild state, and durable post-delete cleanup records.
src/…/chroma-row-cleanup.tsShows how local and replicated row deletes reserve, serialize, record, and retry Chroma cleanup work.
src/…/remap-outbox.tsShows how local project remaps and their cloud mutations commit together.
src/…/ChromaMcpManager.tsExplains Chroma process ownership, writer compatibility, mutation ordering, caller deadlines, reconnects, prewarm recovery, and safe shutdown.