Sync application protocol
The sync application protocol keeps shared memory content aligned between local installations and the cloud. It uses durable local records for recovery and treats Chroma as a rebuildable search view, so search delays do not change sync progress or local data.
Protocol model
The protocol has two durable lanes. The push lane sends local changes to the cloud. The pull lane reads ordered cloud history and applies remote changes to the local database. Cloud sequence order controls replay, while local SQLite remains the usable record store. Pulled rows and the pull cursor commit in one transaction, so a restart replays a whole batch or none of it.
Synchronization path
The path turns a local write into a canonical operation, stores it until the cloud confirms it, and applies that history on other installations. Search indexing starts after the durable local transaction and does not decide whether synchronization succeeds.
flowchart LR
A[Local record or change] --> B[Durable outbox]
B --> C[Canonical operation]
C --> D[Cloud sync hub]
D --> E[Ordered user history]
E --> F[Pull client]
F --> G[Local SQLite apply]
G -. committed content .-> H[Chroma search view]
References: src/…/CloudSync.ts• L1-41 src/…/SyncClient.ts• L1-57 src/…/SyncApply.ts• L590-738
Device identity
One resolved device identity names the installation in both lanes. Remote rows keep their source identity, so the push lane does not send them again and the apply lane can discard echoes of the installation's own operations.
The apply path receives the identity resolved by the cloud sync service. This single source prevents a local row from being attributed to different devices by push and pull. Frozen outbox snapshots made under an old identity are set aside and native rows are snapshotted again under the current identity.
References: src/…/CloudSync.ts• L641-681 src/…/CloudSync.ts• L1525-1678 src/…/SyncApply.ts• L362-371 src/…/SyncApply.ts• L461-471 src/…/SyncApply.ts• L634-637
Canonical content
Canonical content is the shared wire form for records and logical changes. It gives an operation a stable identity, a positive entity revision, a canonical JSON body, and hashes that expose altered data.
Content records
The protocol carries observations, summaries, prompts, and work_state. A content identity comes from its kind, origin device, and origin local id. The receiving installation maps that identity to its own local row instead of reusing the sender's SQLite row id.
Live records carry portable application fields. Local sync markers, row ids, session row ids, and other device-only state stay local. Deletes use tombstones that retain identity and revision but carry no live payload. Work-state records remain an append-only local list log and have no search copy.
Media metadata uses a bounded manifest inside the permitted metadata namespace. The protocol carries media identity and state, not paths, object keys, data URLs, or file bytes.
References: src/…/CanonicalContent.ts• L5-48 src/…/CanonicalContent.ts• L165-202 src/…/CanonicalContent.ts• L270-390 src/…/CanonicalContent.ts• L409-424 src/…/SyncApply.ts• L1006-1051
Operation integrity
Canonical JSON sorts object keys and keeps array order. Large sequence and revision values use decimal strings so they do not lose precision. The operation hash covers the exact serialized body, while the payload hash covers the exact canonical payload.
The shared validator rejects unknown fields, invalid identities, bad hashes, malformed timestamps, unsafe sizes, and unsupported payload shapes. It also checks mutation predicates and work-state fields. Both cloud push and local apply therefore use the same contract instead of repairing different versions of one operation.
References: src/…/CanonicalContent.ts• L69-145 src/…/CanonicalContent.ts• L231-268 src/…/CanonicalContent.ts• L294-407
Mutation operations
Mutation operations carry logical changes that are not independent content rows. The protocol supports custom session titles, prompt-to-session repairs, and bounded project remaps. They use the same canonical envelope and cloud order as content records.
Their order is part of their meaning. A title can arrive before its session, a prompt repair can add a missing session link later, and a project remap can update matching observations and summaries as one bounded change.
References: src/…/CanonicalContent.ts• L13-18 src/…/CanonicalContent.ts• L204-229 src/…/CanonicalContent.ts• L426-475 src/…/SyncApply.ts• L1550-1739
Cloud synchronization
Cloud synchronization uses local SQLite as the durable work queue. A local write signals the service, which later groups pending work into bounded requests for the authenticated per-user sync hub.
Content push
The push path freezes a row into an immutable canonical snapshot before sending it. A later local edit receives a higher revision and a new snapshot. This keeps retries stable while allowing newer local state to remain pending.
Native rows are eligible for push only when their sync marker is empty. Replica rows are excluded because another device owns their origin. Work-state waits until the hub advertises support for that kind, then the service checks again later if it remains pending.
Deletes send higher-revision tombstones. Invalid rows and repeatedly rejected operations move to durable quarantine so one bad item does not block later work. Credential failures pause the lane for a later status check instead of consuming the normal retry cycle.
References: src/…/CloudSync.ts• L70-109 src/…/CloudSync.ts• L711-825 src/…/CloudSync.ts• L965-1017 src/…/CloudSync.ts• L1118-1175
Outbox remapping
Project remapping keeps local adoption and cloud convergence in one logical operation. The local update, its revision stamp, and the bounded mutation outbox entry commit together.
The mutation lane preserves relationships by sending frozen content first, then mutations, then newly materialized row snapshots. Retries reuse one mutation identity. Native rows become pending again for corrected snapshots, while replica rows keep their remote origin.
References: src/…/CloudSync.ts• L756-803 src/…/CloudSync.ts• L1196-1324 src/…/CanonicalContent.ts• L459-475 src/…/SyncApply.ts• L1685-1739
Acknowledgement proof
The local service treats a successful push as complete only when every sent operation has one matching acknowledgement. The acknowledgement must preserve operation identity, revision, hash, and server sequence. Each acknowledged sequence must be covered by the cloud's projected checkpoint. The cloud head may be further ahead.
Only after this proof does the service mark the current native row as synced or remove a mutation from its queue. If the row changed during the request, the current revision no longer matches and the newer state remains pending for another push.
References: src/…/CloudSync.ts• L1413-1522 src/…/CloudSync.ts• L1766-1935
Local synchronization
The local synchronization path reads the ordered cloud history and applies it to SQLite. The pull cursor and cloud-history epoch live in local sync state, so a restart can resume without a separate cursor file.
Pull client
The pull client reads short HTTP pages after the stored cursor and sends each page to the local applier. Active sessions use a faster poll tier. Idle installations poll less often and can suspend the loop after extended inactivity.
Supabase Realtime is an advisory speed lane. It sends advance notices only, which trigger the same HTTP pull. It never carries records or writes the cursor. Malformed frames, stale epochs, join failures, missing heartbeats, and unavailable Realtime fall back to HTTP polling. A push response can also expose a newer head sequence and trigger an immediate pull.
References: src/…/SyncClient.ts• L1-58 src/…/SyncClient.ts• L408-467 src/…/SyncClient.ts• L630-722
Cursor and epoch
The cursor is the last cloud sequence applied locally. The epoch identifies the cloud history that owns those sequences. A page advances the cursor in the same transaction as its row changes.
When the epoch changes, the applier stores the new epoch, resets the cursor, and discards the current page. It requeues eligible native revisions so the rebuilt history can receive them again, while replica rows and launch-baseline exclusions stay out of the queue. The client then pulls from the start. Idempotent entity revisions make replay safe.
Older installations may have set aside work-state changes that they could not decode. Startup makes one controlled replay attempt for those records. Changes that still cannot apply remain quarantined.
References: src/…/SyncApply.ts• L505-584 src/…/SyncApply.ts• L605-708 src/…/SyncApply.ts• L824-868 src/…/SyncClient.ts• L408-417
Row application
Each pulled page runs in one SQLite transaction. A remote row is inserted or updated by origin identity, and an entity head remembers the highest accepted revision even if the local row is later deleted. This prevents stale data from replacing newer data or resurrecting a tombstone.
Each operation also has a savepoint. A malformed or permanently conflicting operation is recorded in pull quarantine and does not block later history. A constraint error that may depend on another row is retried at the end of the batch and in later batches. Other transient database errors fail the page so the whole page can retry. Remote rows are pre-marked as synced, which keeps them out of the local push queue.
Inbound text and JSON fields follow the local redaction policy before storage. Shared work lists also apply their local inbound privacy rule before they are stored. Observation file indexes and replica media links update with the row revision. Base-table writes also feed the normal SQLite search triggers. Chroma forwarding starts only after commit and does not control the durable cursor.
References: src/…/SyncApply.ts• L590-738 src/…/SyncApply.ts• L735-813 src/…/SyncApply.ts• L928-1100 src/…/SyncApply.ts• L1219-1568
Session association
Sessions are local records and are not themselves synchronized. If a remote observation, summary, or prompt needs a missing session, the applier creates a minimal sync stub and later folds or links stubs when the content and memory identities become known.
Titles use the same delayed association model. If a target session is absent, the title is parked in sync state. When the session or its content-to-memory link appears, the parked title is claimed only when no local title already exists.
References: src/…/SyncApply.ts• L1103-1267 src/…/SyncApply.ts• L1269-1568 src/…/SyncApply.ts• L1615-1695
Chroma synchronization
Chroma is a search projection of local SQLite content. The sync path forwards inserted and revised observations, summaries, and prompts after their durable SQLite transaction. A Chroma outage can delay semantic search but cannot roll back local data or cloud progress.
Search projection
Rows expand into deterministic Chroma documents with project and row metadata. Observations can produce narrative, text, fact, or title documents. Summaries produce documents for their searchable fields. Prompts produce one searchable document. An observation with no body or facts can still produce a title document, so searchable title content is not lost.
The writer confirms the number of documents that landed before it marks the row complete. Duplicate ids are reconciled in place. A changed row removes fragments that no longer exist. Live writes, deletes, and backfill use the same per-row order, so an older in-flight write cannot follow a delete or a newer row update. A failed delete is recorded for retry because backfill only sees rows that still exist. Text with NUL bytes is replaced before it reaches the search index. Project remaps update only the merge metadata, so they do not replace newer row content.
Search reads can scope results by project, merged project, platform, or document type. The client reduces duplicate fragments to one result per content row and uses bounded over-fetch or row-id narrowing when a broad Chroma filter would be costly. Each semantic search shares a 20-second Chroma wait budget across its widening reads. If that budget ends, the caller uses the SQLite/FTS5 keyword path while the slow Chroma read remains managed by the connection layer.
References: src/…/ChromaSync.ts• L405-475 src/…/ChromaSync.ts• L923-1265 src/…/ChromaSync.ts• L1267-1609 src/…/ChromaSync.ts• L2322-2627 src/…/chroma-search-budget.ts• L1-47 src/…/ChromaSearchStrategy.ts• L104-170 src/…/SearchManager.ts• L133-197
Watermark state
A watermark records the highest local row id fully represented in Chroma for one project and content kind. Pending ids record holes below that watermark. Fragment-reconciliation state records rows whose old document fragments must be removed before the new row is complete. Pending row deletes record Chroma documents for rows that no longer exist in SQLite.
This state is kept in an atomic local checkpoint outside SQLite. Progress can be grouped into short durable write windows, but a failed checkpoint write stays dirty for retry. An unreadable checkpoint is treated as empty so the next backfill can compare SQLite with Chroma instead of stopping the sync system. A watermark advances only after every document for a row is confirmed. A partial write leaves the row pending, including rows with no searchable text. Empty rows are drained as complete rows rather than treated as missing.
Delete cleanup has a separate durable retry path in the same checkpoint. The delete is recorded before it enters the shared cleanup lane. A failed or interrupted cleanup stays in the checkpoint and retries before backfill writes or when Chroma recovers. If the same row returns before retry, the stale cleanup is discarded so it cannot remove the restored row's documents.
References: src/…/ChromaSyncState.ts• L7-29 src/…/ChromaSyncState.ts• L31-121 src/…/ChromaSyncState.ts• L213-293 src/…/ChromaSyncState.ts• L315-404 src/…/ChromaSyncState.ts• L419-564 src/…/chroma-row-cleanup.ts• L17-102 src/…/ChromaSync.ts• L1312-1415
Backfill and recovery
Startup backfill considers every project represented by observations, summaries, or session-linked prompts. When the checkpoint is missing, it compares SQLite rows with Chroma and counts a row as present only when all of its fragments exist. Missing rows become pending. A project with source rows but no indexed rows is scheduled for a rebuild from zero.
Backfill runs in bounded project groups and shares a paced write turn across projects. It reads the current row inside its per-row turn, so a live update cannot leave an older snapshot in the search view. Isolated write failures leave rows pending while other content kinds and projects continue. Empty rows are marked complete, and older title-only observations can be requeued for a one-time repair.
A confirmed corrupt collection is marked for rebuild before deletion. A generation change stops writers targeting the dropped collection. Every project then rebuilds from SQLite. The next run retries incomplete projects rather than treating the sweep as complete. Post-delete cleanup runs before these writes and also drains when the Chroma connection recovers.
References: src/…/ChromaSync.ts• L1516-1717 src/…/ChromaSync.ts• L1790-2105 src/…/ChromaSync.ts• L2630-2816 src/…/ChromaSyncState.ts• L213-249 src/…/ChromaSyncState.ts• L419-454
Chroma process lifecycle
The Chroma manager owns the local search process and its connection. It prevents two workers from writing the same persistent store, waits for old subprocess cleanup before reconnecting, and serializes local mutations so an in-flight write is not interrupted by a read timeout or shutdown.
Connection recovery notifies the sync layer so pending row deletes drain as soon as Chroma returns. Failed environment prewarms use a circuit breaker and bounded scratch cleanup instead of spawning endlessly. A persistent store records its writer compatibility epoch, and a newer incompatible writer is refused before it can open the store.
Search reads have a shorter caller budget than the manager's transport deadline. When a read reaches the caller budget, the search falls back without killing a slow healthy process. When the transport itself times out, the manager avoids killing the process while a mutation may still be committing, then restarts it only after the write settles.
References: src/…/ChromaMcpManager.ts• L362-412 src/…/ChromaMcpManager.ts• L446-668 src/…/ChromaMcpManager.ts• L786-1069 src/…/ChromaMcpManager.ts• L1277-1453 src/…/ChromaMcpManager.ts• L1455-1632 src/…/ChromaSync.ts• L617-685
Related pages
- Data and sync gives the broader data and synchronization model.
- Storage backends and data models explains local and cloud storage roles.
- Backup and restore explains how local data and derived search state recover together.
- Sync API service explains the cloud service that accepts operations and serves ordered history.
- Sync hub worker processing explains cloud-side operation processing and history projection.
Change checklist
When these files or sections are changed, remember to consider these:
- Changes to canonical fields, identity rules, revision encoding, media manifests, or
work_statesupport require matching changes in local producers, cloud validation, local apply, and compatibility replay. - Changes to push ordering, request limits, frozen snapshots, dead-letter rules, or acknowledgement checks must preserve forward progress and must stamp only the current revision.
- Changes to device identity resolution must account for frozen operations created under an older identity and must not add a second identity source to local apply.
- Changes to cursor or epoch handling must preserve page contiguity, full replay after a rebuilt history, and the exclusion of replica and launch-baseline rows from requeue.
- Changes to entity heads, tombstones, or equal-revision hash checks must preserve convergence after local deletion and must not accept two bodies for one entity revision.
- Changes to quarantine, savepoint, and head-ledger rules must keep permanent bad operations from blocking the cursor, dependent row errors retryable, and stale revisions unable to replace or resurrect newer rows.
- Changes to session stubs, title parking, prompt repair, or project remaps must preserve delayed association and bounded mutation scope.
- Changes to Realtime notices, head-sequence hints, poll suspension, or socket recovery must keep HTTP pulls as a complete durable recovery path.
- Changes to Chroma document expansion, duplicate reconciliation, fragment deletion, row ordering, or watermark rules must keep row completion tied to every generated document.
- Changes to search over-fetch, row-id narrowing, filter predicates, or the shared search budget must preserve exact scope and leave keyword fallback time within the caller deadline.
- Changes to checkpoint persistence, watermark coalescing, pending-id repair, or rebuild flags must preserve restart recovery when the state file or Chroma is unavailable.
- Changes to post-delete cleanup reservation, checkpoint persistence, retry draining, or row restoration checks must keep deleted-row fragments recoverable without deleting documents for a restored row.
- Changes to Chroma collection recovery must persist rebuild intent before deletion and must stop old-generation writers from repopulating the dropped collection.
- Changes to Chroma process ownership, mutation serialization, timeout handling, writer compatibility, or prewarm recovery must not kill an active commit or allow two writers on one store.
- Changes to inbound redaction, file indexes, or replica media links must keep local privacy and derived indexes aligned with the applied row revision.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| A canonical body or hash is invalid | The cloud rejects it or local apply sets it aside | Shared validation, operation savepoints, and pull quarantine |
| A pulled page has a sequence gap or repeats an earlier sequence out of order | The page cannot safely advance the pull cursor | Contiguous-page checks and one transaction for the page |
| A row changes while a push is in flight | The old acknowledgement does not hide newer local state | Current-revision and canonical-body checks before stamping |
| A frozen operation carries an old device identity | The hub rejects it and it can block later queue work | Stale-identity dead-lettering and native-row resnapshot |
The hub does not advertise work_state | Work-state remains pending | Capability probing and later re-probing |
| The cloud epoch changes | The current page is discarded and native work is made eligible again | Epoch reset, launch exclusions, and idempotent replay |
| A push acknowledgement is incomplete or altered | Sent work remains pending | Exact acknowledgement tuple and per-ack projection-checkpoint proof |
| A Realtime socket fails or is unavailable | Live delivery slows or stops | HTTP polling remains the recovery path |
| Two operations claim one entity revision with different bodies | The first accepted body remains and the later operation is set aside | Entity-head hash check and pull quarantine |
| A pulled row depends on a local row that has not arrived yet | The row is set aside without blocking later history | Retryable constraint quarantine and end-of-batch retry |
| A remote row needs a missing session | Foreign-key application can fail or content can be orphaned | Sync stubs, delayed prompt repair, and title parking |
| Chroma writes only part of a row's documents | Search is incomplete while SQLite remains correct | Pending ids and watermark advancement only after full confirmation |
| A Chroma search uses its full caller budget | Semantic results stop for that request | Shared 20-second budget and SQLite/FTS5 keyword fallback |
| A Chroma checkpoint write fails | Progress may be replayed, but SQLite remains the source of truth | Atomic checkpoint writes, dirty-state retry, and backfill comparison |
| A Chroma row delete fails after SQLite commits | Old search fragments remain with no later source-row scan to remove them | Durable pending row-delete records and retry on backfill or connection recovery |
| A restored row reuses an id before a stale delete retry | The restored row's search documents could be removed | Retry checks SQLite and drops the stale cleanup |
| A Chroma collection has repeated corrupt-segment failures | Search writes stop until rebuild | Persisted rebuild flags, collection generation, and SQLite rebuild |
| A revised or deleted row leaves old fragments | Search returns stale content | Per-row writer ordering and fragment reconciliation |
| A scoped Chroma query uses a broad metadata filter | Search becomes slow or misses the intended result window | Bounded over-fetch, row-id narrowing, and safe client-side filtering |
| A Chroma process dies during reconnect or shutdown | A replacement process can race with old descendants or lose cleanup work | Generation checks, captured process identity, tree cleanup, and one writer lock |
| A prewarm host fails repeatedly | Reconnect attempts waste CPU, disk, and build scratch | Exponential breaker cooldown, bounded scratch sweep, and restart latch |
| A Chroma read times out while a write is active | Killing the process can interrupt a persistent index commit | Leave the process running until mutations settle, then restart |
Key files
| File | Why to read it |
|---|---|
| src/…/CanonicalContent.ts | Defines operation identity, canonical JSON, hashes, revisions, tombstones, content payloads, media manifest limits, and mutation shapes. |
| src/…/CloudSync.ts | Shows how local rows and mutations become durable push requests, how identity changes, capabilities, retries, acknowledgements, and queues work. |
| src/…/SyncClient.ts | Shows how cloud history is pulled, how HTTP owns recovery, and how advisory Realtime delivery affects timing only. |
| src/…/SyncApply.ts | Shows how remote changes converge into rows, entity heads, sessions, titles, indexes, media links, quarantine state, and post-commit Chroma jobs. |
| src/…/ChromaSync.ts | Shows the search projection, deterministic document writes, query scope handling, search deadlines, row ordering, fragment cleanup, durable cleanup recovery, backfill, and collection rebuild. |
| src/…/chroma-search-budget.ts | Defines the shared time limit that lets semantic search fall back to SQLite/FTS5 without cancelling a slow Chroma read. |
| src/…/ChromaSyncState.ts | Defines persisted project watermarks, pending ids, atomic checkpoint progress, fragment repair, rebuild state, and durable post-delete cleanup records. |
| src/…/chroma-row-cleanup.ts | Shows how local and replicated row deletes reserve, serialize, record, and retry Chroma cleanup work. |
| src/…/remap-outbox.ts | Shows how local project remaps and their cloud mutations commit together. |
| src/…/ChromaMcpManager.ts | Explains Chroma process ownership, writer compatibility, mutation ordering, caller deadlines, reconnects, prewarm recovery, and safe shutdown. |