Introducing a new way to review your PRs.

Read the announcement

thedotmack / claude-mem

Persistent Context Across Sessions for Every Agent – Captures everything your agent does during sessions, compresses it with AI, and injects relevant context back into future sessions. Works with Claude Code, OpenClaw, Codex, Gemini, Hermes, Copilot, OpenCode + More

View on GitHub

47 documents · Updated Nov 20, 2026

docs / data-and-sync

Persistence and synchronization

Persistence and synchronization keep memory, project, session, prompt, and event data available on one installation and across linked installations. Local SQLite remains the working store. Cloud sync copies selected records through an ordered per-user history, while search indexes and status views remain derived from durable data. Local backup protects the working store through complete SQLite snapshots.

Data responsibilities

The product separates durable records from transport and search state. This lets local work remain usable when a cloud service or search view is unavailable.

Local records

Local SQLite stores the records that support projects, sessions, prompts, observations, summaries, work state, and sync metadata. Local numeric row ids stay local. Shared content uses a stable origin identity when it crosses an installation boundary.

The local device identity is established in an immediate database transaction before native shared work or cloud sync writes begin. A singleton record in the local database pins the identity, while settings and native shared work can recover it when needed. Conflicting saved identities stop writes until the original identity is restored. If the database already has the identity, a failed settings write does not replace that database identity.

References: src/…/local-device-identity.ts• L15-59 src/…/CloudSync.ts• L693-745 src/…/SyncApply.ts• L16-20

Cloud records

The sync hub stores an ordered history for one user. It accepts canonical copies of observations, summaries, prompts, and supported work-state entries, plus a small set of relationship mutations. The hub also tracks how far its projection has processed that history.

Cloud sync starts only when the token, user id, and hub URL are all present. Older local records are normally treated as a launch baseline. New writes are selected for delivery after that baseline, so enabling sync does not silently upload the full local corpus. The hub advertises optional content kinds, so work-state records remain local until the hub confirms that it accepts them.

References: src/…/CloudSync.ts• L600-608 src/…/CloudSync.ts• L775-795 src/…/CloudSync.ts• L1059-1107 src/…/SyncApply.ts• L500-553

Derived state

Outboxes, cursors, epochs, quarantine records, recovery jobs, and search watermarks describe delivery progress or a read view. They do not replace the local records or the cloud history. A failure can therefore leave durable content in place while a later retry rebuilds the missing transport or search state.

The system also keeps bounded evidence about pending work, excluded history, and semantic-index cleanup. These reads run through separate read workers or bounded local records so queue inspection and historical counts do not block normal HTTP reads or sync writes.

References: src/…/SyncQueueEvidence.ts• L14-47 src/…/SyncQueueEvidence.ts• L55-123 src/…/ChromaSyncState.ts• L11-20 src/…/chroma-row-cleanup.ts• L32-40

Backup and restore lifecycle

Local backup protects the working SQLite store separately from cloud sync. A backup is a complete local snapshot for recovery or transfer. It does not merge two installations or replace the cloud history.

This diagram shows how a live database becomes a retained snapshot and how a staged restore replaces the local store before normal worker startup.

flowchart LR
    A[Live local SQLite] --> B[Consistent snapshot]
    B --> C[Verified backup file]
    C --> D[Retained local snapshots]
    D --> E[Staged restore request]
    E --> F[Validated database swap]
    F --> G[Restored local store]
    G --> H[Search rebuild and cache refresh]

Backup snapshots

Automatic backups start after worker boot and run on a single schedule. Each cycle checks the destination, creates one standalone verified SQLite file, and applies retention after the file is complete. Incomplete files are not treated as usable snapshots.

The snapshot uses SQLite's consistent online copy operation in a child process. It is written under a temporary name and becomes visible only after verification. A single-flight schedule prevents a slow snapshot from creating overlapping cycles, while disk-space checks reject a run that cannot safely create the copy.

References: src/…/BackupManager.ts• L47-75 src/…/BackupManager.ts• L107-205 src/…/BackupManager.ts• L207-227 src/…/sqlite-file-job.ts• L14-20 src/…/sqlite-file-job.ts• L65-83

Restore swaps

Restore requests name one snapshot and are staged through a small local marker. Startup validates the marker and the snapshot before it touches the live database. The old database is kept as a pre-restore copy when possible, then the validated snapshot replaces the live file as one swap.

If the live database is busy or the swap fails, the old store remains in place and the outcome is recorded for status display. A successful restore invalidates session-start cache data and marks semantic search for rebuild because search state describes the replaced database, not the restored rows.

References: src/…/snapshot-files.ts• L12-37 src/…/snapshot-files.ts• L39-79 src/…/restore-swap.ts• L131-153 src/…/restore-swap.ts• L257-355

Synchronization lifecycle

Synchronization has two durable directions. A local write enters the upload queue and an installation reads the ordered cloud history from its own cursor. Both directions use short, bounded requests and can resume after an offline period.

This diagram shows how a local record becomes shared history and how another installation applies that history to its own local store.

flowchart LR
    A[Local durable record] --> B[Canonical upload queue]
    B --> C[Authenticated sync hub]
    C --> D[Ordered user history]
    D --> E[HTTP pull]
    E --> F[Transactional local apply]
    F --> G[Local search views]
    C --> H[Cloud projection]
    I[Advisory live hint] --> E

Upload delivery

The upload path freezes a local record into a canonical operation before it sends the operation. Canonical content has a stable identity, a revision, a payload hash, and an explicit delete form. This makes a retry distinguishable from a changed record.

The uploader sends bounded pages. It changes local delivery state only after the hub confirms the exact operations and confirms that its cloud projection covers them. A malformed or incomplete response leaves the queue unchanged. A newer local edit keeps a higher revision and remains pending. Deletes use tombstones so another installation can learn about a deleted record after the local row is gone.

Relationship changes use the same ordered queue. A title update, prompt-to-session repair, or project remap waits behind the content it affects when needed. Invalid work can move to durable dead-letter evidence, while auth failures pause delivery instead of causing an endless retry ladder.

References: src/…/CanonicalContent.ts• L7-13 src/…/CanonicalContent.ts• L151-241 src/…/CloudSync.ts• L820-913 src/…/CloudSync.ts• L1047-1107 src/…/SyncQueueEvidence.ts• L55-123

History application

The receiving installation pulls pages after its stored cursor and applies each page to SQLite in one transaction. The row changes and the cursor advance commit together. A restart therefore sees either the whole page or none of it.

The receiver uses the hub epoch to detect a rebuilt history. It resets the cursor and re-reads the new history. Eligible native records can be queued again, while the launch baseline remains excluded. Remote rows are marked as already synchronized so they cannot echo back as local uploads.

Each operation has its own savepoint. A malformed, unsupported, or permanently conflicting operation is quarantined and does not block later history. A temporary storage or page-order failure rejects the whole page so the client retries it. A constraint failure that may be fixed by a later operation remains retryable. Inbound memory fields can also be redacted by the receiving installation before they are stored. The stored content hash remains the origin's identity value.

Rows deleted by local or remote operations also schedule cleanup of their semantic fragments after the durable delete commits. The cleanup record is reserved before queued deletion work starts, so a worker exit cannot lose the only record of fragments that no longer have a SQLite row. Cleanup retries share one serialized lane with live deletes and backfill writes. If the search service is down, the cleanup remains recorded and is retried when the service returns. If a row id is restored before a deferred cleanup runs, the stale cleanup is discarded so it cannot remove the restored row's current fragments.

References: src/…/SyncClient.ts• L689-785 src/…/SyncApply.ts• L559-723 src/…/SyncApply.ts• L975-1047 src/…/ChromaSync.ts• L1214-1289 src/…/ChromaSyncState.ts• L11-29 src/…/ChromaSyncState.ts• L347-404 src/…/chroma-row-cleanup.ts• L17-101

Live update hints

The optional live channel reduces the delay before a client pulls new history. It sends only an epoch and a head position. HTTP history pulls remain the source of records and the only owner of the cursor.

The client falls back to polling when the live service is absent, unavailable, or sends an invalid frame. The hub can also require polling through a mode hint during an incident. Polling closes gaps after reconnect. The client can slow or suspend background work when there is no active session, then resume when a session starts or a push response shows new history.

References: src/…/SyncClient.ts• L1-58 src/…/SyncClient.ts• L426-439 src/…/SyncClient.ts• L851-877 src/…/SyncClient.ts• L1058-1159

Identity and consistency

Sync consistency comes from stable identity, ordered revisions, and retained heads. These rules protect records when devices retry, delete, restore, or edit the same content at different times.

Canonical content

Each shared content record is identified by its content kind, origin device, and origin local id. Its entity revision and operation hash describe the accepted version. Canonical JSON and bounded schemas make the same logical payload produce the same bytes on every side. Relationship changes use a separate canonical mutation form rather than pretending to be content rows.

The receiver keeps a head even after it deletes a row. An older live operation cannot recreate that row. If two different bodies claim the same entity revision, the first accepted body remains authoritative and the other operation is quarantined.

References: src/…/CanonicalContent.ts• L78-166 src/…/CanonicalContent.ts• L169-241 src/…/SyncApply.ts• L975-1047 src/…/SyncConflictEvidence.ts• L11-29

Session row ids do not cross installations. A receiving installation creates a small session stub when shared content arrives before its local session link. Prompt and title links can be repaired later when the missing identity becomes known.

Project changes travel as bounded mutations instead of unrestricted row rewrites. This keeps project remaps ordered and lets the receiver update related observations, summaries, prompts, and sessions without trusting another installation's numeric ids.

References: src/…/SyncApply.ts• L108-145 src/…/SyncApply.ts• L932-1010 src/…/CloudSync.ts• L820-828

Conflict repair

Conflict repair is an explicit origin-device action for a local record whose revision history no longer matches the hub. The system first reads bounded evidence from local heads, outboxes, quarantine, and dead letters. It then compares that evidence with the hub head before it offers the current local record as the repair choice.

An accepted repair uses a new revision and records a reservation before the remote request. That reservation protects the revision if the reply is lost or a later native write occurs. The repair reports whether the local and remote evidence was fully verified. It does not replay retained quarantine bodies as new content.

References: src/…/SyncConflictEvidence.ts• L43-95 src/…/SyncConflictRecovery.ts• L52-70 src/…/SyncConflictRecovery.ts• L110-140 src/…/SyncRepairReservation.ts• L4-21

Search projections

Search views are rebuilt from durable records. They improve recall but do not own memory data, sync identity, or sync progress.

SQLite keyword search follows the same base-table writes used for local records and applied remote records. A synchronized record becomes searchable through the normal local data path rather than a separate import step.

References: src/…/SyncApply.ts• L147-155 src/…/SyncApply.ts• L706-717 src/…/memory-items.ts• L111-291

Chroma stores searchable fragments for observations, summaries, and prompts. It can use a local persistent store or a remote Chroma service. A single writer owns a local store, and a store record prevents an older writer from opening a newer store format.

The Chroma service is optional and managed as a separate search process. Startup checks the local store boundary before it connects. Repeated setup failures pause or stop new prewarm attempts instead of consuming resources forever, while later probes can recover a host that becomes healthy. A connection recovery also resumes recorded search cleanup work.

Deterministic fragment ids let retries update existing documents instead of creating duplicates. A revised row removes fragments that no longer exist, and a deleted row removes all of its fragments. Chroma progress advances only after the complete document set for a row is confirmed. Partial writes stay pending instead of advancing the row watermark, and rows with no searchable content are drained so they are not reported as missing on every sweep.

Semantic reads share one bounded search budget for the whole search request. If Chroma does not answer in time, the caller stops waiting and uses the SQLite keyword path. The slow provider read is not cancelled at that point, so a healthy but slow search does not interrupt a write or corrupt the persistent index.

If Chroma is unavailable, durable memory and cloud sync continue without the semantic view. A confirmed corrupt collection is dropped and rebuilt from SQLite. A failed post-delete cleanup stays in the local Chroma checkpoint until a later sweep or connection recovery removes the orphaned fragments. Rebuild state is persisted before a corrupt collection is dropped, so an interrupted rebuild starts again from durable SQLite rows. Checkpoint progress is coalesced for normal backfill work, but delete cleanup is recorded before the worker starts that cleanup so a restart can resume it.

References: src/…/ChromaMcpManager.ts• L459-607 src/…/ChromaMcpManager.ts• L736-783 src/…/ChromaMcpManager.ts• L1277-1443 src/…/ChromaMcpManager.ts• L1463-1553 src/…/ChromaSync.ts• L1088-1264 src/…/ChromaSync.ts• L2322-2449 src/…/ChromaSync.ts• L2635-2686 src/…/chroma-search-budget.ts• L1-47 src/…/ChromaSyncState.ts• L79-137 src/…/ChromaSyncState.ts• L347-454 src/…/chroma-row-cleanup.ts• L32-67

Synchronization visibility

The product exposes sync status as evidence about upload, download, projection, and completeness. It does not treat an open connection or an empty queue as proof that all installations are current.

The viewer combines local queue state with an authenticated hub check, the local download cursor, projection progress, quarantine counts, and the known historical scope. It reports distinct states such as current, catching up, paused, offline, partial, stale, or unknown. Status reads are observational and do not schedule uploads, repair records, or move cursors.

References: src/…/CloudSync.ts• L570-598 src/…/SyncQueueEvidence.ts• L125-193 src/…/SyncClient.ts• L426-439 src/…/viewer-sync-status.ts• L1-30 src/…/viewer-sync-status.ts• L48-76

Recovery and safety controls

Recovery separates diagnosis from mutation. Read-only inventory and preview work runs in dedicated workers or bounded transactions. Local recovery actions are allowed only from the local loopback boundary and require an explicit, bounded scope.

Selected history recovery

Selected history recovery can requeue native records that were excluded from the normal launch scope. A preview fixes the project, date range, content kinds, device identity, epoch, revisions, hashes, and byte estimate. Creating a job checks that preview again before it stores the selection.

Jobs move through paused, running, cancelled, and complete states. Each selected item records whether it was queued, acknowledged, superseded, deleted, or failed. An epoch change pauses the job and prevents an acknowledgement from the old history from being treated as proof for the new history.

References: src/…/SyncRecovery.ts• L7-18 src/…/SyncRecovery.ts• L44-139 src/…/SyncRecovery.ts• L170-227 src/…/SyncRecovery.ts• L230-287

Queue diagnosis

Queue evidence reports native pending rows, frozen operations, tombstones, mutation blocks, unsupported work-state records, and shared-state holds. It keeps native records distinct from frozen snapshots so a status view does not count one logical change twice.

Mutation scheduling keeps affected content behind the prerequisite rows and preserves FIFO order for mutations. This avoids a silent state where a mutation is accepted but the content it changes has not yet reached the receiving installation.

References: src/…/SyncQueueEvidence.ts• L14-47 src/…/SyncQueueEvidence.ts• L55-123 src/…/CloudSync.ts• L820-881

Diagnostic redaction

Sync diagnostics treat rejected bodies, nested JSON, field names, and error text as untrusted. Redaction applies a node and depth budget. A transformed diagnostic is not used as a replay source, which prevents a safe error record from becoming a new sync operation.

References: src/…/SyncApply.ts• L903-948 src/…/SyncApply.ts• L846-895 src/…/SyncRecoveryRoutes.ts• L92-114

Change checklist

When these files or sections are changed, remember to consider these:

  • Changes to shared content kinds or payload fields must stay aligned across canonical validation, upload selection, hub capability checks, receiver apply, recovery snapshots, and search projections.
  • Changes to device identity must preserve one origin for native shared records, cloud upload, receiver echo suppression, and repair authority.
  • Changes to revisions, heads, tombstones, or hashes must preserve first-write conflict handling and must not let an older replay resurrect deleted content.
  • Changes to mutation targets or prerequisites must preserve ordering between affected content and relationship changes.
  • Changes to epoch reset behavior must keep cursor reset, eligible native requeue, launch exclusions, and recovery-job pausing consistent.
  • Changes to live updates must keep the channel advisory. Records must still arrive through HTTP pulls and the cursor must remain owned by local apply.
  • Changes to queue evidence or history counts must keep inventory reads off the HTTP and write paths and must distinguish native rows from frozen snapshots.
  • Changes to recovery preview, acknowledgement, or repair reservation rules must prevent stale local evidence or a prior epoch from being treated as current proof.
  • Changes to diagnostic redaction must preserve safe logging and must never turn transformed evidence into replayable sync content.
  • Changes to Chroma fragments, row watermarks, row-delete checkpoints, or rebuild behavior must preserve durable local records when the semantic view is partial or unavailable. Delete checkpoints must be durable before queued cleanup work can run and must not be cleared while another cleanup for the same row remains queued.
  • Changes to semantic search budgets or fallback paths must keep the whole search within its caller deadline without cancelling a provider read that may still be serving a write.
  • Changes to viewer status must keep transport liveness separate from replica freshness and completeness.
  • Changes to hub capability checks or push-response validation must keep unsupported work-state records pending and must prevent local delivery state from changing without a complete acknowledgement proof.
  • Changes to inbound redaction must preserve the distinction between safe stored display fields and the origin content hash used for identity.
  • Changes to snapshot validation, restore swapping, or retention must preserve complete snapshot files and must trigger semantic-index rebuild after a successful database replacement.

Important failure modes

TriggerConsequenceGuard
A local record changes while its upload snapshot is in flightThe old acknowledgement could hide newer contentCurrent revision and canonical operation are checked before the row is stamped delivered
A hub does not advertise work-state supportWork-state rows could be sent to an older or restricted hubThe client checks the advertised content kinds and keeps unsupported work state pending for a later probe
A pull page contains a permanent bad operationOne bad record could block all later historyThe operation is isolated and quarantined while the cursor advances
A pull page has a gap or transient storage errorLocal state could skip part of historyThe whole page fails and the same cursor is retried
A push response is incomplete or does not match the sent operationsLocal rows could be marked delivered without proofResponse validation finishes before any acknowledgement state changes
The hub history epoch changesThe old cursor no longer identifies the same historyThe cursor resets and eligible native revisions are requeued
Two bodies claim one entity revisionDifferent installations could silently divergeThe first accepted head remains authoritative and the later body is quarantined
A remote row arrives without its local sessionThe row could violate local links or lose its contextThe receiver creates a session stub and repairs links when identity data arrives
A repair reply is lost after the hub accepts itA later native write could reuse the accepted revisionThe local repair reservation remains durable until later revisions pass it
A recovery job crosses an epoch changeOld acknowledgements could be counted as proof for new historyThe job pauses and clears stale queued or acknowledged item state
Chroma accepts only part of a row's fragmentsA watermark could skip missing search documentsThe row stays pending until the full document set is confirmed
A Chroma row delete fails after the SQLite row is goneOrphaned search fragments could remain foreverThe cleanup is reserved in the Chroma checkpoint before queueing and retried on a sweep or reconnect
A worker exits while a row cleanup waits behind another Chroma operationThe deleted row's fragments could lose their only cleanup recordThe checkpoint is written before the cleanup lane starts and duplicate reservations remain until every queued cleanup finishes
A row id is restored or reused before a deferred cleanup runsA stale delete could remove the restored row's current search fragmentsThe retry checks that the source row is still absent and drops the stale cleanup when the row exists again
A local Chroma store is opened by two writersSearch data could be corrupted or writes could be lostA persistent writer lock and process identity check allow one live writer
Chroma setup fails repeatedlyPrewarm attempts can waste CPU, disk, and build scratch while semantic search stays unavailableA circuit breaker uses cooldowns and a hard stop, while later probes allow recovery before restart
A semantic search exceeds its caller budgetThe request could miss its response deadline even though keyword search could answerThe whole search shares a bounded Chroma wait and then falls back to SQLite without cancelling the provider read
The live channel is down or sends a bad hintUpdates may arrive later than expectedHTTP polling remains the durable pull and recovery path
Sync credentials are refusedRepeated retries waste work and hide the causeUploads enter an auth pause and use a later authenticated status check
A receiver applies a pulled body with sensitive fieldsPrivate text could remain unredacted on that installationThe receiver applies its local redaction policy before storing supported memory fields
A snapshot is interrupted or lacks enough disk spaceA backup could be mistaken for a complete recovery sourceThe snapshot is written under a partial name, verified, and renamed only after completion
A restore target is outside the backup directory or has an unsupported schemaAn unsafe or unreadable database could replace the live storeThe target is resolved inside the backup directory and validated before the swap
The live database is busy during restoreThe worker could lose access to an active storeRestore retries within its window, then leaves the live database unchanged

Key files

FileWhy to read it
src/…/DatabaseManager.tsShows how local storage, device identity, search, cloud sync, recovery, media, and backup are brought together.
src/…/BackupManager.tsShows the scheduled local snapshot, verification, single-flight behavior, and retention lifecycle.
src/…/restore-swap.tsShows the guarded database replacement and post-restore rebuild actions.
src/…/snapshot-files.tsShows the snapshot names and the staged restore records shared by CLI and worker.
src/…/local-device-identity.tsShows how one local origin identity is recovered, pinned, and protected.
src/…/CanonicalContent.tsDefines the shared identity, revision, hash, payload, mutation, and tombstone contract.
src/…/CloudSync.tsShows how local records and mutations enter the durable upload path and how acknowledgements affect local state.
src/…/SyncApply.tsShows how pulled history becomes local rows, relationships, quarantine records, and later search updates.
src/…/SyncClient.tsShows HTTP history pulls, epoch handling, polling cadence, and advisory live updates.
src/…/SyncQueueEvidence.tsShows how pending work and its hold reasons become safe status evidence.
src/…/SyncRecovery.tsShows bounded selected-history recovery and its job and item states.
src/…/SyncConflictEvidence.tsShows the bounded evidence used before a conflict repair can proceed.
src/…/ViewerSyncService.tsShows how sync status is observed without making a status read part of the sync work path.
src/…/viewer-sync-status.tsDefines the user-facing distinction between current, catching up, paused, offline, partial, stale, and unknown.
src/…/ChromaMcpManager.tsShows the local or remote Chroma boundary, managed process recovery, prewarm limits, and single-writer safety for local stores.
src/…/ChromaSync.tsShows how durable memory feeds a replaceable semantic search view, how scoped queries fall back safely, and how deferred row cleanup is retried.
src/…/chroma-search-budget.tsDefines the shared wait limit that lets semantic search fall back to keyword search without cancelling a provider operation.
src/…/ChromaSyncState.tsShows durable Chroma watermarks, coalesced progress, rebuild flags, and pending row-delete work.
src/…/chroma-row-cleanup.tsShows how committed deletes reserve cleanup work, serialize Chroma operations, and preserve retries across queued duplicate deletes.