Introducing a new way to review your PRs.

Read the announcement

thedotmack / claude-mem

Persistent Context Across Sessions for Every Agent – Captures everything your agent does during sessions, compresses it with AI, and injects relevant context back into future sessions. Works with Claude Code, OpenClaw, Codex, Gemini, Hermes, Copilot, OpenCode + More

View on GitHub

47 documents · Updated Nov 20, 2026

docs / storage-backends-and-data-models

Storage backends and data models

The storage layer preserves the product's captured work and the state needed to process it. SQLite supports the local worker and its older searchable data set. Postgres supports the scoped server data model with teams, projects, observations, authentication, usage, and operational records.

Persistence boundaries

The system has two persistence boundaries rather than one shared database schema. The local boundary keeps worker data and sync state available on a device. The server boundary keeps team-owned data and its processing history available to cloud services.

BoundaryMain purposeData model shape
Legacy worker SQLitePreserve sessions, observations, summaries, prompts, pending work, local sync state, and local delivery stateInteger row IDs with project names and session IDs
Server-owned SQLiteProvide a local server storage boundary for projects, sessions, events, memory items, teams, keys, and audit recordsUUID-like text IDs with project links
PostgresStore team-scoped server data and cloud processing stateText IDs with explicit team_id and project_id scope

The server-owned SQLite tables are additive. They do not replace the legacy observations, session_summaries, or user_prompts readers. A later migration must translate legacy observations into memory items before server-owned readers can treat those items as the source of truth. This separation prevents a local schema change from silently changing search or sync behavior. (See the related data and sync page.)

Data ownership model

Projects and sessions

Projects group captured data into a product scope. A local project can use a name and root path. A server project belongs to a team. Server sessions then attach incoming activity to that project and retain links to content-session and memory-session identities.

The local worker identifies a session with its platform source and content session ID. This pair keeps equal IDs from different agent platforms separate. User prompts also keep a database-session link so a prompt stays attached to the correct local parent while retaining the content session ID for compatibility.

Server session identity is designed to be stable when an agent sends the same session more than once. External session IDs, content session IDs, agent identity, and platform source can contribute to that identity. This lets different agent platforms use the same project without treating equal IDs from different platforms as one session. Server sessions also track generation state and provide the bounded event view used to build a session summary.

References: plugin/…/SessionStore.js• L647-746 plugin/…/SessionStore.js• L763-783 src/…/schema.ts• L24-68 src/…/projects.ts• L19-65 src/…/server-sessions.ts• L10-90 src/…/schema.ts• L149-167 src/…/server-sessions.ts• L9-187 src/…/server-sessions.ts• L190-320 src/…/projects.ts• L6-64

Events and observations

Agent events are the raw record of activity received from hooks, workers, providers, servers, or APIs. Observations are the durable knowledge produced from that activity. The Postgres model keeps both records so the system can trace a stored observation back to the event or session that produced it.

The local worker still stores observations as part of its legacy content model. It uses a content hash within a memory session to avoid storing the same generated observation more than once. The server model stores observations as project- and team-owned records with searchable content, optional metadata, and an optional legacy embedding value. When semantic search is available, separate model-tagged vectors represent parts of an observation so long content can be searched by meaning without replacing the observation record.

Event and session writes remove private sections and apply the configured secret redaction at the persistence boundary. This keeps downstream readers from depending on every event producer to clean its payload first. Server event records also retain the platform source so project reads can apply the same platform scope as session reads.

Server observation reads stay inside a project and team scope. They can return recent observations when no search text is given, search indexed content when text is supplied, or combine keyword and semantic results when a query vector is available. Platform and folder filters narrow the same project view. A summary filter separates summaries from observations, while an optional main-agent filter excludes observations tied to subagent events. This keeps context reads aligned with the session and agent that requested them.

References: src/…/schema.ts• L70-105 src/…/agent-events.ts• L10-85 plugin/…/SessionStore.js• L792-810 src/…/schema.ts• L169-237 src/…/schema.ts• L481-500 src/…/agent-events.ts• L18-143 src/…/observations.ts• L16-262 src/…/observations.ts• L294-541 src/…/observation-embeddings.ts• L6-89

Source lineage and generation work

Observation sources describe why an observation exists. They can point to an agent event, a session summary, an observation reindex, or a manual action. Generation jobs turn those sources into durable observations through a tracked queue lifecycle.

The job records protect the boundary between raw input and derived knowledge. A job has one source model, one project and team scope, an idempotency key, retry limits, and a lifecycle event trail. This makes retries safe and gives operators a record of whether generation was queued, processed, completed, failed, or cancelled. Stale processing locks can be reclaimed so a worker that stops mid-job does not leave work blocked forever.

References: src/…/schema.ts• L188-266 src/…/generation-jobs.ts• L15-161 src/…/generation-jobs.ts• L180-297 src/…/observations.ts• L265-424

Authentication and operational records

Authentication data is persisted as API-key records rather than raw secrets. A key may belong to a team or project and carries scopes, expiry, and revocation state. The stored non-secret prefix narrows active-key candidates before the raw key is checked, while a successful lookup can upgrade an older hash to the salted scheme. Audit records preserve who or what performed an action and which project or team it affected.

Postgres also stores append-only usage events and fixed-window rate-limit counters. Usage events support team totals for quotas and later billing. Rate-limit counters support a shared request limit across server instances. These records serve control and operations needs rather than the searchable memory model.

The legacy worker keeps a separate SQLite ledger for session-end wrap-up delivery. Its identity includes the platform, content session, project, and route. A claim can become sent or be reclaimed after it goes stale, so repeated end events do not normally send the same wrap-up twice while an interrupted delivery can recover.

References: src/…/schema.ts• L35-53 src/…/schema.ts• L119-150 src/…/schema.ts• L286-318 src/…/schema.ts• L494-515 src/…/auth.ts• L6-130 src/…/auth.ts• L133-212 src/…/usage.ts• L11-59 src/…/rate-limit.ts• L10-35 plugin/…/SessionStore.js• L631-643 plugin/…/SessionStore.js• L830-857

Repository boundaries

Repository responsibilities

Repositories are the boundary between application records and database rows. They accept a domain-shaped input, enforce the storage contract, write the row, and return a domain-shaped record. SQLite repositories ensure the server-owned schema exists before use. Postgres repositories require the caller to provide the team and project scope for server-visible reads and writes.

The Postgres repository set is assembled as one storage interface. It covers teams, projects, authentication, sessions, agent events, observations, source links, generation jobs, usage, and rate limits. This keeps each data family separate while allowing a request flow to use one consistent server storage handle.

References: src/…/index.ts• L3-8 src/…/index.ts• L3-59 src/…/project.ts• L5-26 src/…/session.ts• L5-37 src/…/agent-event.ts• L5-38 src/…/memory-item.ts• L5-72 src/…/auth.ts• L5-69

Scope enforcement

The server model treats (team_id, project_id) as an ownership boundary. A project must belong to the team named by a request. Sessions, events, observations, source links, and generation jobs must remain inside that same scope. Reads use scoped lookup methods so an object in another team or project appears as absent rather than becoming visible.

Database foreign keys reinforce the same relationship. Cascades remove dependent content when a team or project is removed. Source rows follow their observations. Audit rows can remain after the referenced scope is removed because their record has operational value.

References: src/…/utils.ts• L51-80 src/…/schema.ts• L99-237 src/…/observations.ts• L75-153 src/…/observations.ts• L265-352 src/…/generation-jobs.ts• L318-366 src/…/data-deletion.ts• L21-52

Serialization boundary

JSON values

The domain model uses objects and arrays for metadata, payloads, scopes, facts, concepts, and file lists. SQLite stores these values as JSON text because its server-owned schema uses text columns. Postgres stores the same kind of flexible data as JSONB.

The repository mapping layer converts database values back into domain values. SQLite treats malformed object and array text as empty values and records a warning. Postgres accepts JSON values from the driver and normalizes non-object or non-array results at the repository boundary. Legacy observation file lists use a narrower rule: JSON arrays stay arrays, scalar JSON values become one path, and malformed text becomes one path so older stored values remain usable.

References: src/…/serde.ts• L5-31 src/…/memory-items.ts• L20-83 src/…/auth.ts• L49-79 src/…/utils.ts• L17-26 src/…/observations.ts• L737-765 src/…/files.ts• L4-13 plugin/…/files.js• L4-12

Time and identity values

Application records expose time as epoch numbers. SQLite stores epoch integers in the server-owned tables and in the legacy worker tables. Postgres stores timestamps and converts them to epoch numbers when it returns a domain record.

The backends also use different identity styles. Legacy SQLite content uses integer row IDs with session IDs as links. Server-owned SQLite and Postgres records use generated text IDs. Postgres adds deterministic idempotency keys for repeated sessions, events, observations, and generation jobs so a retry updates or returns the existing logical record. Session and event keys include the platform when one is known. Local sync revisions use canonical unsigned decimal text so large revision values remain exact across SQLite reads and writes.

References: src/…/database.ts• L25-76 src/…/schema.ts• L24-105 src/…/utils.ts• L13-39 src/…/server-sessions.ts• L360-407 src/…/agent-events.ts• L147-185 src/…/generation-jobs.ts• L431-445 plugin/…/SessionStore.js• L157-187

SQLite persistence

Legacy worker store

The legacy worker store is the local source for session context and search. It keeps sdk_sessions as the session parent and links observations, session summaries, and user prompts to that session. Its schema has grown through in-place checks and migrations so existing local databases can retain their data as the product adds prompt, platform, and sync support.

The local session boundary now distinguishes platform source as part of session identity. Prompt rows link to the numeric session parent, while legacy content-session fields remain available for lookup and import. This prevents a prompt or session lookup from crossing between agent platforms that use the same content session ID.

Observation file lists are stored as JSON text in the legacy tables. When a session summary does not provide its own file lists, the store derives files_read and files_edited from the session's observations. The derived lists remove duplicates while keeping first-seen order. An explicit summary list takes precedence over this rollup.

The store also records session-end wrap-up claims in the local database. The ledger uses the session and delivery route as its identity, records whether delivery is claimed or sent, and reopens claims that remain stale. This keeps delivery coordination local to the worker without changing the observation or sync data model.

Its sync state is part of the local persistence boundary. Content rows carry sync timestamps, origin identity, and revisions. Separate outboxes and ledgers hold pending content or mutation work. Entity heads track the latest local state. Dead-letter records retain work that cannot proceed. A launch baseline marks pre-existing local rows so the sync service does not treat all old data as new work. This state supports the sync system but does not become part of the server-owned memory tables. (See the sync application protocol page.)

References: plugin/…/SessionStore.js• L268-331 plugin/…/SessionStore.js• L631-643 plugin/…/SessionStore.js• L637-646 plugin/…/SessionStore.js• L778-912 plugin/…/SessionStore.js• L830-857 plugin/…/SessionStore.js• L157-252 src/…/SessionStore.ts• L53-76 src/…/SessionStore.ts• L2850-2915

Server-owned SQLite store

The server-owned SQLite store gives local server workflows a compact version of the server data boundary. It creates projects, server sessions, agent events, memory items, memory sources, teams, team members, API keys, and audit logs without requiring the legacy worker tables.

Memory items are the future unified memory shape. They support observations, summaries, prompts, and manual items. A source row can retain a link to a legacy observation. Unique constraints make that link safe to backfill more than once. Full-text search covers memory item text and is kept aligned as items change.

The memory-item repository validates domain input before it writes a record and maps stored JSON text back to arrays and objects when it reads one. Partial updates preserve fields that are omitted and apply an explicit null where the domain allows it. Project search supports normal word forms and compatible Unicode forms, with literal checks for symbol-only terms. Memory sources retain the origin of an item, which lets local server workflows connect the unified shape to legacy observations or other source types. (See the related data and sync page.)

References: src/…/schema.ts• L5-17 src/…/schema.ts• L85-117 src/…/schema.ts• L164-204 src/…/agent-events.ts• L23-68 src/…/server-sessions.ts• L24-90 src/…/memory-items.ts• L86-290 src/…/memory-item.ts• L5-84 docs/server-storage-boundary.md• L25-48

Postgres persistence

Schema bootstrap

Postgres creates the server schema inside a transaction. A migration record identifies the schema version. The bootstrap is safe to call for a pool or an already connected client, and a failed bootstrap rolls back instead of leaving a partial server model. Standalone API-key commands can apply the key-prefix column and index without running the full server bootstrap.

The schema includes the core ownership graph and the operational records around it. Postgres full-text search uses a generated search value on observation content. When the database supports the needed extensions, the schema can also use BM25 keyword ranking and a separate vector table for semantic search. Vector setup is optional, and existing deployments can remain keyword-only when pgvector is unavailable, too old, or disabled.

The vector table stores one model-tagged vector for each searchable observation part. Parts include narrative text, legacy text, facts, or summary fields. The embedding repository limits fact parts and finds observations missing vectors for a model so a backfill can progress in stable time-and-ID order. A change to vector width drops the old vector table so rows can be embedded again in the new space.

The connection boundary supplies a database URL, bounded pool and query timeouts, and an explicit SSL mode when the deployment requires it. An explicit SSL setting takes precedence over URL and process-level SSL modes. Verified modes require certificate checks, while encryption-only modes do not. Indexes support scope, time, platform, source, job status, usage, rate-limit reads, full-text search, and semantic nearest-neighbor reads.

References: src/…/schema.ts• L6-103 src/…/schema.ts• L105-227 src/…/schema.ts• L251-516 src/…/schema.ts• L518-538 src/…/config.ts• L4-45 src/…/config.ts• L57-97 src/…/observation-embeddings.ts• L11-18 src/…/observation-embeddings.ts• L69-165

Observation lifecycle

The main server persistence flow turns incoming agent activity into searchable knowledge. The event is first stored in its project and team scope. A generation job records the work to perform. The resulting observation stores content and points back to its source and job. If semantic search is enabled, an embedding pass stores model-specific vectors for the observation's searchable parts.

The following flow shows the relationship between raw activity, processing state, durable knowledge, and the optional semantic index.

flowchart LR
  A[Agent activity] --> B[Agent event]
  B --> C[Generation job]
  C --> D[Observation]
  B --> E[Observation source]
  C --> E
  E --> D
  D --> F[Observation embeddings]
  D --> G[Keyword and API reads]
  F --> H[Hybrid semantic reads]

Idempotency keys make retries converge on the same event, job, or observation. Source and ownership checks prevent a valid record from being linked to a source in another scope. Observation reads can use recent-first context retrieval or indexed search with platform, folder, summary, and agent-origin filters. Hybrid reads combine keyword rank with the nearest matching vector parts and only compare vectors made by the requested model. The local worker's observation rows feed the earlier transcript-to-observation flow before any server-side record is created. (See the transcript observation processing page.)

References: src/…/agent-events.ts• L72-143 src/…/generation-jobs.ts• L104-161 src/…/generation-jobs.ts• L180-297 src/…/observations.ts• L75-262 src/…/observations.ts• L294-541 src/…/observation-embeddings.ts• L31-66 src/…/observation-embeddings.ts• L95-165 plugin/…/SessionStore.js• L797-810

Operational lifecycle

Operational data records the controls around server use rather than the content itself. Authentication checks API-key state and first narrows active candidates by their non-secret prefix. Request handling can append usage. The rate limiter increments a team-independent subject counter in Postgres. Audit logging records the action and its target.

Project memory deletion uses a transaction and removes observations, jobs, raw events, and sessions while keeping the project shell. This separates removal of captured content from removal of team membership or project configuration.

References: src/…/auth.ts• L63-130 src/…/auth.ts• L133-212 src/…/usage.ts• L13-59 src/…/rate-limit.ts• L17-35 src/…/data-deletion.ts• L18-52

Change checklist

When these files or sections are changed, remember to consider these:

  • Changes to a domain record must stay aligned across its repository mapping, schema contract, and both database schemas when both backends expose that record.
  • Changes to project or team links must preserve scoped lookup conditions, composite foreign keys, and cascade or set-null behavior.
  • Changes to JSON fields must preserve the SQLite text representation and the Postgres JSONB representation so reads return the same domain shape.
  • Changes to session, event, job, or observation identity must preserve the matching idempotency key and its unique scope.
  • Changes to session or event platform fields must preserve platform-aware lookup, indexes, and the parts of the idempotency key that prevent cross-platform collisions.
  • Changes to raw event or session metadata writes must keep private-section removal and secret redaction on every ingest path.
  • Changes to local session identity or prompt ownership must preserve the platform-source and content-session pair and the numeric session-parent link.
  • Changes to session-end wrap-up identity or routes must preserve the platform, content session, project, and route key used by the local delivery ledger.
  • Changes to sync revision fields must preserve canonical text handling, exact value checks, and the relation between content heads and outbox rows.
  • Changes to legacy SQLite content must account for sync origin, revision, outbox, and ledger state when the row body or identity changes.
  • Changes to legacy observation file-list encoding or parsing must preserve array order, duplicate removal in summary rollups, scalar fallback behavior, and the summary-over-observation precedence rule.
  • Changes to memory-item backfill fields must preserve the partial uniqueness rules that prevent duplicate legacy sources.
  • Changes to memory-item create, update, list, or search fields must stay aligned across MemoryItemSchema, SQLite row mapping, JSON encoding, Unicode query handling, and the FTS index inputs.
  • Changes to observation search filters must preserve the project and team scope and the matching behavior for recent reads, platform, folder, summary, and agent-origin filters.
  • Changes to observation content must preserve the search index path for both SQLite FTS5 and Postgres full-text search.
  • Changes to observation embedding parts must preserve part names, fact limits, model identity, and the relation between backfill cursors and hybrid search joins.
  • Changes to vector width or extension setup must preserve the keyword-only fallback and the rule that vectors from different models are not compared.
  • Changes to generation-job sources, status transitions, or retry handling must preserve source ownership, terminal states, stale-lock recovery, and the surviving job ID in merged payloads.
  • Changes to API-key hashing or lookup must preserve the non-secret prefix column and index, the active-key filter, the candidate limit, and the legacy-hash upgrade path.
  • Changes to the Postgres schema or connection settings must preserve transactional bootstrap, standalone API-key schema upgrades, migration versioning, pool limits, query timeouts, SSL-mode precedence, extension fallback, and the deployment's SSL expectation.

Important failure modes

TriggerConsequenceGuard
A server record is read without its team and project scopeData from another tenant can appear in an API resultPostgres repositories expose scope-aware reads and ownership checks
A retry changes the inputs used to make an idempotency keyOne logical event or observation can become duplicate rowsBuild keys from stable source identity and canonical JSON
A platform value is omitted or changed on a repeated session or eventActivity from different agent platforms can merge or fail to matchNormalize platform source and keep it in session, event, lookup, and key scope
A raw event or session metadata path bypasses redactionPrivate content or secrets can persist in the databaseApply tag removal and redaction at the repository write boundary
A SQLite JSON field is changed without updating its decode pathDomain code receives text or empty data instead of objects and arraysKeep JSON text writes and repository parsing paired
A legacy file list is written in an unexpected formatSearch or session summaries can treat the value as one path instead of the intended listKeep file-list producers JSON-compatible and retain the parser's scalar and malformed-text fallback rules
A local sync revision is stored with numeric affinity or non-canonical textLarge revisions can lose precision or fail protocol checksValidate unsigned decimal strings and keep revision columns text-valued
A prompt is resolved by content session ID alone when platforms reuse that IDA prompt can attach to or read from the wrong local sessionUse platform source with content session identity and the session-parent link
A wrap-up claim uses a different route or session identity on retryA delivery can repeat or fail to match the existing claimUse the composite ledger identity and reclaim only stale claims
A generation job source does not match its event, session, or observationWork can produce knowledge outside the intended lineageValidate source ownership and match the source model before writing the job or source link
A conflicting generation-job insert leaves a new job ID in the surviving payloadA worker cannot lock the job named by its payload and silently drops workRe-pin the payload's generation-job ID to the row that won the conflict
A worker dies while a job is processing and no stale reclaim runsThe job remains locked and never reaches retry or failureReclaim stale processing rows under a row lock and respect the attempt limit
A memory item is backfilled without its legacy source identityRepeated backfill can create duplicate memory recordsUse legacy observation and source uniqueness constraints
A memory-item field is updated without updating the FTS trigger or indexed inputProject search returns stale or incomplete resultsKeep memory item writes and the SQLite FTS maintenance path aligned
A compatible Unicode query is reduced to one normalized formSearch can miss memory stored in the original character formKeep original and normalized word forms in the SQLite search query
A server session is linked to a child from another projectProject reads can mix unrelated activityForeign keys, repository ownership checks, and SQLite triggers reject the link
Observation content changes without search-index maintenanceSearch returns stale or missing resultsSQLite triggers and the Postgres generated search value maintain the index
An embedding model is changed without a matching vector width or re-embed passSemantic search can fail or compare vectors from incompatible spacesBootstrap checks vector width and treats rows missing the active model as needing new vectors
pgvector is unavailable or the vector table is not bootstrappedMeaning-based search is unavailable even though observations remain readableSchema bootstrap falls back to keyword search and the embedding repository probes table availability
An embedding part is renamed or facts are numbered differentlyMigrated vectors no longer match their source text and hybrid results become incompleteKeep part names aligned with the embedding formatter and cap facts after blank values are removed
An API-key prefix lookup includes revoked or expired rows or scans an unbounded setAuthentication spends work on unusable keys or allows request cost to grow with table sizeFilter active rows and limit prefix candidates before raw-key verification
A configured Postgres SSL mode is ignored or an invalid mode silently changes transport securityThe database can use weaker transport than the operator expects or fail to connect as intendedApply explicit setting precedence and map verified modes to certificate checks
A standalone key command runs before the full server bootstrapKey creation or rotation fails because the prefix column or index is absentRun the idempotent API-key schema upgrade before key commands
Project memory deletion runs outside one transactionA project can retain part of its captured content after a purgepurgeProjectMemory deletes the content set inside a Postgres transaction

Key files

FileWhy to read it
src/…/schema.tsDefines the server-owned SQLite tables, links, indexes, and search safeguards.
plugin/…/SessionStore.jsShows the legacy worker session model, migrations, prompt ownership, local sync persistence boundary, observation-to-summary file rollup, and session-end delivery ledger.
src/…/agent-events.ts and src/…/server-sessions.tsShow the local server event and session records and their write-time metadata cleaning.
plugin/…/files.jsShows how legacy SQLite file-list values become lists for downstream search and sync use.
src/…/files.tsShows the source-level contract for decoding stored observation file lists.
src/…/SessionStore.tsShows the source-level session write path and file-list rollup used to build summaries.
src/…/SessionSearch.tsShows how legacy observations and summaries become searchable in SQLite.
src/…/schema.tsDefines the Postgres ownership graph, observation model, job records, optional keyword and vector search structures, API-key upgrade, usage, and rate limits.
src/…/config.tsShows the Postgres URL, pool, timeout, and SSL-mode boundary.
src/…/index.tsShows the repository set used as the Postgres storage interface.
src/…/utils.tsShows shared JSON, time, deterministic-key, and ownership rules.
src/…/agent-events.tsShows scoped event persistence, platform-aware identity, and event payload cleaning.
src/…/observations.tsShows the server observation and source boundary, scoped reads, lineage, keyword search, and hybrid search filters.
src/…/observation-embeddings.tsShows how observations become bounded, model-specific searchable parts and how missing vectors are found and stored.
src/…/generation-jobs.tsShows the durable processing lifecycle between events and observations.
src/…/server-sessions.tsShows scoped session identity, platform-aware linkage, generation state, and session-to-event reads.
src/…/auth.tsShows API-key prefix narrowing, hash migration, and audit persistence.
src/…/usage.tsShows append-only usage recording and quota reads.
src/…/data-deletion.tsShows the transaction boundary for project memory removal.
src/…/memory-items.tsShows the local unified memory repository, partial update semantics, Unicode-aware search, source lineage, project listing, and full-text search boundary.
src/…/serde.ts and src/…/utils.tsShow the backend-specific conversion rules for JSON, time, IDs, and ownership checks.
src/core/schemas/Defines the domain shapes used by the server-owned SQLite repositories.