Introducing a new way to review your PRs.

Read the announcement

nousresearch / hermes-agent

The agent that grows with you

View on GitHub

38 documents · Updated Nov 20, 2026

docs / state-storage-integrity

State storage integrity

Hermes state storage keeps session history and related runtime records in a SQLite database. This page explains how the state subsystem keeps that database consistent while several Hermes processes read and write it, and how it detects, contains, and repairs damage. For the wider persistence model, see State and persistence.

State store boundary

The state store is the durable boundary for session records, messages, usage data, routing data, and search indexes. The writable SessionDB owns schema setup and writes, while read-only attachments and maintenance tools use guarded paths so they do not mistake a partial file or a live journal for a complete database.

Schema lifecycle

Schema contract

The schema contract is defined by hermes_state_common.py and applied when a writable state store opens. New tables and columns are created from the declared schema, missing columns are reconciled, and data migrations bring older stores forward without requiring a destructive replacement.

The main schema version records data migrations. The search-index layout has its own state marker because a search rebuild can be large and is not required for the canonical transcript to remain usable.

Schema changes that alter a primary key or table shape use a single write transaction. This keeps a peer opener from seeing the table after it has been renamed but before the replacement table and its rows exist.

References: hermes_state_common.py• L283-306 hermes_state_common.py• L372-507 hermes_state_schema.py• L756-811 hermes_state_schema.py• L922-1005

Search index lifecycle

Full-text search is derived from the canonical messages rows. If its tables or triggers are stale, Hermes can detach the index and continue normal message writes and basic search while a later guarded rebuild restores it. This separates transcript durability from search availability.

The search layout can remain on a legacy form until an explicit storage optimization. The main schema continues to advance so unrelated data migrations do not wait for that optional work.

References: hermes_state_schema.py• L484-582 hermes_state_schema.py• L641-687 hermes_state_schema.py• L1052-1111 website/…/state-db-recovery.md• L12-33

Journal and durability policy

Journal modes

SQLite uses WAL mode when the file system and SQLite runtime can support it. WAL lets readers proceed while a writer commits. If WAL is unsafe or unavailable, Hermes uses the rollback journal mode instead; this preserves correctness at the cost of more reader and writer blocking.

Hermes never changes an existing WAL database to rollback-journal mode while another opener may use its WAL sidecars. A requested downgrade therefore requires a quiet store. Conversely, a WAL setup that cannot prove the current mode leaves the file untouched rather than unlinking a journal generation owned by another process.

Some cross-VM file systems do not provide coherent shared memory for WAL. New stores on those mounts use rollback-journal mode, and an existing WAL store is reported for operator action rather than being changed live.

References: hermes_state_wal.py• L157-184 hermes_state_wal.py• L254-318 hermes_state_wal.py• L397-425

Commit durability

Durability settings apply to every connection type, including read-only attachments. WAL size is bounded so a large transaction does not leave an indefinitely large sidecar. On macOS, checkpoint barriers and a minimum synchronous level protect page ordering during shutdown and checkpoint operations.

Normal close and periodic checkpoint work use passive checkpoints. They do not force a full WAL reset while another live writer may still use the same generation. A full rewrite such as VACUUM is reserved for a quiet store and checks the database generation again before committing its result.

References: hermes_state_wal.py• L86-128 hermes_state_wal.py• L579-647 hermes_state.py• L1469-1488 hermes_state.py• L1507-1570 hermes_state_maintenance.py• L392-419

Concurrent access

Writer ownership

The state subsystem permits many readers but serializes writes through SQLite and the store's write path. A process-wide registry shares one writable SessionDB per resolved file, so independent callers in one process do not create competing close-time checkpoints or token writers.

Cross-process operations that change schema or rewrite the file take a separate repair or maintenance lock. If the lock cannot be opened or acquired, the operation skips its structural work instead of running unlocked surgery.

The following diagram shows the normal choice between a WAL reader and the safe fallback path.

flowchart TD
    Request[Read request] --> WAL{WAL active and read permit available?}
    WAL -->|Yes| RO[Tracked read-only connection]
    WAL -->|No| Writer[Writer connection under store lock]
    RO --> Result[Committed result]
    Writer --> Result
    Write[Write request] --> SQLite[SQLite serialized write]
    SQLite --> Result

References: hermes_state_registry.py• L1-29 hermes_state_registry.py• L195-309 hermes_state_repair.py• L163-220 hermes_state.py• L994-1110

Reader capacity

WAL readers use tracked, read-only connections so they do not queue behind ordinary writer flushes. The read pool has limits per database file and per process, and it also checks file-descriptor headroom. When a limit or open failure is reached, the read uses the locked writer connection instead of opening more descriptors or stalling the process.

A short retry covers a transient WAL checkpoint or reset that briefly prevents a read-only connection from opening. Persistent I/O errors are surfaced as storage failures rather than hidden by repeated retries.

References: hermes_state_readpool.py• L21-60 hermes_state_readpool.py• L140-235 hermes_state.py• L721-763 hermes_state.py• L850-958

Maintenance admission

Repair, journal-mode changes, VACUUM, and other structural work must prove that no other process holds the database or any WAL sidecar. Holder scans include deleted sidecar generations and fail closed when the operating system cannot provide a reliable answer. In-process registry generations are checked separately because a process scan excludes the current process.

This rule protects both writers and rollback-journal readers. A SQLite lock probe alone is not enough because it cannot see every reader and cannot run reliably on a malformed file.

References: hermes_state_holders.py• L427-566 hermes_state_holders.py• L569-655 hermes_state_maintenance.py• L455-528

File identity and portability

SQLite file generations

The main database and its journal sidecars form one SQLite generation. Each writable handle records the main-file identity, application identity, and sidecar identities. If the path is replaced or its WAL generation disappears, the handle stops writes and does not reconnect through stale sidecar assumptions.

When a WAL generation is lost, Hermes captures the retired generation for later inspection. It skips the close-time checkpoint because applying frames from the old generation to a new file could write pages to the wrong database.

References: hermes_state.py• L618-630 hermes_state.py• L1206-1285 hermes_state.py• L1287-1423

Portable database bundles

A portable or forensic copy must preserve the main file together with every present -wal, -shm, or rollback -journal sidecar. Raw file reads are blocked while a tracked connection is live because closing a raw descriptor can cancel process-wide advisory locks.

Repair uses SQLite's online backup path for a complete snapshot and promotes a verified result through SQLite rather than replacing the live inode. This keeps existing handles from silently following a different file and makes the copy represent committed WAL frames as one image.

References: hermes_cli/sqlite_safe_read.py• L1-7 hermes_cli/sqlite_safe_read.py• L147-193 hermes_cli/sqlite_safe_read.py• L224-257 hermes_state_repair.py• L672-693

Health and repair

Health state

The state health model distinguishes structural database damage from derived search damage. Structural corruption sets a process-wide latch for the database path, so readiness, status, and session-list surfaces report the same degraded state instead of showing an empty store as healthy. The latch clears only when the process restarts on a repaired or restored file.

The health probe is read-only. It checks that the schema opens, canonical tables can be read, integrity checks pass, and search can both read and accept a rolled-back write through its triggers. A missing optional tokenizer is treated as a capability gap, not as database corruption.

References: hermes_state_health.py• L1-19 hermes_state_health.py• L49-102 hermes_state_repair.py• L756-834 gateway/readiness.py• L1-108

Repair workflow

Repair is an offline recovery operation. It keeps all writers and readers stopped, takes a forensic backup unless explicitly disabled, stages a complete copy, applies the least destructive successful repair, verifies the staged file, and promotes it only after the checks pass.

The workflow is bounded and fail closed:

  1. A cross-process repair lock serializes repairers.
  2. Holder checks and an exclusive database guard prevent concurrent use.
  3. A disk-space check protects the original and the scratch copy.
  4. Repair strategies try search rebuild, B-tree index rebuild, schema de-duplication, and finally removal of derived search schema.
  5. A clean probe is required before promotion.
  6. Failed attempts on the same unchanged file stop after a small persistent budget and require backup restore or page-level salvage.

The flow below shows why canonical rows remain protected when a repair strategy fails.

flowchart TD
    Start[Unhealthy state.db] --> Stop[Stop all Hermes users]
    Stop --> Lock[Acquire repair lock and holder proof]
    Lock --> Backup[Preserve main file and sidecars]
    Backup --> Guard[Hold exclusive live-file guard]
    Guard --> Snapshot[Create SQLite scratch snapshot]
    Snapshot --> Strategy[Try repair on scratch copy]
    Strategy --> Verify{Health probe passes?}
    Verify -->|No| Keep[Keep original and backup]
    Verify -->|Yes| Promote[Promote through SQLite backup]
    Promote --> Restore[Restore journal policy]
    Restore --> Restart[Restart on verified file]

References: hermes_state_repair.py• L857-924 hermes_state_repair.py• L974-1049 hermes_state_repair.py• L1070-1160 website/…/state-db-recovery.md• L76-111

Space maintenance

Pruning removes eligible ended sessions and keeps their related file cleanup outside the database transaction. Automatic maintenance runs at a bounded interval and skips work when another process or handle holds the store. VACUUM runs only after enough pages are reclaimable and only when its exclusive rewrite will not race a live holder.

References: hermes_state_maintenance.py• L322-353 hermes_state_maintenance.py• L374-419 hermes_state_maintenance.py• L421-528

Change checklist

When these files or sections are changed, remember to consider these:

  • Keep SCHEMA_SQL, column reconciliation, table-shape repairs, and schema-version data migrations aligned. A new column is not enough when a key or constraint also changes.
  • Preserve the separation between canonical transcript rows and derived FTS objects. FTS recovery must not delete or rewrite canonical messages.
  • Any journal-mode or WAL-sidecar change must preserve the no-live-downgrade rule and the holder checks on every maintenance entry point.
  • Any file replacement, restore, or repair promotion must update generation and sidecar identity handling so old connections cannot write into the new image.
  • New raw file inspection must use tracked offline access, or it can cancel SQLite locks held by another connection in the same process.
  • New read connections must use the shared per-path budget and degrade to the writer path when capacity is exhausted.

Important failure modes

TriggerConsequenceGuard
WAL is enabled on a file system without coherent shared memorySQLite can corrupt the shared WAL index or the databaseDetect unsafe mounts and use rollback-journal mode; report existing WAL stores for operator action
A process deletes or replaces a live WAL sidecarThe open handle sees a split generation and could apply old frames to a new fileRecord sidecar identity, halt writes, capture the retired generation, and skip close-time checkpointing
Repair starts while a gateway or reader still holds the storeSnapshot or promotion can overwrite active state and damage B-treesScan main and sidecar holders, add an exclusive guard, and refuse when quiescence is not proven
A read burst exhausts descriptors or read permitsThe process can hit EMFILE and lose unrelated sockets or pipesBound readers per file and process, reserve descriptor headroom, and fall back to the writer connection
FTS shadow tables reject a write while canonical tables remain soundNew messages could fail even though transcript storage is usableMark FTS stale, detach its triggers, keep canonical writes available, and rebuild later
Structural corruption is detected during a writeFurther writes or close-time checkpoints could amplify the damageQuarantine the handle, latch storage health as corrupt, and require recovery or restore

Key files

FileWhy to read it
hermes_state.pyMain state-store lifecycle, writer and reader selection, generation checks, close behavior, and corruption quarantine
hermes_state_common.pyCanonical schema declaration, schema version, and independent FTS layout contract
hermes_state_schema.pySchema reconciliation, data migration flow, table-shape healing, and FTS recovery ownership
hermes_state_wal.pyJournal-mode policy, WAL fallback, durability settings, and cross-VM file-system protection
hermes_state_registry.pyShared writable handles, reference counts, generation retirement, and teardown barriers
hermes_state_readpool.pyPer-file and process-wide reader limits plus descriptor headroom protection
hermes_state_holders.pyCross-process and in-process holder proof used before structural maintenance
hermes_state_repair.pyBackups, scratch snapshots, health probes, repair strategies, promotion, and repair bounds
hermes_cli/sqlite_safe_read.pyTracking rules that make raw file inspection safe around live SQLite connections
hermes_state_health.pyProcess-wide structural-corruption latch shared by writers and health surfaces
gateway/readiness.pyRead-only readiness view of state database and session-store health
website/…/state-db-recovery.mdOperator recovery sequence and post-repair verification contract