Introducing a new way to review your PRs.

Read the announcement

usememos / memos

Memos is an open-source, self-hosted note-taking app that brings your everyday thoughts together in a personal timeline. Your data stays yours, and you choose what to keep private or share.

View on GitHub

25 documents · Updated Nov 20, 2026

docs / memo-payload-processing

Memo payload processing

Memo payload processing keeps the content-derived part of each memo aligned with its Markdown body. It extracts tags and content properties for filtering and presentation, refreshes old records after extraction rules change, and protects user metadata such as location during background work.

This page is a child of Memo lifecycle and collaboration. See Markdown content processing for the shared extraction meaning, Memo query filtering for payload fields used in queries, Space management and membership for memo audience boundaries, and Memo archive import and export for payload creation during archive import.

Derived memo data

Derived memo data is the stored summary of facts that can be read from Markdown without changing the Markdown itself. The payload keeps these facts beside user-controlled memo data so lists, filters, and memo views can use them without parsing the body on every read.

Payload content

The content-derived payload contains tags and properties. Properties record whether the body has links, task lists, code, incomplete tasks, a first-level title, or an inline image. Location is also part of the payload, but it is user metadata rather than a fact rebuilt from Markdown.

Payload areaRole
tagsSupports tag display and tag-based memo filters.
propertySummarizes content features used by views and filters.
locationPreserves memo location data across derived-data refreshes.

References: proto/…/memo.proto• L7-31 store/memo.go• L49-70 markdown/markdown.go• L25-38

Extraction boundary

The Markdown service parses one memo body and returns the facts needed by payload rebuilding. Its result also includes mentions and attachment references for other memo flows, but the payload runner stores only tags and content properties.

This boundary keeps one parser as the source of meaning while giving each caller only the data it owns. The same extraction rules therefore drive normal saves, imports, and later payload refreshes. See Markdown content processing for the parser contexts that decide which text becomes a tag or property.

References: markdown/markdown.go• L40-63 markdown/markdown.go• L425-509 core/…/runner.go• L154-168

Payload refresh lifecycle

Payload refresh has two paths. A normal memo write rebuilds the payload before persistence, while the background runner repairs stored payloads after the extraction contract changes.

Write-time rebuild

Memo creation and content updates compute tags and properties before the memo mutation is committed. Archive import uses the same rebuild step before it creates an imported memo, so imported content enters storage with the same derived meaning as newly authored content.

The write path can then validate managed image references against the memo's attachments and apply the memo change as one operation. Location is added after content-derived data is built, which keeps the two kinds of payload data separate.

References: server/…/memo_create_helpers.go• L25-92 server/…/memo_service.go• L370-397 server/…/user_service_memo_import.go• L343-377

Startup backfill

The payload runner is a one-off startup job. The server starts it without delaying HTTP service, and the runner checks the instance setting before doing work. A recorded payload version at or above the runner version means that the instance is already current.

When the version is older, the runner collects memo IDs in stable time order, reads the bodies in batches, extracts the new data, and refreshes each stored payload. It records the new version only after the full pass succeeds. An interrupted or failed pass leaves the old version in place so the next server start retries it.

The following diagram shows how a payload rule change moves from startup detection to guarded storage.

flowchart LR
    A[Server start] --> B[Read payload version]
    B -->|Current| C[Skip backfill]
    B -->|Older| D[Collect memo IDs]
    D --> E[Read memo batch]
    E --> F[Extract Markdown facts]
    F --> G[Guarded payload refresh]
    G --> H{All writes succeed?}
    H -->|Yes| I[Record new payload version]
    H -->|No or canceled| J[Leave version old]
    J --> K[Retry on next start]

References: core/…/runner.go• L17-77 core/…/runner.go• L79-152 server/server.go• L162-175 proto/…/instance_setting.proto• L47-55

Guarded refresh

Each background refresh carries the memo ID and the exact content that was read. The store updates tags and properties only when that content still matches the stored body. This prevents a slow rebuild from replacing data for a newer edit.

The update changes only the derived payload keys. It keeps location and the memo update time unchanged, and it works across the supported database drivers. A deleted memo or a memo edited during the pass is therefore left alone rather than treated as a successful replacement.

References: store/memo.go• L101-139 store/memo.go• L304-310 store/…/memo.go• L328-340 store/…/memo.go• L337-350 store/…/memo.go• L358-373

Payload consumers

Payload data is a shared read model. The memo service returns it as part of the composed memo resource, while the filter engine reads selected payload fields to choose and order memo results.

Memo reads and filters

Memo reads load the stored payload with the memo row. The API then exposes the memo and its related records as one resource, so the payload can support cards, detail views, statistics, and other downstream readers.

The query layer maps content properties and tags to filter names such as has_link, has_task_list, and tags. A stale payload can therefore make a memo disappear from a valid filtered view or show an incorrect content marker even when the Markdown body is current. See Memo query filtering for the filter contract.

References: store/…/memo.go• L118-275 server/…/memo_service.go• L145-236 filter/schema.go• L189-257 filter/render.go• L365-417

Payload versioning

The instance basic setting stores the highest payload rebuild version applied to all memos. The runner version changes when content extraction produces different derived output for existing bodies, not when a single memo changes.

The version is an instance-wide completion marker, not a per-memo freshness mark. The runner must complete the pass before it advances the marker; normal memo writes do not use the marker to decide whether their own payload is current.

References: core/…/runner.go• L17-23 core/…/runner.go• L42-75 proto/…/instance_setting.proto• L47-55 core/…/runner_test.go• L17-76

Change checklist

When these files or sections are changed, remember to consider these:

  • Keep the payload property schema, Markdown extraction result, write-time rebuild, and background backfill version aligned.
  • If a derived property changes, preserve its visibility in memo filters and any memo statistics or views that read the payload.
  • Keep guarded refresh writes limited to content-derived keys so location and update timestamps survive a rebuild.
  • If extraction rules change for old content, bump the payload version and keep the completion marker after the full pass only.
  • Preserve the exact-content check across all database drivers so a background pass cannot overwrite a concurrent edit.
  • Keep normal creation, content updates, and archive imports on the same rebuild path as the background runner.

Important failure modes

TriggerConsequenceGuard
The runner version advances before every memo is processedSome old payloads stay stale and future starts skip themRecord the version only after a complete successful pass
A memo changes after the runner reads itThe rebuild can overwrite derived data for newer contentMatch the stored content in the refresh write
A refresh replaces the whole payloadUser location data can be lost during a metadata repairUpdate only tags and property and preserve other payload keys
A Markdown body cannot be parsed during backfillOne bad memo can block repair of all other memosLog and skip the unparseable memo, then fail the pass only for failed writes
A payload property changes without a filter mapping updateFiltered lists can return incorrect results or reject the fieldKeep payload schema, filter schema, and dialect rendering aligned

Key files

FileWhy to read it
core/…/runner.goDefines write-time payload rebuilding, startup backfill, versioning, batching, and guarded refresh input.
markdown/markdown.goDefines the extraction result and the Markdown facts used for tags and properties.
proto/…/memo.protoDefines the stored payload shape and content-derived property fields.
store/memo.goDefines the refresh contract and the boundary between derived and user metadata.
store/…/memo.goShows how a guarded refresh preserves the rest of the payload in one database driver.
server/server.goShows when the background runner starts and how its failure affects retry behavior.
server/…/memo_create_helpers.goShows the shared payload preparation path for memo and comment creation.
server/…/memo_service.goShows payload rebuilding during content updates and the later composed memo response.
server/…/user_service_memo_import.goShows how archive imports rebuild payload data before persistence.
filter/schema.goDefines the payload-derived names that memo queries can use.
core/…/runner_test.goCaptures version retry, idempotence, location preservation, and cancellation behavior.
store/…/memo_payload_refresh_test.goCaptures the exact-content guard and the tags/property-only refresh contract.