Introducing a new way to review your PRs.

Read the announcement

thedotmack / claude-mem

Persistent Context Across Sessions for Every Agent – Captures everything your agent does during sessions, compresses it with AI, and injects relevant context back into future sessions. Works with Claude Code, OpenClaw, Codex, Gemini, Hermes, Copilot, OpenCode + More

View on GitHub

47 documents · Updated Nov 20, 2026

docs / sync-hub-worker-processing

Sync-hub worker processing

The sync hub is a Cloudflare Worker service that gives each user an ordered, durable stream of sync operations. The front Worker authenticates and routes requests, while one Durable Object per user stores the canonical stream, serves device cursors, and confirms that the external projection has caught up. An optional forwarding mode sends the whole request to another origin instead of touching the local hub.

The page Cloud sync platform gives the wider product view. The sync-hub control plane describes the control and monitoring concerns that sit beside this processing path.

Processing model

The sync hub separates edge work from durable work. In normal mode, the Worker handles request checks and all outbound calls, while the per-user Durable Object handles ordered state and does not perform outbound I/O.

This diagram shows the normal request path and the advisory notification path.

flowchart LR
    Client[Sync client] --> Worker[Front Worker]
    Worker --> Verify[Token verifier]
    Worker --> Hub[User SyncHub Durable Object]
    Hub --> Log[Canonical ordered log]
    Worker --> Projector[External content projector]
    Worker --> Notify[Discord notifications]
    Hub -. advisory hints .-> Client

The Worker selects the Durable Object by the authenticated canonical user id. This keeps all operations for one user in one ordered stream and prevents a claimed user id from being used without a matching token identity.

Worker entrypoint

The entrypoint in workers/…/index.ts is a stateless HTTP edge. It reads the user and device identity from request headers, checks the bearer token through the configured verifier, and caches only a short-lived positive verdict in KV.

Public processing has four paths:

PathProduct purpose
POST /v1/sync/opsAccept a batch of device operations and wait for projection.
GET /v1/sync/changesReturn ordered operations after a device cursor.
GET /v1/sync/statusReturn the user hub state for a client or probe.
GET /v1/sync/wsOpen an advisory hint channel.

The Worker also exposes secret-gated internal paths for projection repair, metadata reads, device naming, and per-user reset. These paths support the surrounding Pro service without making internal control state part of the public client protocol.

References: workers/…/index.ts• L1-32 workers/…/index.ts• L192-305 workers/…/index.ts• L855-993 workers/…/wrangler.jsonc• L1-24

Authentication boundary

Authentication binds a token to the presented user id before the Worker selects a Durable Object. A missing or invalid token fails with an authentication error. A valid token with a different or missing canonical user id fails with a user-binding error.

KV is an optimization, not the authority. A cache failure causes a fresh verifier check, and a verifier outage blocks the request. The device id and optional device name are transport identity used by the hub; the operation body remains unchanged.

References: workers/…/index.ts• L107-150 workers/…/index.ts• L192-305 workers/…/DEPLOY.md• L38-57

Forwarding boundary

When FORWARD_ORIGIN is set to a valid HTTP origin, the Worker forwards every request unchanged to that origin before it reads the kill switch, KV, or Durable Objects. This lets an alternate service own the hub path while keeping the public Worker address stable.

Forwarding preserves the incoming path and query string. It also applies to WebSocket upgrades. An empty or invalid origin uses the local Worker path instead. Scheduled watchdog and control-plane probe work is skipped in forwarding mode because the local Worker does not own the hub in that mode.

References: workers/…/index.ts• L130-161 workers/…/index.ts• L1061-1066 workers/…/index.ts• L1197-1214

Configuration and secrets

The Worker uses bindings and environment values to select its verifier, durable storage, projection service, cache, and operational mode. These values define which external systems the Worker may trust or call; they are not part of the client sync protocol.

Runtime bindings

The main runtime settings have these roles:

SettingRole
TOKEN_VERIFY_URLVerifies the bearer token and returns its canonical user id.
AUTH_CACHEStores short-lived positive token verdicts.
AUTH_CACHE_TTL_SECONDSBounds how long a positive verdict can remain cached.
SYNC_HUBNames the per-user Durable Object namespace.
FORWARD_ORIGINSelects the alternate-origin boundary instead of local processing.
INTERNAL_PROJECTOR_URLReceives canonical operation pages for external projection.
CMEM_INTERNAL_PROJECTOR_SECRETProtects projection and internal hub requests.

The Worker fails closed when token verification is unavailable. Missing projection settings do not discard accepted operations; they leave projection pending and return a retryable result.

References: workers/…/index.ts• L225-255 workers/…/index.ts• L286-323 workers/…/index.ts• L787-795

Internal credentials

The projector secret protects both the projection boundary and internal hub routes. The Worker compares the supplied bearer value with the configured secret before it allows repair, metadata, device-name, or reset operations. Public client tokens use a separate verifier path and do not grant these internal actions.

References: workers/…/index.ts• L558-648 workers/…/index.ts• L1001-1010

Durable user state

Each SyncHub Durable Object represents one user and owns the state needed to replay operations and measure progress. Its durable state includes the current epoch, the ordered operation head, the projection checkpoint, device cursors, and the latest revision for each entity.

The epoch identifies one history. A reset creates a new epoch and clears the old history so a device cannot silently combine cursors from two different histories. The ordered log remains available from cursor 0; physical compaction is disabled until a snapshot and reset bootstrap flow exists.

Canonical operation stream

The canonical operation is the shared record format used between clients, the Hub, and the projector. It supports observations, summaries, prompts, work-state records, tombstones, and a small set of guarded mutations. Each content record has a stable identity and an entity revision. A tombstone keeps the identity and revision while marking the entity deleted.

The Hub validates the complete request before allocating sequence numbers. It then applies entity revision rules: a newer revision becomes the entity head, an identical retry is acknowledged at its existing sequence, a different payload at the same revision is rejected, and an older revision is rejected. This makes retries safe while preserving one ordered stream.

References: workers/…/canonical-content.ts• L1-28 workers/…/canonical-content.ts• L35-43 workers/…/canonical-content.ts• L62-197 workers/…/canonical-content.ts• L223-362 workers/…/SyncHub.ts• L373-488

Device cursors

A device cursor records the greatest sequence that the device has acknowledged. Pulls return operations in sequence order and report whether more data remains. Push, pull, and WebSocket admission share a limit of 64 distinct devices per user; status reads do not admit unknown devices.

Device names are advisory display data. The first client-provided name is retained until the internal device-name path changes it. Device status includes last-seen time, cursor lag, and advisory connection state, but sync correctness depends on the ordered HTTP stream rather than the socket state.

References: workers/…/SyncHub.ts• L57-103 workers/…/SyncHub.ts• L285-311 workers/…/SyncHub.ts• L490-630 workers/…/METADATA-CONTRACT.md• L57-80

Advisory WebSocket

The WebSocket is a hint lane that reduces notification delay. It does not carry durable data and does not replace cursor reads. When a push commits, the Hub may send the other connected devices either the new operations or a compact advance hint when the frame would be too large.

Clients must use the HTTP changes path as the authority. The Worker can refuse new sockets when the kill switch is active, while existing HTTP push and pull behavior continues.

References: workers/…/index.ts• L904-929 workers/…/SyncHub.ts• L281-367 workers/…/kill-switch.ts• L1-34

Canonical content handling

Canonical content makes the same logical operation produce the same bytes and digest in every part of the system. The Hub accepts only exact wrappers around canonical JSON and checks the operation hash before it stores the record.

The content contract uses decimal strings for sequence-like values. It also uses UTF-8 byte limits, stable object-key order, fixed schema versions, stable document ids, and payload hashes. These rules protect replay, deduplication, and cross-service comparison without relying on JavaScript number precision.

Content kinds

The content body distinguishes normal content from mutations. Normal content uses a stable id derived from kind, origin device, and origin-local id. Mutations have explicit targets and fields so the projector can apply a controlled change without treating the mutation as a content document.

The Hub advertises supported normal content kinds through sync status. A client uses that list before it sends a kind so an older hub does not receive a kind it cannot accept.

KindRole
observationStore captured activity or memory content.
summaryStore a generated or completed session summary.
promptStore prompt-session content.
work_stateStore agent to-do list state.
mutationApply a named change such as a title, session link, or project remap.

Deleted normal content carries a tombstone instead of a payload. The Hub keeps that tombstone in the ordered stream so all devices and the projector see the same entity history.

References: workers/…/canonical-content.ts• L1-43 workers/…/canonical-content.ts• L136-160 workers/…/canonical-content.ts• L223-362 workers/…/index.ts• L541-555

Wire validation

The public push request uses protocol version 2 and carries a list of exact {body, operation_sha256} wrappers. The Worker bounds the request before the RPC call. The Hub then checks the wrapper shape, canonical JSON, body schema, payload schema, hashes, ids, revisions, and field limits.

Validation happens before sequence allocation. Therefore a bad operation in a batch does not leave a partial ordered append, and padded text that is valid remains byte-for-byte unchanged rather than being silently trimmed.

References: workers/…/index.ts• L307-363 workers/…/canonical-content.ts• L154-197 workers/…/SyncHub.ts• L373-395

Projection processing

Projection processing copies the Hub's canonical stream into the external content system. The Hub stores the authoritative projected_seq; the Worker uses a short per-user lease to make one projection drain safe when requests overlap or time out.

This diagram shows how a committed push moves from the Hub to the external projector and then becomes visible as checkpoint progress.

sequenceDiagram
    participant C as Client
    participant W as Worker
    participant H as User Hub
    participant P as Content projector
    C->>W: Push canonical operations
    W->>H: Append and get head_seq
    W->>H: Acquire projection lease
    loop Until projected_seq reaches head_seq
        W->>H: Read bounded page
        W->>H: Heartbeat lease
        W->>P: Send page with epoch and sequence range
        P-->>W: Confirm epoch and through_seq
        W->>H: Fenced checkpoint advance
    end
    W-->>C: Success or durable projection status

The Worker sends at most 100 operations or 4,000,000 encoded bytes in one request. The request includes the user id, epoch, start sequence, end sequence, and operation bytes. The projector response must confirm the same epoch and end sequence before the Hub advances its checkpoint.

Lease and checkpoint

The lease prevents two workers from applying the same user's pending range at the same time. The Worker aborts its upstream response-body wait after 45 seconds. The Hub lease lasts 90 seconds, which leaves time for a late upstream request to remain fenced after the Worker stops waiting.

The checkpoint advance checks the lease token, epoch, expected start sequence, and end sequence in one durable operation. A late predecessor therefore cannot advance the checkpoint after a successor takes the lease. The Worker releases early only after a confirmed checkpoint, a known-complete target, or a deterministic projector rejection.

References: workers/…/projection-protocol.ts• L1-45 workers/…/SyncHub.ts• L632-807 workers/…/index.ts• L542-805 workers/…/DEPLOY.md• L59-110

Push completion

The public push response represents two outcomes: the Hub may have durably accepted the operation while projection is still pending, or both the Hub and projector may have reached the same head. A retryable projection problem returns 503 with durable: true and retryable: true; repeating the same operation is safe and resumes from the unchanged checkpoint.

A deterministic projector rejection returns a non-retryable 409. It does not advance the Hub checkpoint. The repair endpoint uses the same drain process to bring a lagging user to its head without trusting a projector response alone.

References: workers/…/index.ts• L344-363 workers/…/index.ts• L814-852 workers/…/sync-hub.test.ts• L742-819

External notifications

Scheduled Worker tasks notify operators without placing outbound work inside Durable Objects. The hourly watchdog reads Cloudflare Durable Object metrics and can trip the kill switch for cost or behavior breaches. The five-minute control-plane probe checks that the database-backed token verifier still rejects a bogus token with a JSON error.

Both monitors use Discord as a notification channel. Discord failure is logged and does not block the durable guardrail or the next scheduled run. In forwarding mode, these local monitors do not run because the alternate origin owns request processing.

Kill-switch behavior

The kill switch is a KV flag owned by the Worker. When present, the Worker refuses new WebSocket upgrades and adds X-Sync-Mode: poll to every public HTTP sync response, including errors. Poll mode keeps reads and writes available while removing the socket path that can keep Durable Objects active.

The watchdog trips the switch for duration, rows-written, or rows-read breaches. It does not auto-trip for request-count breaches because poll mode still uses HTTP requests. A human clears the flag after the incident is resolved.

References: workers/…/kill-switch.ts• L37-117 workers/…/watchdog.ts• L1-53 workers/…/watchdog.ts• L422-519 workers/…/index.ts• L885-954

Control-plane probe

The probe runs outside the Pro application's failure domain and checks the DB-backed verifier rather than a static page. It treats 401 or 403 with a valid JSON body as healthy because that result shows that the app and its user lookup are working.

The first failure is confirmed by an immediate retry. A confirmed failure pages Discord and records state in KV. Continued failure is suppressed for 30 minutes between alerts, and recovery sends one green notification before clearing the state. A maintenance silence marker prevents the probe request and alert for its active period.

References: workers/…/control-plane-probe.ts• L1-58 workers/…/control-plane-probe.ts• L104-203 workers/…/control-plane-probe.ts• L332-477 workers/…/discord.ts• L1-43

Change checklist

When these files or sections are changed, remember to consider these:

  • Keep the canonical body schema, stable-id rules, payload hashes, supported content kinds, and client fixtures aligned across canonical-content.ts, the Hub validator, the status response, and any projector consumer.
  • Keep the projection page limits, serialized envelope, timeout, and lease duration aligned. A change in one can permit an oversized request or let a timed-out predecessor overlap a successor.
  • Preserve the rule that only the Hub checkpoint proves projection progress. Do not make a successful projector response alone advance client-visible health.
  • Keep the verifier's canonical user-id response contract aligned with Worker authentication and the deployment order in DEPLOY.md.
  • Treat the WebSocket as advisory when changing its frames or kill-switch handling. HTTP cursor replay must remain sufficient for recovery.
  • Keep forwarding mode as a complete boundary. It must bypass local Durable Objects, KV, authentication, and scheduled monitors together.
  • Keep the watchdog auto-trip metric set aligned with the poll-mode remedy. Request volume must not be treated as fixed by switching from WebSocket to polling.

Important failure modes

TriggerConsequenceGuard
Verifier returns 2xx without the canonical user idThe request cannot be safely bound to a user and is refused.Worker requires a non-empty canonical id and exact match before caching a verdict.
A batch contains an invalid or conflicting operationThe batch is refused before new sequence numbers are allocated.Full-batch canonical validation and entity revision checks.
Projection response times out or becomes ambiguousThe operation stays durable but the client receives a retryable failure.The lease remains held until expiry and the checkpoint stays unchanged.
A stale projection request reaches the Hub after a successor startsThe stale request cannot move the checkpoint.Lease token, epoch, and compare-and-set checks.
A user reaches the device admission limit with a new deviceThe new push, pull, or socket admission receives 409; known devices continue.Atomic per-user admission and read-only status handling for unknown devices.
Forwarding origin is configuredThe request bypasses local authentication, KV, and Durable Objects.Treat forwarding as a deployment boundary and verify the alternate origin owns the same public contract.
Watchdog detects a duration or SQLite row runawayNew sockets stop and clients move to poll mode.KV kill switch trips before the Discord alert.
Control-plane verifier returns a 2xx for a bogus tokenThe event is treated as a security incident rather than a normal outage.The external probe uses deliberately invalid credentials and a distinct alert.

Key files

FileWhy to read it
workers/…/index.tsWorker routes, authentication boundary, forwarding boundary, projection drain, repair endpoint, and scheduled entrypoint.
workers/…/SyncHub.tsPer-user durable state, ordered append, cursor reads, advisory hints, leases, and checkpoints.
workers/…/canonical-content.tsCanonical content, supported content kinds, and mutation contract shared by the Hub and projector.
workers/…/projection-protocol.tsProjection envelope and page-size contract.
workers/…/kill-switch.tsPoll-mode fallback and emergency cost guardrail.
workers/…/watchdog.tsScheduled metric checks and escalation behavior.
workers/…/control-plane-probe.tsExternal verifier health check and anti-flap alert state.
workers/…/METADATA-CONTRACT.mdPayload-free metadata contract used by the external control plane.
workers/…/DEPLOY.mdProduction ordering, projection contract, repair behavior, and operational limits.