Introducing a new way to review your PRs.

Read the announcement

thedotmack / claude-mem

Persistent Context Across Sessions for Every Agent – Captures everything your agent does during sessions, compresses it with AI, and injects relevant context back into future sessions. Works with Claude Code, OpenClaw, Codex, Gemini, Hermes, Copilot, OpenCode + More

View on GitHub

47 documents · Updated Nov 20, 2026

docs / sync-hub-control-plane

Sync-hub control-plane safeguards

The sync-hub control plane protects availability, user isolation, and cloud cost around the synchronization service. It checks the external verifier, watches Durable Object use, and can move clients to a safe polling mode without stopping data sync.

Safeguard model

The control plane runs in the stateless Worker beside the public sync routes. Scheduled checks observe systems outside the per-user hub, while a shared control state can change how clients use the real-time lane.

flowchart LR
    C[Scheduled Worker] --> P[Control-plane probe]
    C --> W[Hourly watchdog]
    P --> V[DB-backed token verifier]
    W --> A[Cloudflare analytics]
    P --> D[Discord notification]
    W --> D
    W --> K[Kill-switch state]
    K --> R[Public sync routes]
    R --> M[Poll-mode signal]
    M --> F[Client HTTP polling]

The controls are designed to reduce harm instead of making a temporary fault larger. A monitor can alert or select polling, but it does not own the durable per-user change history.

Control-plane health probe

The control-plane probe checks whether the application and its database-backed token lookup are alive. Every five minutes it sends a deliberately invalid token and an all-zero user id to workers/…/control-plane-probe.ts; a JSON 401 or 403 is healthy because the verifier had to process and reject the request. control-plane-probe.ts index.ts

The probe treats timeouts, network errors, server errors, non-JSON rejection pages, and unexpected statuses as failures. A 2xx response for the bogus token is a security alarm because it suggests an authentication bypass rather than a normal outage.

References: workers/…/control-plane-probe.ts• L1-18 workers/…/control-plane-probe.ts• L104-203 workers/…/index.ts• L957-989

Probe alert state

The probe uses the control:uptime-probe record in AUTH_CACHE to avoid noisy alerts. It confirms the first failure with one immediate retry, alerts again at most every 30 minutes while the failure continues, and sends one recovery notice when the healthy response returns. control-plane-probe.ts control-plane-probe.ts

Healthy steady state stays quiet: it writes no state and sends no message. Maintenance can place a future silenced_until value at the same key. During that window the probe does not even call the verifier. Discord is only a notification path, so a failed post does not block state recording.

References: workers/…/control-plane-probe.ts• L20-37 workers/…/control-plane-probe.ts• L332-477 workers/…/DEPLOY.md• L425-477

Metric watchdog

The hourly watchdog compares the previous hour of sync-hub activity with limits for requests, Durable Object duration, database rows read, and database rows written. It reads Cloudflare Analytics from the Worker schedule and sends one alert for a threshold breach. watchdog.ts watchdog.ts

The default limits separate early warning from structural action:

MetricAlertKillPoll-mode action
Requests per hour60,000600,000No automatic trip; polling still creates HTTP requests
Duration GB-s per hour50450Yes; removes the socket lane that can keep hubs active
Rows written per hour150,0001,500,000Yes
Rows read per hour5,000,00050,000,000Yes

The watchdog does not invent an incident when its analytics query fails. It logs a skipped or failed monitoring state instead. For an automatic trip, it writes the kill-switch state before it tries to notify Discord.

References: workers/…/watchdog.ts• L61-126 workers/…/watchdog.ts• L229-260 workers/…/watchdog.ts• L422-519 workers/…/DEPLOY.md• L242-271

Kill-switch fallback

The kill switch is the presence of control:kill-switch in AUTH_CACHE. When present, the Worker refuses new WebSocket upgrades with a 503 response and marks every HTTP sync response with X-Sync-Mode: poll, including error responses. Clients can then close or avoid sockets while continuing to push and pull data through HTTP. kill-switch.ts index.ts

The switch changes latency, not sync correctness. It does not clear itself. An operator clears the flag after the cause is understood, so the system does not restore the socket lane while the triggering condition may still exist.

The read uses a short per-isolate cache to limit KV traffic. A KV read error fails open because this flag guards cost and socket use; treating an infrastructure error as a global outage would stop sync for every user. KV propagation and client polling determine how fast all clients observe a trip or clear.

References: workers/…/kill-switch.ts• L1-35 workers/…/kill-switch.ts• L65-117 workers/…/index.ts• L885-954 workers/…/DEPLOY.md• L287-313

Authentication boundary

The public Worker fails closed when it cannot verify a request. It accepts a positive verifier result only when the result contains the canonical user id and that id matches X-User-Id; an invalid token returns 401, a user mismatch returns 403, and an unreachable verifier returns 503. index.ts

Only a bound positive result enters the short-lived AUTH_CACHE verdict cache. This limits repeated verifier calls without allowing a valid token for one user to become access to another user. The control keys use a separate control: prefix in the same namespace, so an emergency switch does not depend on a second binding.

References: workers/…/index.ts• L192-305 workers/…/wrangler.jsonc• L26-35 workers/…/kill-switch.ts• L11-21

Secret configuration

Deployment keeps credentials outside committed Worker variables. The internal projector credential, Cloudflare Analytics token, and Discord webhook are secret bindings set with wrangler secret put; the typed declarations mark them optional so an unconfigured watchdog skips safely instead of crashing. secrets.d.ts

The public verifier URL, projector URL, cache settings, account selectors, and threshold overrides are deployment variables. The verifier and projector routes must be deployed before the hub is activated, and the verifier must be canaried with the exact token-to-user binding. A Discord URL is never hardcoded or committed.

References: workers/…/secrets.d.ts• L1-26 workers/…/wrangler.jsonc• L50-89 workers/…/DEPLOY.md• L38-57 workers/…/DEPLOY.md• L59-75

Deployment safeguards

Production deployment activates both scheduled controls through the Worker configuration. The five-minute probe and the hourly watchdog use separate cron values and dispatch paths, while the Durable Object stays free of outbound monitoring work. wrangler.jsonc index.ts

Deployment order

The deployment order protects the two cross-service contracts that the hub needs before it can safely serve users:

  1. Deploy and canary the database-backed token verifier.
  2. Deploy the internal projection route and set the shared projector secret.
  3. Configure the Worker variables and secret bindings.
  4. Deploy the Worker and confirm both cron triggers in the Cloudflare dashboard.
  5. Run the canary and rehearse the alert chain with lowered thresholds before normal operation.

The Worker must not become active while token verification is absent or while its response does not bind a token to the canonical user. This order prevents a healthy-looking edge from granting unsafe access or sending accepted sync data to an unready projector. DEPLOY.md DEPLOY.md DEPLOY.md

References: workers/…/wrangler.jsonc• L36-49 workers/…/DEPLOY.md• L38-75 workers/…/DEPLOY.md• L273-283 workers/…/DEPLOY.md• L481-493

Canary verification

The long-running canary uses one synthetic user and two devices to check that a small write reaches the other device. It records convergence, latency, and the current sync mode, so it also shows when the kill switch has moved the service to polling. canary.ts

The normal canary provides a known low workload for the watchdog. Its flood mode is reserved for an alert-chain rehearsal after thresholds are lowered; running that mode at production defaults would create real request load. canary.ts DEPLOY.md

References: workers/…/canary.ts• L1-48 workers/…/canary.ts• L167-190 workers/…/DEPLOY.md• L317-362 workers/…/DEPLOY.md• L366-401

Change checklist

When these files or sections are changed, remember to consider these:

  • Changes to the verifier response must preserve canonical user-id binding before a positive verdict is cached.
  • Changes to AUTH_CACHE bindings or key prefixes must preserve access to both short-lived auth verdicts and non-expiring control records.
  • Changes to the kill-switch header or WebSocket response must preserve the client mode contract on both successful and failed HTTP responses.
  • Changes to watchdog metric filters or thresholds must keep each automatic trip tied to a failure that polling can reduce.
  • Changes to cron values must preserve distinct dispatch for the five-minute probe and hourly watchdog.
  • Changes to secret names or variable names must update the Worker bindings and deployment order together.
  • Changes to canary workload or flood behavior must keep production rehearsal separate from normal low-cost monitoring.

Important failure modes

TriggerConsequenceGuard
The verifier returns 2xx without the canonical user id or with a different user idThe request is refused instead of gaining cross-user accessAuthentication requires an exact token-to-user match before it caches a verdict
The verifier becomes unreachablePublic sync requests fail closed with 503 and the uptime probe alerts after confirmationNo authentication bypass exists; the probe runs outside the verifier's hosting failure domain
A duration, row-read, or row-write kill limit is reachedClients move from sockets to polling while sync remains availableThe watchdog trips the switch before it sends the notification
The request-count kill limit is reachedA severe alert is sent but the switch is not trippedPolling still uses HTTP requests, so a human must choose the remedy
A watchdog analytics query failsNo alert or automatic trip is created from incomplete evidenceThe watchdog records query_failed and leaves sync state unchanged
A Discord webhook failsOperators may miss a message, but the control state still changesDiscord is best-effort and follows the kill-switch write
KV cannot be read for the kill switchThe edge treats the switch as clear until KV works againThe guardrail fails open to avoid taking down sync for all users; the failure is logged
A maintenance silence remains active after the windowThe probe does not check the verifier or send alertsOperators delete the silence key or let an expired marker be cleared by a healthy run

Key files

FileWhy to read it
workers/…/index.tsShows how authentication, kill-switch signals, scheduled jobs, and route behavior meet at the Worker boundary.
workers/…/control-plane-probe.tsDefines the independent verifier health check, security alarm, maintenance silence, and anti-flap state.
workers/…/watchdog.tsDefines the hourly cost and behavior monitor and its escalation choices.
workers/…/kill-switch.tsDefines the operator-controlled fallback from the socket lane to polling.
workers/…/secrets.d.tsShows which deployment credentials are secrets and how missing monitoring credentials are handled.
workers/…/wrangler.jsoncShows the KV binding, external URLs, cron triggers, and threshold configuration at deployment level.
workers/…/canary.tsShows the production health signal for convergence and the observed sync mode.
workers/…/DEPLOY.mdGives the deployment order, emergency switch operations, probe silence, and alert-chain rehearsal.