Sync-hub control-plane safeguards
The sync-hub control plane protects availability, user isolation, and cloud cost around the synchronization service. It checks the external verifier, watches Durable Object use, and can move clients to a safe polling mode without stopping data sync.
Safeguard model
The control plane runs in the stateless Worker beside the public sync routes. Scheduled checks observe systems outside the per-user hub, while a shared control state can change how clients use the real-time lane.
flowchart LR
C[Scheduled Worker] --> P[Control-plane probe]
C --> W[Hourly watchdog]
P --> V[DB-backed token verifier]
W --> A[Cloudflare analytics]
P --> D[Discord notification]
W --> D
W --> K[Kill-switch state]
K --> R[Public sync routes]
R --> M[Poll-mode signal]
M --> F[Client HTTP polling]
The controls are designed to reduce harm instead of making a temporary fault larger. A monitor can alert or select polling, but it does not own the durable per-user change history.
Control-plane health probe
The control-plane probe checks whether the application and its database-backed token lookup are alive. Every five minutes it sends a deliberately invalid token and an all-zero user id to workers/…/control-plane-probe.ts; a JSON 401 or 403 is healthy because the verifier had to process and reject the request. control-plane-probe.ts index.ts
The probe treats timeouts, network errors, server errors, non-JSON rejection pages, and unexpected statuses as failures. A 2xx response for the bogus token is a security alarm because it suggests an authentication bypass rather than a normal outage.
References: workers/…/control-plane-probe.ts• L1-18 workers/…/control-plane-probe.ts• L104-203 workers/…/index.ts• L957-989
Probe alert state
The probe uses the control:uptime-probe record in AUTH_CACHE to avoid noisy alerts. It confirms the first failure with one immediate retry, alerts again at most every 30 minutes while the failure continues, and sends one recovery notice when the healthy response returns. control-plane-probe.ts control-plane-probe.ts
Healthy steady state stays quiet: it writes no state and sends no message. Maintenance can place a future silenced_until value at the same key. During that window the probe does not even call the verifier. Discord is only a notification path, so a failed post does not block state recording.
References: workers/…/control-plane-probe.ts• L20-37 workers/…/control-plane-probe.ts• L332-477 workers/…/DEPLOY.md• L425-477
Metric watchdog
The hourly watchdog compares the previous hour of sync-hub activity with limits for requests, Durable Object duration, database rows read, and database rows written. It reads Cloudflare Analytics from the Worker schedule and sends one alert for a threshold breach. watchdog.ts watchdog.ts
The default limits separate early warning from structural action:
| Metric | Alert | Kill | Poll-mode action |
|---|---|---|---|
| Requests per hour | 60,000 | 600,000 | No automatic trip; polling still creates HTTP requests |
| Duration GB-s per hour | 50 | 450 | Yes; removes the socket lane that can keep hubs active |
| Rows written per hour | 150,000 | 1,500,000 | Yes |
| Rows read per hour | 5,000,000 | 50,000,000 | Yes |
The watchdog does not invent an incident when its analytics query fails. It logs a skipped or failed monitoring state instead. For an automatic trip, it writes the kill-switch state before it tries to notify Discord.
References: workers/…/watchdog.ts• L61-126 workers/…/watchdog.ts• L229-260 workers/…/watchdog.ts• L422-519 workers/…/DEPLOY.md• L242-271
Kill-switch fallback
The kill switch is the presence of control:kill-switch in AUTH_CACHE. When present, the Worker refuses new WebSocket upgrades with a 503 response and marks every HTTP sync response with X-Sync-Mode: poll, including error responses. Clients can then close or avoid sockets while continuing to push and pull data through HTTP. kill-switch.ts index.ts
The switch changes latency, not sync correctness. It does not clear itself. An operator clears the flag after the cause is understood, so the system does not restore the socket lane while the triggering condition may still exist.
The read uses a short per-isolate cache to limit KV traffic. A KV read error fails open because this flag guards cost and socket use; treating an infrastructure error as a global outage would stop sync for every user. KV propagation and client polling determine how fast all clients observe a trip or clear.
References: workers/…/kill-switch.ts• L1-35 workers/…/kill-switch.ts• L65-117 workers/…/index.ts• L885-954 workers/…/DEPLOY.md• L287-313
Authentication boundary
The public Worker fails closed when it cannot verify a request. It accepts a positive verifier result only when the result contains the canonical user id and that id matches X-User-Id; an invalid token returns 401, a user mismatch returns 403, and an unreachable verifier returns 503. index.ts
Only a bound positive result enters the short-lived AUTH_CACHE verdict cache. This limits repeated verifier calls without allowing a valid token for one user to become access to another user. The control keys use a separate control: prefix in the same namespace, so an emergency switch does not depend on a second binding.
References: workers/…/index.ts• L192-305 workers/…/wrangler.jsonc• L26-35 workers/…/kill-switch.ts• L11-21
Secret configuration
Deployment keeps credentials outside committed Worker variables. The internal projector credential, Cloudflare Analytics token, and Discord webhook are secret bindings set with wrangler secret put; the typed declarations mark them optional so an unconfigured watchdog skips safely instead of crashing. secrets.d.ts
The public verifier URL, projector URL, cache settings, account selectors, and threshold overrides are deployment variables. The verifier and projector routes must be deployed before the hub is activated, and the verifier must be canaried with the exact token-to-user binding. A Discord URL is never hardcoded or committed.
References: workers/…/secrets.d.ts• L1-26 workers/…/wrangler.jsonc• L50-89 workers/…/DEPLOY.md• L38-57 workers/…/DEPLOY.md• L59-75
Deployment safeguards
Production deployment activates both scheduled controls through the Worker configuration. The five-minute probe and the hourly watchdog use separate cron values and dispatch paths, while the Durable Object stays free of outbound monitoring work. wrangler.jsonc index.ts
Deployment order
The deployment order protects the two cross-service contracts that the hub needs before it can safely serve users:
- Deploy and canary the database-backed token verifier.
- Deploy the internal projection route and set the shared projector secret.
- Configure the Worker variables and secret bindings.
- Deploy the Worker and confirm both cron triggers in the Cloudflare dashboard.
- Run the canary and rehearse the alert chain with lowered thresholds before normal operation.
The Worker must not become active while token verification is absent or while its response does not bind a token to the canonical user. This order prevents a healthy-looking edge from granting unsafe access or sending accepted sync data to an unready projector. DEPLOY.md DEPLOY.md DEPLOY.md
References: workers/…/wrangler.jsonc• L36-49 workers/…/DEPLOY.md• L38-75 workers/…/DEPLOY.md• L273-283 workers/…/DEPLOY.md• L481-493
Canary verification
The long-running canary uses one synthetic user and two devices to check that a small write reaches the other device. It records convergence, latency, and the current sync mode, so it also shows when the kill switch has moved the service to polling. canary.ts
The normal canary provides a known low workload for the watchdog. Its flood mode is reserved for an alert-chain rehearsal after thresholds are lowered; running that mode at production defaults would create real request load. canary.ts DEPLOY.md
References: workers/…/canary.ts• L1-48 workers/…/canary.ts• L167-190 workers/…/DEPLOY.md• L317-362 workers/…/DEPLOY.md• L366-401
Related pages
- Cloud synchronization platform gives the system-wide view of the sync hub and its role in cloud synchronization.
- Sync-hub worker processing explains public request processing, durable user state, projection, and notification paths.
Change checklist
When these files or sections are changed, remember to consider these:
- Changes to the verifier response must preserve canonical user-id binding before a positive verdict is cached.
- Changes to
AUTH_CACHEbindings or key prefixes must preserve access to both short-lived auth verdicts and non-expiring control records. - Changes to the kill-switch header or WebSocket response must preserve the client mode contract on both successful and failed HTTP responses.
- Changes to watchdog metric filters or thresholds must keep each automatic trip tied to a failure that polling can reduce.
- Changes to cron values must preserve distinct dispatch for the five-minute probe and hourly watchdog.
- Changes to secret names or variable names must update the Worker bindings and deployment order together.
- Changes to canary workload or flood behavior must keep production rehearsal separate from normal low-cost monitoring.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
The verifier returns 2xx without the canonical user id or with a different user id | The request is refused instead of gaining cross-user access | Authentication requires an exact token-to-user match before it caches a verdict |
| The verifier becomes unreachable | Public sync requests fail closed with 503 and the uptime probe alerts after confirmation | No authentication bypass exists; the probe runs outside the verifier's hosting failure domain |
| A duration, row-read, or row-write kill limit is reached | Clients move from sockets to polling while sync remains available | The watchdog trips the switch before it sends the notification |
| The request-count kill limit is reached | A severe alert is sent but the switch is not tripped | Polling still uses HTTP requests, so a human must choose the remedy |
| A watchdog analytics query fails | No alert or automatic trip is created from incomplete evidence | The watchdog records query_failed and leaves sync state unchanged |
| A Discord webhook fails | Operators may miss a message, but the control state still changes | Discord is best-effort and follows the kill-switch write |
| KV cannot be read for the kill switch | The edge treats the switch as clear until KV works again | The guardrail fails open to avoid taking down sync for all users; the failure is logged |
| A maintenance silence remains active after the window | The probe does not check the verifier or send alerts | Operators delete the silence key or let an expired marker be cleared by a healthy run |
Key files
| File | Why to read it |
|---|---|
| workers/…/index.ts | Shows how authentication, kill-switch signals, scheduled jobs, and route behavior meet at the Worker boundary. |
| workers/…/control-plane-probe.ts | Defines the independent verifier health check, security alarm, maintenance silence, and anti-flap state. |
| workers/…/watchdog.ts | Defines the hourly cost and behavior monitor and its escalation choices. |
| workers/…/kill-switch.ts | Defines the operator-controlled fallback from the socket lane to polling. |
| workers/…/secrets.d.ts | Shows which deployment credentials are secrets and how missing monitoring credentials are handled. |
| workers/…/wrangler.jsonc | Shows the KV binding, external URLs, cron triggers, and threshold configuration at deployment level. |
| workers/…/canary.ts | Shows the production health signal for convergence and the observed sync mode. |
| workers/…/DEPLOY.md | Gives the deployment order, emergency switch operations, probe silence, and alert-chain rehearsal. |