Cloud synchronization platform
The cloud synchronization platform combines the sync-hub edge service with a standalone sync API service to move canonical content between local installations and the cloud. It authenticates each request at the active cloud boundary, keeps an ordered per-user change history, projects accepted changes to the cloud search layer, and applies controls that reduce cost and limit the effect of service failures.
Platform boundary
The platform separates request handling from durable user state. A stateless edge worker checks identity and chooses the route, while one durable hub represents each user and remains the authority for ordered changes, device cursors, and projection progress. The standalone sync API service provides a separate cloud-side boundary for synchronized clients and projection consumers; its request and storage behavior is described in Sync API service.
Request boundary
The public boundary accepts pushes, cursor-based change reads, status reads, and an advisory WebSocket lane. A token is valid only when the external verifier confirms that it belongs to the claimed user id. This binding prevents a valid token from reading or changing another user's synchronized content. src/index.ts
The edge worker also owns outbound calls. This keeps authentication, cloud projection, and notification work outside the per-user hub. Internal metadata, device rename, reset, and projection-repair routes use a separate shared credential rather than public user authentication. src/index.ts
The standalone sync API can also operate as a forwarding boundary during service transition. It keeps health checks local and sends supported HTTP sync requests to the upstream sync function, which remains the authentication and synchronization authority in this mode. It retires the legacy WebSocket and internal routes, so clients use the HTTP sync path for reliable progress. forward.ts forward.ts
References: workers/…/index.ts• L192-305 workers/…/index.ts• L413-517 services/…/forward.ts• L1-13 services/…/forward.ts• L119-139
Per-user synchronization state
Each user has an ordered hub with a current epoch, a log head, and a projection checkpoint. Devices keep independent cursors into that history. The hub admits a bounded number of devices and reports device presence as operational information, not as a source of sync truth. SyncHub.ts METADATA-CONTRACT.md
The epoch identifies one history. A reset creates a new epoch and clears the user's hub, so an old cursor cannot be mistaken for a cursor in the new history. The ordered log stays replayable because launch does not remove old operations. SyncHub.ts DEPLOY.md
References: workers/…/SyncHub.ts• L74-129 workers/…/SyncHub.ts• L247-277 workers/…/DEPLOY.md• L112-117
Synchronization lifecycle
The lifecycle turns local content into a shared, ordered stream and lets each device catch up from its own cursor. The WebSocket lane can reduce delay where it is available, but HTTP push and pull remain the correctness path; the standalone forwarding mode retires its legacy socket route.
This diagram shows how a local change reaches the shared history, the cloud projection, and other devices.
flowchart LR
A[Local device] --> B[Authenticated edge worker]
B --> C[Per-user sync hub]
C --> D[Ordered canonical log]
D --> E[Cursor-based device reads]
D --> F[Projection pages]
F --> G[Cloud projector]
G --> H[Projection checkpoint]
C -. advisory hint .-> A
E --> A
Canonical content
Canonical content is a versioned record whose stable identity, revision, payload, and hashes can be checked by any participant. The platform supports observations, summaries, prompts, deletion tombstones, and a small set of synchronization mutations. Canonical JSON and digest checks make retries safe to recognize and prevent altered content from entering the shared log. canonical-content.ts canonical-content.ts
The hub compares each incoming revision with the current entity revision. A newer revision becomes the next ordered change. An identical revision and hash is acknowledged again, while an older revision or a same-revision hash conflict is refused. The origin device id must also match the authenticated device. SyncHub.ts
References: workers/…/canonical-content.ts• L1-40 workers/…/canonical-content.ts• L154-197 workers/…/SyncHub.ts• L373-488
Client convergence
A device pushes a bounded batch and receives acknowledgements with server sequence numbers. It reads later changes with a cursor and advances that cursor only as it accepts the returned stream. This lets offline devices retry the same operation and resume from durable progress.
The WebSocket carries advance or operation hints only. A hint can wake a client or reduce latency, but a lost socket or dropped frame does not lose data because the next HTTP read uses the ordered log. SyncHub.ts SyncHub.ts
References: workers/…/SyncHub.ts• L281-367 workers/…/SyncHub.ts• L373-488 workers/…/SyncHub.ts• L490-557
Projection path
Projection copies the authoritative ordered log into the cloud content service. The hub supplies bounded pages and a short per-user lease; the edge worker performs the network call and advances the hub checkpoint only after the cloud service returns a matching result.
Projection checkpoint
The projected_seq value records the last sequence known to be applied by the cloud projector. A successful public push is complete only when this checkpoint covers the new log head. If projection fails after the log append, the response says that the data is durable and retryable, so a later request or repair job can continue without appending a second copy. index.ts DEPLOY.md
The lease and checkpoint form a fence around an in-flight page. A timed-out or unclear cloud request keeps its lease until expiry, and a later worker can advance the checkpoint only when its lease token, epoch, and expected prior sequence still match. This prevents two workers from claiming the same progress or allowing a stale request to move the checkpoint backward. SyncHub.ts index.ts
The cloud projector can also request a repair drain for a user whose checkpoint lags. The hub checkpoint remains authoritative; a successful response from the projector alone does not prove that the hub has recorded the same progress. index.ts DEPLOY.md
References: workers/…/index.ts• L307-364 workers/…/index.ts• L600-805 workers/…/index.ts• L814-852 workers/…/SyncHub.ts• L632-781 workers/…/DEPLOY.md• L81-110
Operational controls
Operational controls protect both availability and cost without making synchronization stop when one lane is unhealthy. The control plane runs in the stateless worker so monitoring and outbound alerts cannot block the durable per-user hub.
Poll-mode protection
The kill switch is a durable control flag. When it is present, the worker rejects new WebSocket upgrades and marks every HTTP sync response with X-Sync-Mode: poll. Clients then use the HTTP path, which keeps data movement and correctness available while removing the socket lane that can keep durable objects active. kill-switch.ts index.ts
The switch does not clear itself. A human clears it after the cause is understood. This avoids a loop in which an automatic clear repeatedly restores an unsafe mode while the underlying metric still looks healthy only because the switch is active. kill-switch.ts DEPLOY.md
References: workers/…/kill-switch.ts• L1-35 workers/…/kill-switch.ts• L100-117 workers/…/index.ts• L885-954 workers/…/DEPLOY.md• L287-313
Metric watchdog
The hourly watchdog compares hub activity with alert and kill thresholds. It sends an alert for a threshold breach. It trips poll mode first for duration, row-write, or row-read runaway behavior because poll mode directly removes the cause. A request-count breach raises a severe alert but does not auto-trip because polling still uses HTTP requests. watchdog.ts watchdog.ts
A failed analytics query is logged as an unknown monitoring state. It does not create a false incident or trip the switch. Discord is also a notification path only, so a failed alert post cannot prevent a kill-switch write. watchdog.ts watchdog.ts
References: workers/…/watchdog.ts• L61-126 workers/…/watchdog.ts• L422-519
Control-plane probe
The five-minute probe checks the database-backed token verifier from outside the application hosting domain. A healthy result is a JSON 401 or 403 for a deliberately invalid token. This tests that the application and its database lookup both answer, rather than only checking whether a static web page is reachable. control-plane-probe.ts control-plane-probe.ts
The probe confirms a first failure before it alerts, limits repeat alerts while the failure continues, and posts one recovery notice when the service returns. Maintenance can silence the check through its control state. Probe or Discord failures remain isolated from sync requests. control-plane-probe.ts control-plane-probe.ts
References: workers/…/control-plane-probe.ts• L1-37 workers/…/control-plane-probe.ts• L104-203 workers/…/control-plane-probe.ts• L332-477
Related pages
- Sync-hub worker processing explains request routing, hub operations, canonical content validation, projection exchange, and notification hints.
- Sync API service explains the standalone sync API's authentication, content storage, projection and socket protocols, configuration, and health behavior.
- Sync-hub control-plane safeguards explains authentication dependencies, kill-switch operation, watchdog thresholds, uptime probing, and deployment safeguards.
Change checklist
When these files or sections are changed, remember to consider these:
- Changes to canonical body or payload versions require coordinated updates to local producers, hub validation, projection consumers, and any replay or reset process.
- Changes to sequence, epoch, or projection-checkpoint rules must preserve the client cursor contract and the repair endpoint's meaning of authoritative progress.
- Changes to the kill-switch signal must preserve both parts of the client contract: WebSocket refusal and the poll-mode header on successful and failed HTTP responses.
- Changes to token verification must preserve the canonical user-id binding before positive verdicts enter the cache.
- Changes to the standalone forwarding boundary must preserve the upstream user and device context, response status and retry signals, and the HTTP fallback for retired socket routes.
- Changes to forward-mode body handling must preserve streaming for large sync exchanges and the upstream encoding contract for operation pushes.
- Changes to watchdog thresholds or metric dimensions must keep the auto-trip choice aligned with the failure that poll mode can actually remediate.
- Changes to cron dispatch or shared KV key prefixes must keep the watchdog, uptime probe, and emergency switch isolated from each other.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| A verifier returns success without the canonical user id or for a different claimed user | A request is refused instead of gaining cross-user access | The edge worker requires an exact token-to-user binding before caching a positive verdict |
| Projection fails after a durable append | The client sees a retryable failure even though the operation is safe to retry | The response reports durable state and the hub keeps the checkpoint behind the log head |
| A projection request times out or returns an unclear result | A successor cannot immediately overlap the possibly still-running request | The per-user lease remains until expiry and the checkpoint uses a fenced compare-and-set |
| The standalone forward upstream is unreachable | A sync request gets a retryable 503, while health can still report that the proxy process is alive | The boundary returns sync_hub_unavailable with Retry-After and clients retry the HTTP path |
| The kill switch is present | WebSockets are refused and clients move to polling; sync remains available with higher latency | Every HTTP sync response carries the poll-mode signal and the switch requires deliberate clearing |
| Analytics or Discord is unavailable | Monitoring loses evidence or notification, but sync is not stopped | Watchdog query failures do not fabricate trips and notification failure cannot block the control state |
| The control-plane verifier accepts the deliberately invalid probe token | The event is treated as a security incident rather than a normal outage | The probe uses a fixed bogus token and sends a distinct security alert for any 2xx response |
Key files
| File | Why to read it |
|---|---|
| workers/…/index.ts | Shows the platform boundary, public and internal routes, authentication, projection orchestration, and scheduled control tasks. |
| workers/…/SyncHub.ts | Shows the per-user authority for ordered changes, device cursors, advisory hints, and projection checkpoints. |
| workers/…/canonical-content.ts | Defines the shared content identity and validation rules that make operations safe to replay and project. |
| workers/…/projection-protocol.ts | Defines the bounded wire envelope between the hub and the cloud projector. |
| services/…/forward.ts | Shows the standalone API's forwarding boundary, HTTP fallback, and upstream failure behavior. |
| services/…/forward.test.ts | Shows the compatibility contract for forwarded requests, streamed content, retired routes, and retryable upstream failures. |
| workers/…/kill-switch.ts | Explains the poll-mode safety signal and its operator-controlled state. |
| workers/…/watchdog.ts | Explains metric-based escalation and the cost controls that can trip poll mode. |
| workers/…/control-plane-probe.ts | Explains independent control-plane health checks and anti-flap alert behavior. |
| workers/…/wrangler.jsonc | Shows the worker, durable-object, KV, and scheduled-trigger relationships at deployment level. |
| workers/…/DEPLOY.md | Gives the operational contract for deployment order, repair, emergency controls, and production verification. |