Cloud relay and notification services
Cloud services keep Orca devices connected when direct links are not possible. They also carry work from team chat to a desktop host and deliver mobile alerts when a live path is not available. The relay, connector, push gateway, and private operations boundary use separate admission, storage, identity, and recovery rules so each path can scale without weakening the others.
Service boundaries
The cloud system separates live device traffic from queued connector work and delayed mobile notifications. This lets each service protect its own state and use the recovery model that fits its traffic.
| Service | Purpose | Main boundary |
|---|---|---|
| Relay director | Authenticates a host and assigns a serving cell or edge path | Public relay control service |
| Relay cell | Holds device sessions and forwards encrypted frames | Regional relay endpoint |
| Relay edge | Serves a controlled host cohort from a global edge path | Edge host and splice objects |
| Connector API | Authenticates Orca machines and manages durable work | Machine API and inbox |
| Slack edge | Receives signed Slack events and sends replies | App-scoped Slack routes |
| Discord edge | Receives gateway messages and signed interactions | Allowlisted Discord guilds |
| Push gateway | Authenticates desktop control and accepts mobile alerts | Separate push service |
| Fence broker | Performs a narrow protected relay mutation | Private IAM-only service |
| Deployment plane | Builds, validates, and changes cloud resources | Private cloud operations repository |
The relay, connector, and push services keep separate durable state and runtime images. The public Cloud workspace defines runtime and wire contracts. Production infrastructure and rollout workflows remain in the private cloud operations repository. The relay edge adds a separate serving path whose assignment and device-state handoff can also be coordinated by an optional edge-native director.
References: cloud/README.md• L17-34 cloud/…/connector-runtime.ts• L12-44 cloud/…/edge-relay.ts• L44-69 cloud/…/index.ts• L19-33 cloud/…/package.json• L1-27
Relay service
The relay gives a paired phone and desktop host a secure path without requiring either device to accept an inbound connection. Both devices open outbound connections to the cloud, and the relay pairs those connections inside a regional cell or an approved edge path.
Relay edge
The relay edge is an additional serving path for a controlled cohort of hosts. The director records the edge as the host's assignment, while a per-host edge object keeps the host control session, phone connection bookkeeping, and edge-side credential state needed to serve that host.
The move onto the edge is a guarded handoff. The director pushes the host's device state before it commits the edge assignment, and assignment epochs prevent an older move from becoming current. A host leaving the cohort exports its device state before normal placement. If the edge stays unavailable, the director eventually places the host without that export and removes the old durable device rows so revoked or unknown edge state is not restored.
The edge cohort uses stable host buckets with account, host, region, allow, and deny controls. A runtime mode can keep the cohort active, evacuate it, or treat the edge as unavailable. The edge does not replace the regional continuity rules: it is another serving target that must be fenced when a host moves away.
The edge can also expose an optional edge-native director for a controlled migration path. When enabled for its director hostname, it answers host assignment, phone resume, region catalog, and invite-move requests from the host object's own state. Country-aware routing can send selected countries to a front door and other clients to the edge. The feature is dark when its hostname is unset, so ordinary edge routes keep their existing behavior.
The edge-native director keeps the same assignment epoch when a host receives the same serving URL. A URL change creates a newer epoch, and an account change clears the earlier account's edge data before the new assignment takes effect. A per-user host index supports account erasure for hosts that the edge-native director assigned without a regional assignment row; it stores host identifiers rather than credentials and converges through retryable updates.
This diagram shows how a selected host moves between the director, the edge path, and the normal regional relay path.
flowchart LR
Host[Desktop host] --> Director[Relay director]
Director -->|Cohort admission| Edge[Relay edge]
Edge -->|Host control and phone sessions| Device[Paired phone]
Edge -->|Export on exit| Director
Director -->|Regional placement| Cell[Relay cell]
Director -.->|Fence old edge assignment| Edge
Edge -.->|Optional native assignment and resolve| Native[Edge-native director]
Native -.->|Host object state| Edge
The edge fence is durable work. If an edge reports a stale assignment or a retry is due later, the director re-reads the host's current assignment before sending the fence. Only one director claims a due retry at a time, and the pending record clears after the current fence succeeds. This prevents a retry from fencing an earlier assignment or from running twice during concurrent recovery.
References: cloud/…/edge-relay.ts• L44-69 cloud/…/edge-relay.ts• L126-198 cloud/…/edge-relay.ts• L256-346 cloud/…/edge-relay-cohort.ts• L52-85 cloud/…/edge-relay-fence.ts• L65-100 cloud/…/director.ts• L97-160 cloud/…/director-config.ts• L36-87 cloud/…/host-relay-director.ts• L20-63 cloud/…/user-hosts.ts• L1-20 cloud/…/director-contract.test.ts• L66-155 cloud/…/edge-relay-fence.test.ts• L84-116 cloud/…/user-index-write.test.ts• L63-119 cloud/…/edge-relay-fence.ts• L105-179
Relay ingress
Relay ingress is the trust boundary in front of the edge path. The public proxy terminates client TLS, removes client-controlled identity and hop headers, and forwards only relay routes to the edge over TLS with a signed claim that carries the public origin and client address. WebSocket upgrades become a byte pipe so the edge keeps control of session admission and close behavior.
The edge also accepts a load-balancer trust mode for front doors that cannot sign a claim. That mode requires the configured origin, a matching secret token, and a source address in the approved load-balancer ranges. The edge validates the forwarded client address and country before it treats them as trusted data. Requests that do not meet a trust mode do not gain the front-door identity used for admission or rate limits.
The proxy keeps health on a private listener and supports controlled drain. It fails health first, stops new upgrades, allows existing tunnels a bounded time to close, and then cuts the tunnels that remain. Logs contain counts rather than paths, headers, or addresses because relay paths can contain host identifiers and ingress headers can contain tokens.
References: cloud/…/proxy.ts• L1-5 cloud/…/proxy.ts• L28-55 cloud/…/proxy.ts• L63-105 cloud/…/proxy.ts• L143-166 cloud/…/ingress.ts• L1-11 cloud/…/ingress.ts• L20-43 cloud/…/index.ts• L145-155 cloud/…/Dockerfile• L1-34
Relay admission and continuity
The director turns a host identity into a serving-cell assignment. Durable assignment state lets a reconnect keep its home cell when that cell is safe. Health, capacity, region preference, and migration state guide new placement or a controlled move.
Public admission uses separate bounded lanes for fresh placement, verified reconnects, and returns from a draining cell. This keeps recovery work from being starved by new placement traffic and makes temporary overload a retryable result. A reserve placement path can book capacity from a cell's live view during a controlled database interruption, while normal database placement remains the source of truth for other hosts.
Continuity rules keep a live host pinned to a serving cell unless the system has proof that a move is safe. A cell that serves from memory can return to database admission after its director contact expires. Its live seats remain protected during that change, and the cell re-registers them before normal database maintenance resumes. Stale cell evidence, an unknown reserve-mode view, or a missing connection-cap snapshot fails closed rather than creating a second seat.
Relay admission also protects continuity during cell drain, regional change, and temporary database or cell failure. The system treats an uncertain control result as uncertain instead of releasing a live assignment without proof. Detailed admission states, reconnect lanes, reserve admission, credential resume, and regional rehome rules are covered in cloud relay admission and continuity.
References: cloud/…/app.ts• L212-294 cloud/…/app.ts• L412-694 cloud/…/assignment-store.ts• L666-867 cloud/…/reserve-assignment.ts• L173-279 cloud/…/cell-reserve-dead-man.ts• L48-109
Secure session path
The session path starts with host assignment and ends with paired WebSocket sessions that the serving target can authenticate and splice. The shared relay contract keeps message shapes, assignment epochs, and continuity fields consistent across cloud, desktop, and mobile clients.
This diagram shows how the director places a host and how the serving target carries the live connection.
flowchart LR
Phone[Paired phone] -->|Outbound connection| Director[Relay director]
Desktop[Desktop host] -->|Authenticated control| Target[Assigned cell or edge]
Director -->|Assignment and endpoint| Phone
Phone -->|Connect to assigned target| Target
Target -->|Verify sessions and forward frames| Desktop
Contract[Shared relay contract] -.-> Director
Contract -.-> Target
The host proves control of its identity before the cell or edge accepts the session. The serving target checks the assignment against its durable view and uses the assignment epoch to reject an old or moved host. A signed assignment lease can speed this check, but it does not replace the durable admission decision. On the edge, a host control can rebind within its grace window so live phone splices survive a transient object reset; a new generation replaces the old control when rebind proof is not available.
References: cloud/…/relay-server.ts• L627-724 cloud/…/host-session-registry.ts• L811-1270 cloud/…/host-relay-base.ts• L90-118 cloud/…/host-relay.ts• L25-57 cloud/…/host-relay.ts• L181-249 cloud/…/host-relay-handshake.ts• L13-86 cloud/…/host-relay-handshake.ts• L89-157 cloud/…/host-relay-base.ts• L88-100 cloud/…/assignment-lease.test.ts• L63-90 cloud/…/package.json• L1-52
Connector service
The connector service links signed-in Orca machines with external team conversations. It accepts signed events from Slack and Discord, maps each conversation to one account-owned machine and project, queues work durably, and returns machine progress to the source conversation.
Connector delivery path
Connector requests enter through source-specific ingress or a machine-facing API. Shared core rules bind a conversation to one host, keep delivery order per conversation, and use a held poll or wake signal so a connected machine can receive work without an open request for the full wait.
Machine state is versioned by each hello. A new hello makes old polls, acknowledgements, and events stale. The hello contract keeps unknown capability, feature, and source names so newer machines can still describe themselves to older cloud code. It also bounds project count and display text so host metadata stays limited.
Durable inbox entries can be recovered after a restart, while expiry and retention sweeps remove work that can no longer be delivered.
This diagram shows how external conversation work moves through the connector to a machine and back.
flowchart LR
Source[Slack or Discord] -->|Verified event| Edge[Source edge]
Edge --> Core[Shared connector core]
Core -->|Account and project choice| Inbox[Durable machine inbox]
Inbox -->|Wake or held poll| Machine[Signed-in Orca machine]
Machine -->|Acks, events, replies| Core
Core -->|Queued status and messages| Outbox[Durable source outbox]
Outbox --> Edge
Edge --> Source
The connector also keeps thread state, prompt state, trusted teammate context, and source posts as separate projections of the same conversation. Its delivery rules belong to the shared connector service rather than to the relay or push paths.
References: cloud/…/connector-runtime.ts• L12-44 cloud/…/connector-core.ts• L29-45 cloud/…/connector-inbox.ts• L36-40 cloud/…/machine-routes.ts• L24-40 cloud/…/contract.test.ts• L33-83
Slack source
The Slack source admits requests only after checking the raw-body signature, the app or legacy workspace scope, and the workspace install. It stores bot and app secrets in sealed form, supports one Orca app per workspace, and lets operators pause a workspace without stopping the service.
Slack messages from linked full members become connector input. The source formats status posts, prompt cards, acknowledgements, private notices, and agent replies through the shared outbox. Ambiguous sends are reconciled through Slack message metadata or remain uncertain rather than risking a duplicate.
The source keeps older thread messages bounded and sends only messages from trusted workspace members as agent context. Its request limits, five-minute signature window, workspace pause state, and provider retry results form the boundary between Slack and the shared core.
References: cloud/…/slack-edge.ts• L32-63 cloud/…/slack-edge.ts• L65-121 cloud/…/slack-installs.ts• L13-29 cloud/…/slack-sealing.ts• L3-24 cloud/…/slack-post-sender.ts• L41-61
Discord source
The Discord source uses one long-lived gateway role to read allowlisted guild messages and a separate API role to handle signed interactions and send posts. A database checkpoint records the last gateway dispatch committed with its connector effects, so the next leader can resume after a restart.
Discord admits only valid Ed25519-signed interactions from allowed guilds. It ignores bots, webhooks, direct messages, and unrelated servers. The source creates a thread for a top-level mention when needed, turns prompt options into bounded buttons, and sends owner-only answers through the shared machine inbox.
A database leadership lock allows only one gateway worker to own the Discord session across instances. A failed dispatch stops that session so the next leader replays from the last committed checkpoint.
References: cloud/…/discord-gateway-worker.ts• L15-19 cloud/…/discord-gateway-worker.ts• L42-89 cloud/…/discord-gateway-checkpoint.ts• L7-30 cloud/…/discord-edge.ts• L101-181 cloud/…/discord-signature.ts• L16-37
Push notification service
The push gateway delivers alerts when a live relay path is idle or unavailable. A desktop host proves its identity, registers a paired phone's native token, and submits an event that the gateway stores before provider delivery.
Notification acceptance and delivery
The gateway separates request acceptance from provider delivery. It validates the host session and notification shape, records the event durably, then lets workers send it through Apple Push Notification service or Firebase Cloud Messaging.
Each delivery has an expiry, an identity for deduplication, and a device target. Per-device ordering prevents a later event from passing an earlier one. Leases and periodic polling let another worker continue after a crash or temporary provider failure. Before a persisted retry, the worker checks that the device is still registered; an unregistered device does not receive the retry. Worker shutdown stops new queued claims, while deliveries already in progress can finish. Work for another device can continue while a provider call remains pending for an earlier device.
This diagram shows the durable handoff from desktop request to native mobile delivery.
sequenceDiagram
participant Host as Desktop host
participant Gateway as Push gateway
participant Store as Durable push state
participant Provider as APNs or FCM
participant Phone as Paired phone
Host->>Gateway: Prove host identity
Host->>Gateway: Register phone token
Host->>Gateway: Submit notification
Gateway->>Store: Record event
Gateway-->>Host: Queued result
Store->>Provider: Send due event
Provider-->>Store: Delivery result
Provider-->>Phone: Native notification
The gateway applies host and request limits before accepting work. It exposes separate health and readiness checks and rejects new application work while a controlled shutdown drains active requests. Provider failures can cause a retry or terminal result. An invalid token is retired so later events do not keep targeting a dead registration. The live notification socket supports dismissal and state reconciliation; it is not the durable banner path. Detailed provider rules, ordering, token handling, and mobile handoff are covered in push notification delivery.
The gateway also uses separate limits for unauthenticated clients, authenticated client addresses, and authenticated hosts. Host challenge and session requests are bounded before they reach session state. Device registration and send requests require a valid host session, and a send is accepted only for a device owned by that host. A successful acceptance wakes delivery work while the durable store remains the source of recovery after a worker or provider failure.
References: cloud/…/push-server.ts• L47-100 cloud/…/push-server.ts• L143-221 cloud/…/push-server.ts• L223-292 cloud/…/durable-push-store.ts• L9-80 cloud/…/durable-push-worker.ts• L17-97 cloud/…/durable-push-worker.test.ts• L68-159 cloud/README.md• L36-53
Notification contract
The push contract defines the bounded event accepted by the gateway. It carries event identity, sequence, expiry, source, device-facing text, and optional dismissal or execution-host data.
The contract rejects unknown fields, over-sized text, invalid collapse identifiers, empty targets, and batches above the registration limit. It also removes repeated registration IDs before the gateway applies per-request work.
References: cloud/…/send-messages.ts• L9-72 cloud/…/send-messages.test.ts• L18-107
Multi-region infrastructure
The infrastructure presents managed public services while giving relay cells stable regional identities. A director chooses a cell, and that cell remains the serving target for assigned hosts until a guarded drain, roll replacement, or regional rehome changes the assignment.
Regional relay shape
The relay image runs in director or cell mode. The director owns placement and migration decisions. Cells own live connections and publish evidence needed for readiness, capacity, and continuity decisions.
Regional placement is a preference and capacity decision. It is not permission to move active work without continuity checks. Reserve placement keeps a host in its preferred region when a live reserve cell has room, then spills to another region only when the preferred region cannot safely serve it. Idle regional rehome moves eligible hosts only when both the source and target pass current safety checks.
The edge-native director adds an optional global serving choice without changing the regional model. Its region catalog gives clients probe origins owned by the director hostname, and a preferred region becomes a location hint only when a new host object is created. Existing objects keep their placement. A configured front door is selected by trusted client country; all other traffic uses the edge URL.
References: cloud/…/index.ts• L49-82 cloud/…/assignment-store.ts• L948-1034 cloud/…/reserve-assignment.ts• L173-279 cloud/…/regional-rehome-worker.ts• L23-127 cloud/…/director-config.ts• L69-87 cloud/…/director-config.test.ts• L46-79 cloud/…/package.json• L33-52
State and identity boundaries
Relay assignments, credentials, and cell control state form the durable basis for live-session continuity. Connector thread and inbox state and push delivery state remain separate because queued work has different ownership, retention, ordering, and retry needs.
Service identities limit which runtime or workflow can read state or request a control action. Reserve-cell writes use a cell-only route with regional rehome credentials. The fence broker provides one narrow protected mutation path. It owns the durable mutation lease and target change, while its caller does not receive those mutation permissions. Edge host state is likewise isolated from director state, with signed admin calls used for assignment, export, drain, erase, and fencing.
The edge-native director adds a signed control boundary for its own operations. Tokens bind an operation to the exact request body and can be single-use through a transaction identity. A separate erasure identity can list and erase user indexes but cannot assign hosts. The user index contains host IDs and timestamps only, so it helps find edge-owned data without becoming another credential store.
References: cloud/README.md• L22-34 cloud/…/assignment-store.ts• L666-867 cloud/…/admin.ts• L1-22 cloud/…/admin.ts• L113-166 cloud/…/host-relay.ts• L25-75 cloud/…/user-hosts.ts• L51-93 cloud/…/edge-relay.ts• L201-253 cloud/…/edge-relay-fence.ts• L65-100 cloud/…/cell-reserve-routes.ts• L26-65 cloud/…/connector-schema.ts• L3-121 cloud/…/durable-push-store.ts• L29-40
Runtime packaging and security posture
Relay, connector, push, and ingress build as separate production images from the Cloud workspace. The images use the pinned Node 24.22.2 runtime, install only production dependencies in the runtime stage, and run as the non-root node user. The edge Worker is separately packaged with its host and splice objects, optional user index, and private deployment bindings.
The security review is a release input rather than proof of live deployment state. It records clean relay and push runtime image scans, remaining upstream findings in external tools, and the need to bind immutable image digests to rollout inputs.
References: cloud/.node-version• L1 cloud/…/Dockerfile• L1-37 cloud/…/Dockerfile• L1-47 cloud/…/Dockerfile• L1-45 cloud/…/Dockerfile• L1-34 cloud/…/wrangler.jsonc• L19-55 cloud/…/relay-security-2026-10-21.md• L3-15 cloud/…/relay-security-2026-10-21.md• L112-119
Operational controls
Operational controls make placement, deployment, and recovery changes deliberate. Public runtimes expose readiness, health, bounded admission, durable queue state, and aggregate evidence. The private cloud operations system owns infrastructure plans and rollout steps.
Deployment and recovery controls
Deployment control follows a candidate-first model. A new image or topology change must be built and checked before it can receive traffic or new assignments. A known-good image and assignment state remain available so an unhealthy change can be stopped or reversed.
Global cell admission belongs to the director-capable role rather than each cell. Startup reconciliation converges the configured cell set and retries transient database pressure within a bounded window. Cell control flags start from safe defaults and apply only after a valid control read. Director-owned background work also records assignment and migration inventory, samples database pressure, and drives regional rehome decisions. This prevents a new worker from enabling itself before the candidate has passed the deployment checks. The private deployment plane owns Terraform, cloud backends, workflow trust, and regional rollout steps. The public workspace builds and tests runtime code, checks dependencies and images, and records security findings without cloud credentials. The fence broker adds a narrow IAM boundary for the relay mutation that needs protected cloud access.
The relay edge adds runtime cohort control to this model. Its applied mode and rollout share can change without a director deploy. A control change re-arms an evacuation sweep, which drains hosts that no longer belong to the cohort. A host move exports or deliberately discards edge-side device state before normal placement, then fences the old edge assignment so phones resolve the new target.
The optional edge-native director uses the same separation of duties. Its public endpoints are enabled only by a valid hostname and HTTPS origin configuration. Signed admin calls control assignment, export, drain, verification, and erasure. The user index is updated outside the client assignment request and retries from object alarms, so a missing binding or temporary failure does not silently lose the erasure record.
References: cloud/…/cell-admission-startup.ts• L18-47 cloud/…/index.ts• L49-69 cloud/…/index.ts• L88-180 cloud/…/edge-relay-control.ts• L11-109 cloud/…/edge-relay-evacuation.ts• L11-27 cloud/…/edge-relay-evacuation.ts• L62-103 cloud/…/edge-relay-fence.ts• L105-179 cloud/…/package.json• L1-27 cloud/…/admin.ts• L113-166 cloud/…/host-relay-director.ts• L66-103 cloud/…/user-index.ts• L30-47 cloud/README.md• L66-75 cloud/…/relay-security-2026-10-21.md• L166-174
Runtime safety evidence
The relay reports readiness and live operating evidence before it accepts or moves work. The evidence includes cell identity, connection counts, database pressure, admission mode, migration state, regional safety, event-loop delay, and network counters where the runtime can read them. Missing or stale evidence must block a safety-sensitive action instead of being read as a healthy zero.
Connection admission uses the lower of the configured cap and the latest cap reported by the cell. A cap raise therefore affects placement only after the cell reports the new value. The relay also samples database lock waits and retires a connection only when its recent latency is slow compared with peers. On shutdown, it stops background work, drains sessions, gives sockets and database work time to settle, and then exits. Push services have separate readiness and request-drain behavior. They stop new work before shutdown while durable state remains available to the next worker or poll.
The edge also records route, director, ingress, pre-auth, object, and control outcomes as aggregate metrics. Metric failure cannot affect the relay path. This gives operators a view of front-door trust, director availability, object recovery, and user-index convergence without placing request data or secrets in the telemetry path.
References: cloud/…/index.ts• L127-180 cloud/…/index.ts• L183-237 cloud/…/relay-observability.ts• L378-415 cloud/…/assignment-store.ts• L307-333 cloud/…/assignment-connection-headroom-query.ts• L1-17 cloud/…/postgres-connection-latency.ts• L63-153 cloud/…/push-server.ts• L92-100 cloud/…/metrics.ts• L53-79
For detailed admission continuity, push delivery, and deployment controls, see cloud relay admission and continuity, push notification delivery, and cloud operational deployment controls. The remote connectivity page explains how cloud relay fits with other device connection paths. The mobile client page explains the phone-side relationship with relay sessions and push notifications.
Change checklist
When these files or sections are changed, remember to consider these:
- Keep relay assignment, cell or edge identity, region origin, readiness, admission mode, effective connection cap, and migration state aligned across the director, serving targets, and shared contract.
- Keep edge cohort mode, rollout share, allow and deny lists, leave ceiling, evacuation cadence, and assignment reads aligned across runtime control and director decisions.
- Keep edge device export, fallback removal, assignment epochs, fences, and phone resolution behavior aligned across the director and edge object.
- Keep optional director hostname matching, front-door country routing, region probe ownership, location hints, response schemas, and dark-mode behavior aligned across the edge Worker and desktop or phone clients.
- Keep native director URL changes, assignment epochs, account-switch cleanup, legacy fallback placement, and object state aligned across host objects and serving paths.
- Keep signed admin operation, issuer, audience, purpose, body hash, expiry, key rotation, and single-use identity rules aligned across director callers, edge verification, and object handlers.
- Keep user-host index add, remove, backfill, erase, binding rollout, and retry state aligned with account ownership and host-object erasure.
- Keep signed ingress claims, token-mode origin and source checks, forwarded client identity, stripped headers, and trusted rate-limit inputs aligned across the proxy, edge, and load balancer.
- Keep live relay state separate from connector state and push delivery state when changing databases, retention, credentials, or service identity.
- Keep host proof, assignment epochs, resume credentials, signed assignment leases, and cell or edge admission rules compatible across desktop, mobile, and cloud clients.
- Keep reserve bookings, seat feeds, director contact, dead-man fallback, memory phone admission, and database re-registration aligned across directors and cells.
- Keep connector token scope, hello revisions, host ownership, inbox order, wake or poll behavior, expiry rules, and hello size limits aligned across the machine client and service.
- Keep unknown connector capability, feature, and source values forward-compatible while preserving bounded project and display data.
- Keep source ingress identity, workspace or guild scope, linked account mapping, and pause controls aligned for Slack and Discord.
- Keep connector event ordering, prompt ownership, source message references, and outbox post identity aligned across the shared core and each source edge.
- Keep gateway leadership, dispatch checkpoints, replay behavior, and Discord gateway configuration aligned across worker instances.
- Keep push host challenge and session admission, per-IP and per-host limits, device ownership, event identity, device order, expiry, provider result, token retirement, retry state, and worker stop behavior aligned across the gateway, contract, store, and worker.
- Keep runtime image digests, security findings, private rollout inputs, and current provider exceptions aligned before a cloud change receives traffic.
- Keep fence-broker, edge-admin, and erasure permissions narrower than the workflow permissions that invoke them.
- Keep readiness and shutdown behavior compatible with durable queue claims, connector polls, relay drains, edge evacuation, edge fences, reserve-cell re-registration, and regional moves.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| A cell loses fresh identity, readiness, or connection-cap evidence | New work can reach a target that no longer proves it is serving | Fresh heartbeat evidence, incarnation checks, and guarded placement decisions |
| A reserve-mode view is unknown or stale | Database placement or maintenance can treat memory-owned seats as absent | Fail-closed reserve-mode reads and cell-local admission protection |
| Director contact expires while a cell serves from memory | The cell can hold memory seats without a path back to durable control | Dead-man fallback, bounded trip slots, and re-registration before database mode |
| A relay move loses its durable assignment proof | A host can be released while its new serving path is not ready | Durable assignment state, assignment epochs, and source-retention rules |
| A host moves to the edge before its device state is accepted there | The host can appear assigned while phones cannot use its credentials | Push device state before commit and defer the move when the edge is busy |
| Edge control lowers the leave ceiling below a prior rollout | A host can skip the edge-row read and be placed without exporting edge state | Monotonic ceiling checks and evacuation reporting for rows above the ceiling |
| An edge export stays unavailable during a move away | The director can loop on temporary failures or restore stale device state | Bounded export wait, deliberate fallback removal, and later edge fencing |
| A failed edge fence is never retried | Phones can see the old edge assignment instead of resolving the host's new target | Durable pending fences, current-row retries, single-claim retry, and bounded backoff |
| The native director is enabled with a bad hostname or cell origin | Clients can miss assignment routes or be sent into a director loop | Strict configuration validation and disabled endpoints on invalid setup |
| A native director URL changes without a newer assignment epoch | A desktop or phone can continue using a stale serving path | Epoch bump on URL change and strict client response contracts |
| A host object changes account without clearing earlier data | The new account can inherit old credentials or the erasure index can point to the wrong user | Account-switch wipe, assignment ownership checks, and remove-then-add index intent |
| A user-host index binding is absent or unreachable | Account erasure can miss an edge-owned host or remain incomplete | Pending marker, alarm retry with capped backoff, and explicit unbound metrics |
| A front door sends an unapproved token or client address claim | The request cannot use trusted origin or client identity for edge admission | Signed claims or matching token, origin, source CIDR, and parsed client address checks |
| The ingress proxy drains while a tunnel remains open | A live relay session can outlast the load balancer's replacement window | Early health failure, bounded tunnel drain, and forced close at the deadline |
| A cap is raised before the cell reports the new cap | Placement can exceed the cell's real connection capacity | Effective cap remains at the last cell-reported value |
| An admin token is reused or its body changes | A protected assignment, move, or erasure can run twice or act on different data | Short-lived signed token, exact body hash, purpose and audience checks, and per-object replay guard |
| A workflow receives broad cloud mutation rights | A failed or compromised workflow can change more infrastructure than intended | Private operations identities and narrow fence-broker, edge-admin, and erasure boundaries |
| A connector hello becomes stale while a poll, ack, or event is in flight | Old machine state can change a newer session or delivery view | Host locks and hello revision checks |
| A machine hello exceeds the project or display limits | The machine cannot enter the connector with unbounded host metadata | Contract bounds on project count and UTF-8 display size |
| A connector inbox entry expires without a wake or poll completing | External work can remain undelivered or appear stale to the machine | Durable inbox state, leases, expiry sweeps, and reconnect recovery |
| A source event is accepted under the wrong app, workspace, or guild | Untrusted external input can reach an Orca account or source thread | Raw-body signature checks and source installation or allowlist checks |
| A Slack or Discord post has an unclear provider result | Retrying can create a duplicate or stopping can lose a visible update | Source-specific reconciliation and uncertain post state |
| A Discord gateway leader dies after processing but before checkpoint commit | A message can be replayed after failover | Commit the checkpoint with the routed effects in one leadership transaction |
| A push request uses an invalid or expired host session | An untrusted caller can register devices or enqueue alerts for another host | Challenge proof, session expiry, bearer checks, and host ownership checks |
| Push acceptance is attempted after request drain starts | New application work can enter while the service is shutting down | Request-drain middleware and separate readiness behavior |
| A persisted push retry targets a device that was unregistered | The worker can send an alert to a stale provider token | Recheck device ownership before retry and clear the pending delivery |
| A push worker stops while it is choosing another queued device | The remaining delivery can be leased or delayed after shutdown | Stop-aware claim admission and release of an unstarted claim |
| A push event passes an earlier event for the same device | The phone can show stale or misleading state | Per-device ordering, claims, and delivery leases |
| A provider reports an invalid push token | Future sends keep targeting a dead registration | Terminal token retirement |
| A queued worker stops after the gateway accepts an event | The request succeeds but delivery waits for later recovery | Durable queue state, wake-up, polling, and restart-safe leases |
| A candidate receives traffic before checks pass | A rollout can replace a working service with an unusable revision | Candidate validation, readiness gates, and retained recovery state |
Key files
| File | Why to read it |
|---|---|
| cloud/README.md | Product boundary, relay shape, push behavior, connector scope, deployment ownership, and local workspace model |
| cloud/…/app.ts | Public relay admission, assignment, edge cohort routing, continuity, regional controls, and protected control boundary |
| cloud/…/edge-relay.ts | Edge assignment, device-state push and export, fallback behavior, and fencing handoff |
| cloud/…/edge-relay-cohort.ts | Edge cohort membership, rollout bounds, allow and deny rules, and epoch floor |
| cloud/…/edge-relay-control.ts | Runtime edge mode, rollout, leave ceiling, and safe control parsing |
| cloud/…/edge-relay-evacuation.ts | Cohort-change evacuation and bounded edge drain work |
| cloud/…/edge-relay-fence.ts | Durable retry of edge assignment fences after a host leaves |
| cloud/…/edge-relay-fence.test.ts | Guarantees that fence retries use the current assignment and allow only one concurrent claim |
| cloud/…/index.ts | Edge routing, trust boundaries, optional director activation, and admin operation dispatch |
| cloud/…/director.ts | Edge-native assignment, resume, region, and invite-move behavior |
| cloud/…/director-config.ts | Director hostname validation, front-door country routing, and region location hints |
| cloud/…/director-assign.ts | Native director assignment epochs, account changes, and move decisions |
| cloud/…/host-relay-director.ts | Host-object assignment answers, credential checks, and user-index retry handoff |
| cloud/…/admin.ts | Signed edge control operations, key rotation, body binding, and replay rules |
| cloud/…/user-hosts.ts | Per-user host index used to find edge-owned data during account erasure |
| cloud/…/user-index.ts | Pending index updates and retry convergence after account or binding changes |
| cloud/…/host-relay.ts | Edge host lifecycle, signed director operations, drain, export, and erase behavior |
| cloud/…/host-relay-base.ts | Host session state, object alarms, rebind grace, splice ownership, and cleanup lifecycle |
| cloud/…/host-relay-handshake.ts | Edge host proof, assignment validation, generation, and rebind behavior |
| cloud/…/ingress.ts | Trusted ingress modes and validation of load-balancer source and client identity |
| cloud/…/proxy.ts | TLS front door, header filtering, signed relay claims, WebSocket proxying, health, and drain |
| cloud/…/relay-server.ts | Cell admission, WebSocket upgrades, connection limits, readiness, and runtime evidence |
| cloud/…/host-session-registry.ts | Host control sessions, phone admission, memory resume, and live splice ownership |
| cloud/…/index.ts | Relay startup, role ownership, control flags, background maintenance, heartbeat, and shutdown lifecycle |
| cloud/…/assignment-store.ts | Durable placement, cell state, admission modes, capacity evidence, migration continuity, and regional rehome state |
| cloud/…/reserve-assignment.ts | Reserve placement, regional spill, reconnect placement, and cell-seat reconciliation |
| cloud/…/regional-rehome-worker.ts | Director-driven idle regional rehome dispatch and safety gating |
| cloud/…/relay-observability.ts | Runtime, readiness, admission, database, connection, and regional safety evidence |
| cloud/…/connector-runtime.ts | Assembly of the shared connector core, durable inbox, and source edges |
| cloud/…/connector-core.ts | Shared conversation routing, machine events, replies, status, and source outboxes |
| cloud/…/connector-inbox.ts | Durable machine delivery, ordering, leases, expiry, and recovery |
| cloud/…/machine-routes.ts | Machine hello, held polling, acknowledgements, events, and replies |
| cloud/…/contract.test.ts | Forward-compatible machine capabilities and bounded hello and event data |
| cloud/…/slack-edge.ts | Slack signature admission, workspace scope, ingress, interactions, and adapter boundary |
| cloud/…/discord-edge.ts | Discord message routing, signed interactions, guild scope, and adapter boundary |
| cloud/…/discord-gateway-worker.ts | Single-leader Discord session, checkpoint commit, replay, and shutdown behavior |
| cloud/…/slack-post-sender.ts | Slack outbox delivery and reconciliation of unclear sends |
| cloud/…/discord-post-sender.ts | Discord outbox delivery, thread creation, and provider result handling |
| cloud/…/push-server.ts | Push authentication, rate limits, device ownership, readiness, queue acceptance, and request drain |
| cloud/…/durable-push-store.ts | Durable event identity, device ordering, expiry, claims, and retries |
| cloud/…/durable-push-worker.ts | Provider handoff, lease renewal, retry timing, and shutdown recovery |
| cloud/…/durable-push-worker.test.ts | Guarantees device recheck before retry, stop-aware claims, and progress across devices |
| cloud/…/send-messages.ts | Bounded notification payload and registration limits |
| cloud/…/package.json | Protected relay mutation service boundary |
cloud/apps/relay/Dockerfile | Relay production image boundary and runtime packaging |
cloud/apps/connector/Dockerfile | Connector production image boundary and runtime packaging |
cloud/apps/push/Dockerfile | Push production image boundary and runtime packaging |
cloud/apps/relay-ingress/Dockerfile | Ingress proxy image boundary, non-root runtime, and health listener |
| cloud/…/wrangler.jsonc | Edge object bindings, migrations, local runtime shape, and dark director configuration |
| cloud/…/director-contract.test.ts | End-to-end compatibility between edge-native director answers and shipped clients |
| cloud/…/director-dark.test.ts | Guarantee that optional director routes do not change normal edge behavior when disabled |
| cloud/…/user-index-write.test.ts | Guarantees user-index convergence after assignment, account switches, and transient failure |
| cloud/…/relay-security-2026-10-21.md | Current security review scope, findings, and rollout conditions |