Introducing a new way to review your PRs.

Read the announcement

stablyai / orca

Orca lets you run a fleet of bots with your own subscription (e.g. Claude Code, Grok, Agy, Codex). Available on desktop, mobile and remote runtime.

View on GitHub

34 documents · Updated Nov 20, 2026

docs / cloud-operational-deployment-controls

Cloud operational deployment controls

Cloud operational deployment controls keep Relay changes safe while the service is live. They join operator views, topology checks, pre-drain health gates, incident evidence, infrastructure rules, and ordered release workflows so a change does not proceed from stale or incomplete evidence.

Operational control plane

The Relay operations plane gives engineers one view of cloud health and guarded actions for the serving fleet. It is read-only by default. Any action that can change service state must name the target and the state that the operator reviewed.

Operations dashboard

The operations dashboard combines resource state, Relay metrics, alert policy status, workflow history, capacity, and cost into an environment snapshot. It covers production and staging from the durable Terraform cell list, so the view follows the reviewed topology instead of a hand-maintained list. A failed refresh uses the last good snapshot and marks it stale rather than presenting old data as current.

References: cloud/…/dashboard-snapshot.ts• L11-95 cloud/…/dashboard-snapshot.ts• L106-166 cloud/…/environment-config.ts• L92-125

Fleet topology and capacity

Relay capacity is managed as a set of fixed cells behind a shared public entry point. Each cell has a known region, machine shape, serving state, and placement budget. Operational checks protect the relation between those cell limits, the director, Cloud SQL, and the admission state used by the service.

Resource inventory

Resource inventory answers whether the control plane, database, certificate, cells, and cell endpoints match the expected cloud shape. It checks serving health and readiness without waking a sleeping staging environment, and it reports unknown values when a cloud read fails. A missing cloud read therefore cannot look like a healthy or powered cell.

References: cloud/…/resource-inventory.ts• L61-101 cloud/…/resource-inventory.ts• L272-314 cloud/…/resource-inventory.ts• L353-437

Infrastructure topology

The infrastructure model keeps each GCE cell at zero or one instance, uses one non-public backend per cell, and gives long-lived WebSockets a reviewed timeout and drain period. Production keeps the Relay director at a fixed Cloud Run size while serving cells run on fixed-one GCE groups. Retired cells stay fenced and at zero instances.

The production topology has a root US region and an added Asia region. The Asia-shaped fleet uses a higher request and connection budget, while two cells with that shape remain in the US region. A third Asia cell is a migration-only spare. Cells use health and readiness checks, durable region placement, immutable image inputs, and no public VM address. These rules make a cell replacement a controlled topology change rather than an uncontrolled scale event.

References: cloud/…/relay-gce-cells.tf• L13-27 cloud/…/relay-gce-cells.tf• L65-117 cloud/…/relay-gce-cells.tf• L165-199 cloud/…/relay-gce-cells.tf• L267-375 cloud/…/production.tfvars• L18-46 cloud/…/production.tfvars• L354-467

Admission selector

An admission selector is the durable map that places every configured cell in exactly one of three states: existing-only, migration-only, or general. Its generation ties an expected topology to the evidence, so a changed membership set cannot reuse an old green result.

This selector is the bridge between rollout intent and live director behavior. Monitoring freezes on any generation or membership mismatch. Recovery and capacity-transition policies allow only their named and narrowly scoped migration exception.

References: cloud/…/incident-selector.ts• L3-77 cloud/…/incident-monitor.ts• L440-587 cloud/…/incident-monitor.ts• L589-648

Pre-drain safeguards

Draining a cell moves live hosts through the director and can create a reconnect surge. The pre-drain controls therefore require both a sustained healthy fleet window and checks that cover failures a single point-in-time sample cannot show.

Incident safety gate

The incident monitor is a fail-closed read-only gate for ordinary mutations and incident watches. It compares the expected admission selector with live state, checks endpoint probes, Cloud Monitoring, Relay logs, director admin data, cell runtime state, migration state, connection use, and queue pressure, then records a green or frozen result.

Missing or stale data is not treated as healthy. A small number of consecutive freshness gaps can be tolerated and recorded, but a sustained gap, a threshold breach, a selector mismatch, or an unsafe migration state freezes the run. A pre-drain dry run must complete a fresh 15-minute window within its lineage limit.

References: cloud/…/incident-monitor.ts• L8-32 cloud/…/incident-monitor.ts• L589-654 cloud/…/incident-monitor.ts• L841-1071

Drain disconnect measurement

Drain disconnect measurement turns exported cell and director logs into a read-only view of reconnect gaps. It matches the first drain close for each host with its first reconnect grant outside the source cell, then reports coverage, dropped phone sessions, cutoff gaps, and whether hosts that crossed regions returned.

The measurement uses the controls held at drain start as its denominator. This exposes incomplete log exports and unresolved reconnects instead of allowing a small observed sample to look like the full drain result.

References: cloud/…/measure-relay-drain-disconnect-gap.mjs• L4-12 cloud/…/measure-relay-drain-disconnect-gap.mjs• L67-131

Incident monitoring

Incident monitoring turns cloud and Relay signals into private, durable evidence for operators and deployment gates. It supports both a short pre-drain verdict and a longer incident watch without collecting tokens, request bodies, user IDs, or host IDs.

Monitoring windows

The production monitor runs a 15-minute dry-run for deployment authority and a longer watch for incident response. A watch can continue from a prior attempt when the restored state has the same incident, environment, selector, policy, commit, and timing values. The monitor refreshes identity before the next segment so an expiring token does not weaken the safety boundary.

References: cloud/…/incident-monitor-cli.ts• L283-369 cloud/…/incident-monitor-cli.ts• L382-421

Telemetry sources

Each sample joins four views of the system: active endpoint probes, Google Cloud metrics, Relay runtime and log metrics, and authenticated director status. The sources cover service health, Cloud SQL pressure, director load and errors, pool waits, retry exhaustion, cell heartbeats, admission, and migration state.

The monitor uses fixed aggregation and freshness rules. This makes a reading comparable across runs and prevents a workflow input from weakening the safety bar for one deployment.

References: cloud/…/incident-monitor-sources.ts• L63-148 cloud/…/incident-monitor-sources.ts• L659-730 cloud/…/monitoring-snapshot.ts• L7-100

Durable evidence

The monitor saves state, checkpoint records, and a secret-safe Markdown summary. State writes are atomic and private. Checkpoints are idempotent, so a restart does not create a second record for the same window and time.

Deployment workflows restore the exact artifact by run and attempt, check its commit and content, require a completed green dry run, and mark it consumed before mutation. A shared rollout lease prevents concurrent production operations from using the same safety boundary.

References: cloud/…/incident-monitor-cli.ts• L283-369 cloud/…/incident-monitor-cli.ts• L382-421

Alert response

Cloud Monitoring alerts provide ongoing visibility while the monitor provides a workflow gate. Paging alerts cover customer-impacting or fleet-wide pressure such as assignment errors, retry exhaustion, connection headroom, cell exits, lock contention, Cloud SQL pressure, and region placement problems.

Region alerts compare where clients ask to connect with where the director places them. Lock alerts distinguish a cell hold that calls for pausing regional rehome from a director hold that does not call for a drain or restart. Disconnect and pool-herd alerts use different evidence so an operator can separate a load-balancer event from a database or cell event.

References: cloud/…/relay-observability.tf• L404-505 cloud/…/relay-observability.tf• L507-640 cloud/…/relay-observability.tf• L819-949 cloud/…/relay-observability.tf• L940-1065 cloud/…/relay-observability.tf• L1078-1117 cloud/…/relay-reconnect-herd-alerts.test.mjs• L24-45

Deployment workflow controls

Deployment workflows separate source review, build qualification, artifact publication, and service mutation. They pin the source identity and keep incomplete artifacts or failed checks away from users and production systems.

Cloud validation

Cloud validation is the first boundary for changes to the cloud workspace and its deployment controls. It scans reachable history and the current cloud snapshot for secrets, audits dependencies, builds and type-checks the cloud workspace, tests it with PostgreSQL, and checks the Relay and Push runtime images for unsafe or fixable vulnerabilities. It also confirms that each image runs as a non-root Node process without package-manager tools in the image.

References: .github/…/cloud-verify.yml• L1-27 .github/…/cloud-verify.yml• L32-110 .github/…/cloud-verify.yml• L112-167 .github/…/cloud-verify.yml• L170-207

Release source identity

Release publication resolves a requested ref to an immutable commit before it creates version or artifact state. A release cut checks the version order and source history, and downstream platform builds use the selected tag or commit rather than whichever branch happens to move during the run.

This identity check matters because a release can span several jobs and platforms. The same source identity is carried into the server template, desktop artifacts, and channel-specific builds, so the published package describes the code that was qualified.

References: .github/…/release-cut.yml• L106-165 .github/…/release-cut.yml• L725-811 .github/…/release-mac-build.yml• L63-75 .github/…/release-mac-build.yml• L180-215

Artifact publication

Release publication keeps a release invisible until its required update artifacts exist and pass their manifest checks. macOS artifacts are signed and notarized before publication. Dev-channel Windows builds validate the exact source, channel, tag, version, and update manifest before they publish or stage an artifact.

Hourly, daily, and ad hoc channels use separate release stores so experimental builds do not displace stable release entries. Their freshness and retention rules also prevent an unchanged source from creating needless builds or a failed draft from becoming a user-visible release.

References: .github/…/hourly-mac-build.yml• L82-117 .github/…/hourly-mac-build.yml• L312-426 .github/…/daily-mac-build.yml• L145-178 .github/…/daily-mac-build.yml• L331-480 .github/…/adhoc-mac-build.yml• L91-155 .github/…/adhoc-mac-build.yml• L680-717 .github/…/dev-channel-win-build.yml• L134-184 .github/…/dev-channel-win-build.yml• L350-444

Deployment evidence

Release checks use different evidence levels for different risks. Required platform golden checks block a release when they cover the supported release path, while broader E2E and performance runs preserve evidence for diagnosis without delaying publication. Failed runs retain traces or bounded reports so an operator can inspect the exact source and scenario that produced the result.

References: .github/…/release-cut.yml• L813-825 .github/…/release-cut.yml• L862-988 .github/…/e2e.yml• L196-320 .github/…/terminal-perf.yml• L138-196

Change checklist

When these files or sections are changed, remember to consider these:

  • Keep the admission selector states, expected topology, monitor generation, and recovery exceptions aligned. A valid-looking generation with a different membership set must remain unable to reuse evidence.
  • Keep monitor thresholds, freshness rules, pre-drain hard rules, and the live safety check on the same evaluator and tolerance model.
  • Keep drain log markers, source-cell boundaries, close coverage, reconnect matching, and region maps aligned when changing Relay logs or drain behavior.
  • Keep the Terraform cell list, Relay region contract, monitoring metric fields, alert queries, and operations inventory in step when adding or moving a region.
  • Keep capacity limits, Cloud SQL operating budgets, and cell shape definitions consistent across infrastructure and admission policy.
  • Keep image digest, protocol, selector generation, rollback digest, cell class, and drain pace bound to one rollout identity. Do not allow evidence from one rollout to authorize another.
  • Keep release source SHAs, server-template artifacts, platform outputs, update manifests, and channel tags tied to the same build identity.
  • Keep draft-release cleanup and publication checks fail-closed. A missing manifest must not create a release that the updater can see but cannot install.
  • Keep cloud secret scans, dependency audits, image checks, and cloud tests in the validation path when cloud workflow triggers or workspace boundaries change.

Important failure modes

TriggerConsequenceGuard
Live selector generation or membership differs from the requested stateEvidence can describe a different fleet shape than the one being changedExact selector comparison in monitoring and mutation preflight
Cloud telemetry is missing or too old for the allowed continuity budgetThe safety window resets or freezes instead of authorizing a drainSource freshness checks, bounded tolerated gaps, and a lineage deadline
Drain logs omit close lines or reconnect grantsDisconnect gaps and session loss can be understatedClose coverage against controls at drain start and unresolved-gap counts
A cell reports the wrong region, image, or capacity shapeThe director can admit a resource that does not match the reviewed topologyRuntime identity checks before admission transition
A lock alert comes from a director hold rather than a cell holdAn unnecessary regional pause or cell drain can increase riskPaired cell and director hold policies with distinct operator actions
A disconnect burst comes from the load balancer rather than the cellA drain or restart can worsen a fleet-wide stream failureLoad-balancer evidence and status details before cell action
A dry-run artifact comes from the wrong commit or attemptOld health evidence can authorize a new mutationExact artifact provenance, run and attempt selection, and single-use markers
A release is published without its updater manifestThe client sees a release but cannot install itDraft publication, required manifest checks, and final asset verification
A dev-channel build uses a different source, tag, or versionArtifacts can land in the wrong channel or describe another commitFull source reachability and channel/tag/version validation
Cloud validation skips a secret scan, image check, or cloud testAn unsafe credential or broken runtime can reach deploymentCloud Verify security, build, container, and test jobs

Key files

FileWhy to read it
cloud/…/dashboard-snapshot.tsShows how the operator view joins monitoring, resources, workflows, warnings, and stale-state handling.
cloud/…/incident-monitor.tsDefines the shared safety evaluation, continuity behavior, durable window model, and dry-run pass rules.
cloud/…/incident-monitor-sources.tsShows how cloud, Relay, endpoint, and director readings become one incident sample.
cloud/…/incident-monitor-cli.tsShows how monitor state and private evidence are saved and restored.
cloud/…/incident-selector.tsDefines the admission states and exact selector contract used by monitoring and rollout gates.
cloud/…/measure-relay-drain-disconnect-gap.mjsMeasures reconnect gaps and log coverage after a cell drain without changing cloud state.
.github/…/cloud-verify.ymlDefines the cloud security, build, container, and test validation boundary.
.github/…/release-cut.ymlDefines immutable release source resolution, version safety, tagging, and release qualification.
.github/…/release-mac-build.ymlShows how a release source becomes signed macOS artifacts while the release remains protected.
.github/…/dev-channel-win-build.ymlDefines source and channel checks plus Windows artifact staging and manifest verification.
.github/…/hourly-mac-build.ymlShows freshness, draft publication, update-manifest checks, and retention for fast dev builds.
.github/…/daily-mac-build.ymlShows the lower-frequency dev release path and its post-publication evidence flow.
cloud/…/relay-gce-cells.tfDefines the reviewed shape and network rules for fixed-one Relay cells.
cloud/…/production.tfvarsDefines production director settings, cell topology, regional budgets, and fenced capacity.
cloud/…/relay-observability.tfDefines runtime metrics, disconnect and pool-herd signals, alert policies, and operator response guidance.
Cloud servicesParent page for the cloud service boundary and its operational systems.
Cloud relay admission and continuityRelated page for durable admission state and continuity behavior.
Push notification deliveryRelated page for the adjacent cloud delivery path.