Cloud operational deployment controls
Cloud operational deployment controls keep Relay changes safe while the service is live. They join operator views, topology checks, pre-drain health gates, incident evidence, infrastructure rules, and ordered release workflows so a change does not proceed from stale or incomplete evidence.
Operational control plane
The Relay operations plane gives engineers one view of cloud health and guarded actions for the serving fleet. It is read-only by default. Any action that can change service state must name the target and the state that the operator reviewed.
Operations dashboard
The operations dashboard combines resource state, Relay metrics, alert policy status, workflow history, capacity, and cost into an environment snapshot. It covers production and staging from the durable Terraform cell list, so the view follows the reviewed topology instead of a hand-maintained list. A failed refresh uses the last good snapshot and marks it stale rather than presenting old data as current.
References: cloud/…/dashboard-snapshot.ts• L11-95 cloud/…/dashboard-snapshot.ts• L106-166 cloud/…/environment-config.ts• L92-125
Fleet topology and capacity
Relay capacity is managed as a set of fixed cells behind a shared public entry point. Each cell has a known region, machine shape, serving state, and placement budget. Operational checks protect the relation between those cell limits, the director, Cloud SQL, and the admission state used by the service.
Resource inventory
Resource inventory answers whether the control plane, database, certificate, cells, and cell endpoints match the expected cloud shape. It checks serving health and readiness without waking a sleeping staging environment, and it reports unknown values when a cloud read fails. A missing cloud read therefore cannot look like a healthy or powered cell.
References: cloud/…/resource-inventory.ts• L61-101 cloud/…/resource-inventory.ts• L272-314 cloud/…/resource-inventory.ts• L353-437
Infrastructure topology
The infrastructure model keeps each GCE cell at zero or one instance, uses one non-public backend per cell, and gives long-lived WebSockets a reviewed timeout and drain period. Production keeps the Relay director at a fixed Cloud Run size while serving cells run on fixed-one GCE groups. Retired cells stay fenced and at zero instances.
The production topology has a root US region and an added Asia region. The Asia-shaped fleet uses a higher request and connection budget, while two cells with that shape remain in the US region. A third Asia cell is a migration-only spare. Cells use health and readiness checks, durable region placement, immutable image inputs, and no public VM address. These rules make a cell replacement a controlled topology change rather than an uncontrolled scale event.
References: cloud/…/relay-gce-cells.tf• L13-27 cloud/…/relay-gce-cells.tf• L65-117 cloud/…/relay-gce-cells.tf• L165-199 cloud/…/relay-gce-cells.tf• L267-375 cloud/…/production.tfvars• L18-46 cloud/…/production.tfvars• L354-467
Admission selector
An admission selector is the durable map that places every configured cell in exactly one of three states: existing-only, migration-only, or general. Its generation ties an expected topology to the evidence, so a changed membership set cannot reuse an old green result.
This selector is the bridge between rollout intent and live director behavior. Monitoring freezes on any generation or membership mismatch. Recovery and capacity-transition policies allow only their named and narrowly scoped migration exception.
References: cloud/…/incident-selector.ts• L3-77 cloud/…/incident-monitor.ts• L440-587 cloud/…/incident-monitor.ts• L589-648
Pre-drain safeguards
Draining a cell moves live hosts through the director and can create a reconnect surge. The pre-drain controls therefore require both a sustained healthy fleet window and checks that cover failures a single point-in-time sample cannot show.
Incident safety gate
The incident monitor is a fail-closed read-only gate for ordinary mutations and incident watches. It compares the expected admission selector with live state, checks endpoint probes, Cloud Monitoring, Relay logs, director admin data, cell runtime state, migration state, connection use, and queue pressure, then records a green or frozen result.
Missing or stale data is not treated as healthy. A small number of consecutive freshness gaps can be tolerated and recorded, but a sustained gap, a threshold breach, a selector mismatch, or an unsafe migration state freezes the run. A pre-drain dry run must complete a fresh 15-minute window within its lineage limit.
References: cloud/…/incident-monitor.ts• L8-32 cloud/…/incident-monitor.ts• L589-654 cloud/…/incident-monitor.ts• L841-1071
Drain disconnect measurement
Drain disconnect measurement turns exported cell and director logs into a read-only view of reconnect gaps. It matches the first drain close for each host with its first reconnect grant outside the source cell, then reports coverage, dropped phone sessions, cutoff gaps, and whether hosts that crossed regions returned.
The measurement uses the controls held at drain start as its denominator. This exposes incomplete log exports and unresolved reconnects instead of allowing a small observed sample to look like the full drain result.
References: cloud/…/measure-relay-drain-disconnect-gap.mjs• L4-12 cloud/…/measure-relay-drain-disconnect-gap.mjs• L67-131
Incident monitoring
Incident monitoring turns cloud and Relay signals into private, durable evidence for operators and deployment gates. It supports both a short pre-drain verdict and a longer incident watch without collecting tokens, request bodies, user IDs, or host IDs.
Monitoring windows
The production monitor runs a 15-minute dry-run for deployment authority and a longer watch for incident response. A watch can continue from a prior attempt when the restored state has the same incident, environment, selector, policy, commit, and timing values. The monitor refreshes identity before the next segment so an expiring token does not weaken the safety boundary.
References: cloud/…/incident-monitor-cli.ts• L283-369 cloud/…/incident-monitor-cli.ts• L382-421
Telemetry sources
Each sample joins four views of the system: active endpoint probes, Google Cloud metrics, Relay runtime and log metrics, and authenticated director status. The sources cover service health, Cloud SQL pressure, director load and errors, pool waits, retry exhaustion, cell heartbeats, admission, and migration state.
The monitor uses fixed aggregation and freshness rules. This makes a reading comparable across runs and prevents a workflow input from weakening the safety bar for one deployment.
References: cloud/…/incident-monitor-sources.ts• L63-148 cloud/…/incident-monitor-sources.ts• L659-730 cloud/…/monitoring-snapshot.ts• L7-100
Durable evidence
The monitor saves state, checkpoint records, and a secret-safe Markdown summary. State writes are atomic and private. Checkpoints are idempotent, so a restart does not create a second record for the same window and time.
Deployment workflows restore the exact artifact by run and attempt, check its commit and content, require a completed green dry run, and mark it consumed before mutation. A shared rollout lease prevents concurrent production operations from using the same safety boundary.
References: cloud/…/incident-monitor-cli.ts• L283-369 cloud/…/incident-monitor-cli.ts• L382-421
Alert response
Cloud Monitoring alerts provide ongoing visibility while the monitor provides a workflow gate. Paging alerts cover customer-impacting or fleet-wide pressure such as assignment errors, retry exhaustion, connection headroom, cell exits, lock contention, Cloud SQL pressure, and region placement problems.
Region alerts compare where clients ask to connect with where the director places them. Lock alerts distinguish a cell hold that calls for pausing regional rehome from a director hold that does not call for a drain or restart. Disconnect and pool-herd alerts use different evidence so an operator can separate a load-balancer event from a database or cell event.
References: cloud/…/relay-observability.tf• L404-505 cloud/…/relay-observability.tf• L507-640 cloud/…/relay-observability.tf• L819-949 cloud/…/relay-observability.tf• L940-1065 cloud/…/relay-observability.tf• L1078-1117 cloud/…/relay-reconnect-herd-alerts.test.mjs• L24-45
Deployment workflow controls
Deployment workflows separate source review, build qualification, artifact publication, and service mutation. They pin the source identity and keep incomplete artifacts or failed checks away from users and production systems.
Cloud validation
Cloud validation is the first boundary for changes to the cloud workspace and its deployment controls. It scans reachable history and the current cloud snapshot for secrets, audits dependencies, builds and type-checks the cloud workspace, tests it with PostgreSQL, and checks the Relay and Push runtime images for unsafe or fixable vulnerabilities. It also confirms that each image runs as a non-root Node process without package-manager tools in the image.
References: .github/…/cloud-verify.yml• L1-27 .github/…/cloud-verify.yml• L32-110 .github/…/cloud-verify.yml• L112-167 .github/…/cloud-verify.yml• L170-207
Release source identity
Release publication resolves a requested ref to an immutable commit before it creates version or artifact state. A release cut checks the version order and source history, and downstream platform builds use the selected tag or commit rather than whichever branch happens to move during the run.
This identity check matters because a release can span several jobs and platforms. The same source identity is carried into the server template, desktop artifacts, and channel-specific builds, so the published package describes the code that was qualified.
References: .github/…/release-cut.yml• L106-165 .github/…/release-cut.yml• L725-811 .github/…/release-mac-build.yml• L63-75 .github/…/release-mac-build.yml• L180-215
Artifact publication
Release publication keeps a release invisible until its required update artifacts exist and pass their manifest checks. macOS artifacts are signed and notarized before publication. Dev-channel Windows builds validate the exact source, channel, tag, version, and update manifest before they publish or stage an artifact.
Hourly, daily, and ad hoc channels use separate release stores so experimental builds do not displace stable release entries. Their freshness and retention rules also prevent an unchanged source from creating needless builds or a failed draft from becoming a user-visible release.
References: .github/…/hourly-mac-build.yml• L82-117 .github/…/hourly-mac-build.yml• L312-426 .github/…/daily-mac-build.yml• L145-178 .github/…/daily-mac-build.yml• L331-480 .github/…/adhoc-mac-build.yml• L91-155 .github/…/adhoc-mac-build.yml• L680-717 .github/…/dev-channel-win-build.yml• L134-184 .github/…/dev-channel-win-build.yml• L350-444
Deployment evidence
Release checks use different evidence levels for different risks. Required platform golden checks block a release when they cover the supported release path, while broader E2E and performance runs preserve evidence for diagnosis without delaying publication. Failed runs retain traces or bounded reports so an operator can inspect the exact source and scenario that produced the result.
References: .github/…/release-cut.yml• L813-825 .github/…/release-cut.yml• L862-988 .github/…/e2e.yml• L196-320 .github/…/terminal-perf.yml• L138-196
Change checklist
When these files or sections are changed, remember to consider these:
- Keep the admission selector states, expected topology, monitor generation, and recovery exceptions aligned. A valid-looking generation with a different membership set must remain unable to reuse evidence.
- Keep monitor thresholds, freshness rules, pre-drain hard rules, and the live safety check on the same evaluator and tolerance model.
- Keep drain log markers, source-cell boundaries, close coverage, reconnect matching, and region maps aligned when changing Relay logs or drain behavior.
- Keep the Terraform cell list, Relay region contract, monitoring metric fields, alert queries, and operations inventory in step when adding or moving a region.
- Keep capacity limits, Cloud SQL operating budgets, and cell shape definitions consistent across infrastructure and admission policy.
- Keep image digest, protocol, selector generation, rollback digest, cell class, and drain pace bound to one rollout identity. Do not allow evidence from one rollout to authorize another.
- Keep release source SHAs, server-template artifacts, platform outputs, update manifests, and channel tags tied to the same build identity.
- Keep draft-release cleanup and publication checks fail-closed. A missing manifest must not create a release that the updater can see but cannot install.
- Keep cloud secret scans, dependency audits, image checks, and cloud tests in the validation path when cloud workflow triggers or workspace boundaries change.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| Live selector generation or membership differs from the requested state | Evidence can describe a different fleet shape than the one being changed | Exact selector comparison in monitoring and mutation preflight |
| Cloud telemetry is missing or too old for the allowed continuity budget | The safety window resets or freezes instead of authorizing a drain | Source freshness checks, bounded tolerated gaps, and a lineage deadline |
| Drain logs omit close lines or reconnect grants | Disconnect gaps and session loss can be understated | Close coverage against controls at drain start and unresolved-gap counts |
| A cell reports the wrong region, image, or capacity shape | The director can admit a resource that does not match the reviewed topology | Runtime identity checks before admission transition |
| A lock alert comes from a director hold rather than a cell hold | An unnecessary regional pause or cell drain can increase risk | Paired cell and director hold policies with distinct operator actions |
| A disconnect burst comes from the load balancer rather than the cell | A drain or restart can worsen a fleet-wide stream failure | Load-balancer evidence and status details before cell action |
| A dry-run artifact comes from the wrong commit or attempt | Old health evidence can authorize a new mutation | Exact artifact provenance, run and attempt selection, and single-use markers |
| A release is published without its updater manifest | The client sees a release but cannot install it | Draft publication, required manifest checks, and final asset verification |
| A dev-channel build uses a different source, tag, or version | Artifacts can land in the wrong channel or describe another commit | Full source reachability and channel/tag/version validation |
| Cloud validation skips a secret scan, image check, or cloud test | An unsafe credential or broken runtime can reach deployment | Cloud Verify security, build, container, and test jobs |
Key files
| File | Why to read it |
|---|---|
| cloud/…/dashboard-snapshot.ts | Shows how the operator view joins monitoring, resources, workflows, warnings, and stale-state handling. |
| cloud/…/incident-monitor.ts | Defines the shared safety evaluation, continuity behavior, durable window model, and dry-run pass rules. |
| cloud/…/incident-monitor-sources.ts | Shows how cloud, Relay, endpoint, and director readings become one incident sample. |
| cloud/…/incident-monitor-cli.ts | Shows how monitor state and private evidence are saved and restored. |
| cloud/…/incident-selector.ts | Defines the admission states and exact selector contract used by monitoring and rollout gates. |
| cloud/…/measure-relay-drain-disconnect-gap.mjs | Measures reconnect gaps and log coverage after a cell drain without changing cloud state. |
| .github/…/cloud-verify.yml | Defines the cloud security, build, container, and test validation boundary. |
| .github/…/release-cut.yml | Defines immutable release source resolution, version safety, tagging, and release qualification. |
| .github/…/release-mac-build.yml | Shows how a release source becomes signed macOS artifacts while the release remains protected. |
| .github/…/dev-channel-win-build.yml | Defines source and channel checks plus Windows artifact staging and manifest verification. |
| .github/…/hourly-mac-build.yml | Shows freshness, draft publication, update-manifest checks, and retention for fast dev builds. |
| .github/…/daily-mac-build.yml | Shows the lower-frequency dev release path and its post-publication evidence flow. |
| cloud/…/relay-gce-cells.tf | Defines the reviewed shape and network rules for fixed-one Relay cells. |
| cloud/…/production.tfvars | Defines production director settings, cell topology, regional budgets, and fenced capacity. |
| cloud/…/relay-observability.tf | Defines runtime metrics, disconnect and pool-herd signals, alert policies, and operator response guidance. |
| Cloud services | Parent page for the cloud service boundary and its operational systems. |
| Cloud relay admission and continuity | Related page for durable admission state and continuity behavior. |
| Push notification delivery | Related page for the adjacent cloud delivery path. |