Introducing a new way to review your PRs.

Read the announcement

paperclipai / paperclip

The open-source app everyone uses to manage agents at work

View on GitHub

30 documents · Updated Nov 20, 2026

docs / application-platform-operations

Application platform operations

Paperclip is a Node.js application platform that joins an HTTP API, browser UI, PostgreSQL, file storage, background work, and WebSocket services in one running instance. The platform keeps company data, agent work, external tools, and operational state under one control plane. For product workflows, see Agent work management, Organization work management, Automation and control, Security and governance, and Extensibility platform.

Server composition

The server process is the main control point for requests, data access, scheduled work, provider workers, storage, realtime connections, and cleanup. Startup enables tracing and error monitoring before it opens database connections or creates the HTTP server. It then loads configuration, selects the database, creates the application, opens the listener, and recovers work that may have survived an earlier process (server/…/index.ts• L312-343, server/…/index.ts• L527-766).

The listener opens before startup recovery finishes. This lets surviving runtime clients reconnect while recovery repairs durable state. Startup tracks recovery and background work separately from request readiness. It waits for external adapters and managed environments before normal work resumes. Managed environments are applied only after their bundled plugin workers are ready. Fresh queued work starts after adapter readiness and remains queued if that startup dispatch fails (server/…/index.ts• L1045-1149, server/…/index.ts• L1280-1307, server/…/index.ts• L1355-1383). It closes every database client it created on failed startup or orderly shutdown (server/…/index.ts• L758-767, server/…/index.ts• L2243-2249).

One HTTP server serves the API and, when enabled, the UI. The same server accepts WebSocket upgrades and hosts background services such as heartbeats, delivery queues, retention sweeps, provider reconciliation, backups, recovery, and cleanup. The process also wakes durable work when committed changes create a new deadline or delivery need. Warm-standby and task-drain admission checks can stop new work before maintenance or sleep (server/…/app.ts• L321-362, server/…/index.ts• L1295-1364, server/…/index.ts• L1860-1919).

Each server boot can register a database-backed controller lease. The lease gives work ownership a boot identity and a fencing point, so a replaced process cannot keep acting only because its host or process ID was reused. During a deployment overlap, dispatch markers let the newer boot take responsibility for new run starts while the older boot remains available for active work (server/…/index.ts• L1146-1151, server/…/index.ts• L1317-1333, packages/…/controller_instances.ts• L3-34).

The adapter registry makes built-in and external agent adapters available to the API. Startup waits for external adapters before normal queued work resumes. Authorized operators can install, disable, replace, reload, or remove adapters when deployment policy permits (server/…/adapters.ts• L267-347, server/…/adapters.ts• L492-681, server/…/index.ts• L2098-2115).

Ephemeral deployment mode adds a handoff boundary between overlapping web servers. The shared database and object storage carry durable state. Deployment notifications and durable delivery work route owner commands to the process that still controls an execution. During shutdown, the old server stops new admission but keeps handoff and response channels alive until active work drains (server/…/index.ts• L1035-1073, server/…/index.ts• L2190-2228).

Automatic recovery also has a company-level safety brake. When automatic runs fail before they start useful provider work, the brake can pause new automatic starts, test recovery with limited probe runs, and release work in steps. Person-requested work bypasses this protection so a broad recovery problem does not block direct human action (server/…/automatic-run-brake.ts• L8-31, server/…/automatic-run-brake.ts• L325-375, server/…/automatic-run-brake.ts• L408-450, packages/…/automatic_run_brakes.ts• L4-17).

References: server/…/index.ts• L312-343 server/…/index.ts• L527-766 server/…/index.ts• L1045-1151 server/…/index.ts• L1280-1383 server/…/index.ts• L2098-2131 server/…/automatic-run-brake.ts• L8-31 packages/…/automatic_run_brakes.ts• L4-17

HTTP routing

HTTP routing places request-wide checks around product routes. The outer layer handles request size, compression, logs, host checks, identity resolution, and short-lived runtime connection intents. The /api router then applies mutation guards and sends requests to the route group for the product area (server/…/app.ts• L343-432).

Connection management is a shared authorization boundary for AI, chat, email, and tool routes. Its guard resolves referenced resources before it checks the requested action. It binds those resources to one company, derives the responsible user from authentication, limits selector size, and can reauthorize a saved request after a human completes setup (server/…/connection-management-authz.ts• L20-145, server/…/connection-management-authz.ts• L390-472).

The API has several trust boundaries. Browser and board clients use a session or board API key. Agents use agent credentials. Runtime tools use a short-lived token bound to a company, agent, and run. Public assets, health data, invite flows, selected challenge flows, MCP gateway entry points, and workload identity discovery are public only where their own validation is complete. Workload identity status and key rotation remain board-only operations (server/…/openapi.ts• L1349-1415, server/…/openapi.ts• L1423-1430, server/…/openapi.ts• L1472-1475, server/…/openapi.ts• L1723-1743).

Connection setup can pause at a human boundary. An agent request does not supply a user identity in its body. The server records the authenticated responsible user and returns a saved setup action when provider sign-in, consent, or credential entry is required. The saved action can continue only after the same user completes the required step (server/…/connection-management-authz.ts• L20-32, server/…/connection-management-authz.ts• L424-430).

OpenAPI is the public description of this boundary. The registry turns route schemas into request, response, and parameter descriptions. Document fixups then assign operation security levels, authorization metadata, forbidden responses, and status-code corrections. MCP Apps views are available to board users through task-scoped routes. Connected-app MCP Skills are reviewed by board users and their approved files are available only to agents in an active run (server/…/openapi.ts• L1323-1358, server/…/openapi.ts• L1729-1887, server/…/openapi.ts• L1445-1459).

The UI is served in two modes. A built UI uses static assets and a single-page fallback. Development mode uses Vite middleware while API traffic stays on the application server. Managed HTTPS development also keeps a companion HMR listener because the runtime supervisor expects both listeners (server/…/app.ts• L698-836).

References: server/…/app.ts• L343-432 server/…/connection-management-authz.ts• L20-145 server/…/openapi.ts• L1323-1358 server/…/openapi.ts• L1349-1475 server/…/openapi.ts• L1729-1887

Database lifecycle

PostgreSQL is the system of record for application state. The platform can use an external PostgreSQL service or a persistent embedded PostgreSQL cluster. Both paths create the same application database and use the shared database layer (server/…/index.ts• L527-766, packages/…/client.ts• L118-122).

Database sourceOperational role
DATABASE_URL or another configured connection stringUse external PostgreSQL.
No connection stringStart or reuse embedded PostgreSQL in the instance data directory.

Authenticated public deployments require external PostgreSQL. This prevents an internet-facing instance from silently using an embedded database. Embedded PostgreSQL persists across restarts, checks that a reused process owns the expected data directory, and attempts controlled recovery after an unexpected exit. Exhausted recovery stops the application instead of serving against an unknown database state (server/…/index.ts• L445-465, server/…/index.ts• L604-731).

The database also holds durable coordination state. Wake requests, run events, cost records, budget work, runtime leases, controller boots, MCP authority, provider task handles, issue recovery checkpoints, content deletion work, harness backups, and provider artifacts let later workers recover work that cannot depend on one process remaining alive (packages/…/agent_wakeup_requests.ts• L15-118, packages/…/heartbeat_runs.ts• L31-133, packages/…/controller_instances.ts• L13-34, packages/…/issue_recovery_full_passes.ts• L5-35, packages/…/tool_access.ts• L896-970, packages/…/tool_connection_content_deletions.ts• L6-31).

References: server/…/index.ts• L445-465 server/…/index.ts• L527-766 packages/…/client.ts• L118-122 packages/…/issue_recovery_full_passes.ts• L5-35 packages/…/tool_access.ts• L896-970 packages/…/tool_connection_content_deletions.ts• L6-31

Database migrations

Migrations keep the deployed schema aligned with the application version. Startup inspects the migration journal before it creates the normal database client. It can repair safe journal drift and apply pending changes when policy allows. If a stale schema needs approval and approval is declined, startup refuses to run (server/…/index.ts• L345-434).

The migration flow moves from database selection to schema inspection and then to either client creation or a controlled stop. This diagram shows that a refused migration never reaches normal request handling.

flowchart TD
    A[Resolve database] --> B[Inspect schema and journal]
    B -->|Up to date| C[Create database client]
    B -->|Pending changes| D[Repair safe journal drift]
    D --> E{Apply allowed?}
    E -->|Yes| F[Apply migrations]
    E -->|No| G[Stop startup]
    F --> H[Verify schema]
    H --> C

Empty databases can be bootstrapped. A non-empty database with no migration journal is treated as schema drift and can proceed only after an explicit or configured migration decision. A managed-cloud first boot with no tables can report a supervised transient refusal while its external migrator prepares the schema. The command-line migration path uses the same inspection and apply rules (packages/…/client.ts• L671-798, packages/…/migrate.ts• L4-26, server/…/index.ts• L378-434).

The current migration tail extends the platform's durable operating model. Migration 0488 stores company and queue recovery-pass checkpoints and deferred issue checks. Migration 0489 adds company- and viewer-scoped link-preview expiry. Migration 0490 records retryable tool-connection content deletion work. Migration 0491 preserves access to legacy plugin tools while plugin connectors are introduced. Migrations 0492 through 0497 add responsibility areas, ownership, metrics, status reports, dashboard history, and model-summary accounting. Migration 0498 adds company ES256 workload identity keys with controlled rotation states. Migration 0499 lets a responsibility link a company skill while preserving one target per link and company-safe cleanup. The ordered journal and schema snapshots now extend through migration 0499 (packages/…/0499_numerous_whizzer.sql• L1-8, packages/…/responsibilities.ts• L126-169, packages/…/_journal.json• L3477-3490, packages/…/0499_snapshot.json• L53152-53460).

References: server/…/index.ts• L345-434 packages/…/client.ts• L671-798 packages/…/_journal.json• L3477-3490 packages/…/0499_numerous_whizzer.sql• L1-8 packages/…/responsibilities.ts• L126-169 packages/…/0499_snapshot.json• L53152-53460

Realtime events

Realtime updates use WebSockets for company-scoped live events. An upgrade request proves a board session, agent API key, or trusted cloud identity before the socket subscribes to that company. Separate channels support temporary interactive sessions for custom image setup and runner protocol communication (server/…/live-events-ws.ts• L124-225, server/…/index.ts• L1090-1113, server/…/environment-custom-image-terminal-ws.ts• L484-625).

Live events are process-local notifications. A publisher assigns an event ID and timestamp, sends the event to a company or global channel, and the WebSocket layer forwards it to connected clients. Connections receive periodic health checks and unsubscribe when they close. Durable run events use ordered sequences so clients can resume after a known point (server/…/live-events.ts• L7-53, server/…/live-events-ws.ts• L241-347, packages/…/heartbeat_run_events.ts• L18-63).

Plugin lifecycle delivery adds a durable per-resource cursor. A plugin can acknowledge a contiguous event prefix and retain separate acknowledgment rows for events after a gap. This lets a restarted plugin resume without losing a late or unacknowledged event (packages/…/plugin_lifecycle_resource_cursors.ts• L5-20, packages/…/0462_plugin_lifecycle_resource_cursors.sql• L13-31).

Agent responses that arrive while a run is active use durable wake queues. Questions, approvals, and connection intents remain queued through task drain or restart rather than becoming lost live notifications. Queued conversation messages can also resume after a run is stopped or maintenance ends. Idempotency keys prevent one response from producing duplicate continuation work (server/…/issues.ts• L6957-7078, server/…/connection-intents.ts• L174-242, packages/…/agent_wakeup_requests.ts• L76-118).

References: server/…/live-events-ws.ts• L124-347 server/…/index.ts• L1090-1113 server/…/live-events.ts• L7-53 packages/…/plugin_lifecycle_resource_cursors.ts• L5-20 packages/…/agent_wakeup_requests.ts• L15-118

File storage

File storage keeps uploaded content outside PostgreSQL while giving routes one contract for writing, reading, checking, and deleting objects. The provider can change without changing the application route contract (server/…/types.ts• L37-67, server/…/service.ts• L90-129).

ProviderOperating modelSuitable deployment
local_diskStore files below a local directory with safe replacement.Local or single-node use.
s3Store objects in an S3-compatible bucket with an optional endpoint and prefix.Cloud or multi-node use.

Object keys include the company scope, a normalized namespace, a date path, and a unique name. Reads and deletes reject keys outside the requested company. This keeps tenant ownership independent from the storage provider (server/…/service.ts• L46-69, server/…/local-disk-provider.ts• L20-27, server/…/s3-provider.ts• L66-83).

References: server/…/types.ts• L37-67 server/…/service.ts• L46-129

Deployment modes

Deployment mode defines who may use an instance. Exposure defines whether an authenticated instance is private or public. The server validates these choices with the bind address before it starts (server/…/config.ts• L161-193, server/…/config.ts• L284-301, server/…/index.ts• L807-830).

ModeAccess modelMain use
local_trustedNo login is required. The server binds to loopback and stays private.One operator on one machine.
authenticated + privateBetter Auth login is required. The server can bind to a private network.Team access over a trusted network.
authenticated + publicBetter Auth login and an explicit public URL are required.Internet-facing hosting.

Local trusted mode creates a local board identity with instance and company access. Authenticated instances use user sessions and the board-claim flow. Managed configuration can hide or fix instance, connection, plugin, and execution settings so an instance cannot override platform policy (server/…/index.ts• L467-525, server/…/index.ts• L915-947).

Task drain is an operator control that stops new task admission before maintenance or restart. Its live state and audit record change together through a serialized transition. Managed instances also expose protected lifecycle controls for reading and restoring instance state (server/…/instance-settings.ts• L110-133, server/…/instance-settings.ts• L310-539).

References: server/…/config.ts• L161-193 server/…/index.ts• L445-465 server/…/index.ts• L807-830 server/…/index.ts• L915-947

Health checks

/api/liveness reports process startup status without database access. /api/ready reports public in-memory traffic admission and returns 503 while the process starts, recovers, or drains. Both endpoints also support HEAD for probes. /api/health reports a public-safe result and build commit. Authorized callers can receive readiness details such as database reachability, authentication and bootstrap state, deployment data, backup health, and startup recovery state (server/…/openapi.ts• L1890-1931, server/…/health.ts• L231-291, server/…/health.ts• L318-432).

When a database is configured, the health endpoint probes it. A failed probe returns HTTP 503 with database_unreachable, while the response still carries build identity for deployment checks. Full details are limited to suitable board or agent actors. Backup freshness and failure warnings are part of the health contract. A protected operation can also coordinate a development-server restart (server/…/health.ts• L44-77, server/…/health.ts• L138-229, server/…/openapi.ts• L1933-2069).

References: server/…/openapi.ts• L1890-2069 server/…/health.ts• L231-432

Telemetry

Telemetry is optional. When an OTLP endpoint is configured, OpenTelemetry starts before database connections or the HTTP server so early startup work can be traced. Without an endpoint, the tracing path is a no-op (server/…/index.ts• L329-337, server/…/instrumentation.ts• L276-303). Product telemetry and analytics controls are described in Product telemetry and analytics controls.

The exporter supports gRPC, HTTP protobuf, and HTTP JSON. Automatic Node instrumentation covers server and database activity. Release or source revision data links telemetry to a build. Shutdown waits for the final export, but collector failure does not stop application control flow (server/…/instrumentation.ts• L305-339, server/…/instrumentation.ts• L425-519, server/…/index.ts• L2373-2379).

Optional Sentry monitoring adds error events when a backend DSN and a compatible optional SDK are present. Missing or mismatched monitoring dependencies leave the application running without error monitoring. Run failures can carry run context without leaking that context into unrelated errors (server/…/sentry.ts• L1-17, server/…/sentry.ts• L101-113, server/…/sentry.ts• L261-303).

Instance operators can enable /api/instance/server-metrics for a short rolling view of one server process. The protected endpoint reports process load, event-loop health, database statement rate, run-event lateness, agent request delay, missed lease renewals, and dedicated connection use. It is disabled by default and returns no cached response (server/…/instance-server-metrics.ts• L5-12, server/…/server-metrics.ts• L15-40, server/…/server-metrics.ts• L180-221, server/…/openapi.ts• L1642-1657).

References: server/…/index.ts• L329-337 server/…/index.ts• L2373-2379 server/…/instrumentation.ts• L276-519 server/…/sentry.ts• L1-303 server/…/instance-server-metrics.ts• L5-12 server/…/server-metrics.ts• L180-231 server/…/openapi.ts• L1642-1657

Backups and restore

Database backups protect the PostgreSQL system of record. A run writes a compressed SQL archive to the configured directory and applies retention. Scheduled, manual, and pre-sleep runs share one active-run guard. Retention is read from instance settings so an operator can change it without a restart (server/…/index.ts• L950-1027, packages/…/backup-lib.ts• L527-575).

Retention keeps recent daily backups, the newest backup for retained weeks, and the newest backup for retained months. Restore loads a compressed archive into a target PostgreSQL database and can use either native PostgreSQL restore or the platform's statement-by-statement fallback (packages/…/backup-lib.ts• L992-1027). Backup freshness and failure markers appear in health data (server/…/database-backup-health.ts• L105-152).

When idle sleep is enabled, a pre-sleep backup leaves a wake marker. The next restart verifies or refreshes that checkpoint before the instance becomes idle again. A failed verification keeps the marker so a later restart can retry the protection step (server/…/index.ts• L903-920, server/…/index.ts• L2083-2095).

Native execution also retains company-scoped harness backups and provider artifacts. A backup is identified by execution scope and manifest. Ready provider artifacts are deduplicated by closure digest and may be referenced by runs or backups. These records let cleanup and recovery distinguish reusable evidence from active work (packages/…/native_harness_backups.ts• L4-19, packages/…/native_provider_artifacts.ts• L13-64, server/…/index.ts• L1357-1363).

References: server/…/index.ts• L903-920 server/…/index.ts• L950-1027 packages/…/backup-lib.ts• L527-575 server/…/database-backup-health.ts• L105-152

Development and release automation

The development runner supervises the local server. It prepares shared artifacts, checks migration state, watches relevant source changes, and restarts an idle server when source or schema changes require it. A local service registry prevents duplicate development servers (scripts/dev-runner.ts• L24-68, scripts/dev-runner.ts• L412-514, scripts/dev-service.ts• L13-38).

Managed HTTPS development prefers a fresh built UI bundle. If that build fails, it falls back to Vite middleware and reports the slower or incomplete-page risk (scripts/dev-runner.ts• L165-173, scripts/dev-runner.ts• L614-645, scripts/dev-runner.ts• L885-899).

The PR workflow delegates to a trusted reusable workflow. Its routing gate checks repository, event, actor, pull request state, ancestry, merge tree, and workflow run before it selects an ephemeral runner. Failed checks fall back to a GitHub-hosted runner. The workflow also decides when a stacked pull request, release-notes-only change, or profiling request needs the full CI scope (.github/…/pr.yml• L1-19, .github/…/ci-routing.yml• L19-25, .github/…/ci-routing.yml• L57-224).

Release verification turns one requested source ref into type checks, grouped tests, native checks, a pre-release runner evaluation, and a build. Release automation moves through stable, beta, nightly, preview, and cloud-migrator channels. Docker publication builds each supported architecture, publishes by digest, and assembles one manifest only after all architecture builds complete (.github/…/release-verify.yml• L1-37, .github/…/release.yml• L1-31, .github/…/docker.yml• L35-76).

References: scripts/dev-runner.ts• L24-68 scripts/dev-runner.ts• L412-514 .github/…/ci-routing.yml• L19-224 .github/…/release-verify.yml• L1-37 .github/…/docker.yml• L35-76

Graceful shutdown

Shutdown stops new scheduler work before it drains active work. It closes realtime clients and drains admitted HTTP requests first. It keeps required handoff delivery alive, flushes telemetry and run-log mirrors, drains or finalizes runs, releases process identity for settled native runs, closes application services and database pools, and then stops embedded PostgreSQL when this process started it (server/…/index.ts• L2232-2249, server/…/index.ts• L2297-2379, server/…/shutdown.ts• L10-53, server/…/shutdown.ts• L100-132).

This order supports restart recovery. Durable records for incomplete leases, runtime services, login sessions, controller ownership, and execution finalization can be reconciled at the next startup instead of remaining active after a process exit (server/…/index.ts• L1035-1151, packages/…/0238_graceful_infant_terrible.sql• L1-7, packages/…/controller_instances.ts• L13-34).

References: server/…/index.ts• L2232-2249 server/…/index.ts• L2297-2379 server/…/shutdown.ts• L10-132

Change checklist

When these files or sections are changed, remember to consider these:

  • Route or WebSocket changes must preserve the split between normal HTTP handling and upgrade handling.
  • Route changes must update shared request schemas, response contracts, OpenAPI status codes, operation security levels, and connection-management metadata together.
  • MCP Apps and MCP Skills changes must keep board review, task access, active-run access, manifest identity, and company scope aligned.
  • Project environment changes must keep compatibility grouping, project creation validation, storage-owner data, and the separation from agent execution environments aligned.
  • Connection-management changes must resolve every referenced resource to one company, derive identity from authentication, preserve responsible-user checks, reauthorize after human setup, and keep collection filtering consistent with single-resource reads.
  • Adapter changes must keep external loading, runtime reload behavior, disabled-adapter discovery, and managed install floors aligned.
  • Startup changes must keep telemetry and error monitoring before first requests, the listener before recovery, external adapter and managed-environment loading before normal work, controller lease fencing, database-pool cleanup on every failure path, and handoff delivery alive until owner commands and responses drain.
  • Migration changes must keep journal order through 0499, migration inspection, concurrent-index rules, online-compatibility rules, data repair semantics, startup refusal behavior, and managed-cloud transient refusal handling aligned.
  • Responsibility-link changes must keep the one-target rule, company-skill foreign key, duplicate prevention, and migration snapshot aligned.
  • Issue recovery changes must keep full-pass checkpoints, deferral deadlines, queue names, company scope, and successor-server recovery aligned.
  • Settled native-run cleanup changes must keep process identity fields, recovery lookup, shutdown release, and the partial-index predicates aligned.
  • Automatic run brake changes must keep company scope, person-work bypass, probe ownership, release limits, and queued-run admission aligned.
  • Plugin lifecycle delivery changes must preserve company and resource scope, contiguous cursor meaning, acknowledgment gaps, and replay behavior.
  • Gateway token changes must keep pending reveal separate from the token hash and ensure agents cannot receive the token value.
  • Secret proposal changes must preserve cascade behavior when the referenced secret is deleted.
  • New durable coordination records must preserve company scope, idempotency fences, ownership or lease expiry, and recovery behavior across process replacement.
  • Storage changes must preserve company-prefix checks and equivalent local-disk and S3 behavior.
  • Link-preview changes must preserve company and viewer isolation, canonical URL identity, and expiry handling.
  • Tool-content deletion changes must preserve one pending request per connection, retry state, and company or connection ownership.
  • Legacy plugin connector changes must preserve access only for plugins covered by the migration and keep later installs on normal connector activation rules.
  • Responsibility summary changes must keep the model-call record, cost event, and exactly-one budget source aligned.
  • Workload identity changes must keep public issuer discovery, board-only rotation, ES256 key state, and one active or next key per company aligned.
  • Deployment changes must stay aligned with host binding, authentication setup, public readiness admission, health detail visibility, controller fencing, and managed configuration floors.
  • Backup changes must preserve archive freshness checks, idle wake-marker handling, and failure-marker detection.
  • Release workflow changes must keep exact source identity, trusted runner selection, migration checks, native artifact identity, Docker digest assembly, and release-channel publication aligned.
  • Shutdown changes must preserve scheduler quiescence, run recovery records, settled native process release, listener draining, database availability for cleanup, telemetry flushing, and subscription removal for process-local cleanup signals.

Important failure modes

TriggerConsequenceGuard
Public authenticated deployment has no valid external PostgreSQL URLStartup refuses to use embedded PostgreSQL.Database contract validation in server/…/index.ts.
A non-empty database has no migration journal or pending migration approval is declinedStartup stops instead of serving against an unknown schema.Migration inspection and refusal in server/…/index.ts and packages/…/client.ts.
A migration repair or concurrent index is applied outside its expected database stateStartup or migration execution fails before normal traffic can use a partial schema.Ordered migration journal and database-package inspection rules.
Responsibility-link data names no target or more than one targetThe schema rejects the link and responsibility views cannot rely on a single target.responsibility_links_single_target_check in packages/…/responsibilities.ts and migration 0499.
Automatic recovery runs fail before starting provider work at a high rateNew automatic runs pause for that company while human-requested work remains possible.Durable automatic_run_brakes state with probes and gradual release in server/…/automatic-run-brake.ts.
A settled native run keeps process identity after the owning server exitsRecovery can keep rechecking a process that has already finished or delay safe cleanup.Shutdown release step and settled-process lookup index in server/…/index.ts and migration 0487.
A controller lease expires or a newer boot fences the old processThe old process must stop owning durable work.Boot identity, database lease, dispatch handoff, and fencing state in controller_instances.
A project points to a deleted environmentThe project keeps its own data but no longer has a project environment link.Nullable foreign key with on delete set null in packages/…/projects.ts.
A plugin loses its cursor or acknowledges past a gap incorrectlyLifecycle events can be replayed or skipped for that resource.Contiguous cursor and per-event acknowledgment records in plugin_lifecycle_resource_cursors.
A database pool remains open after failed startupRestart loops can exhaust PostgreSQL connections.Startup teardown closes every pool it created.
Database probe failsHealth returns HTTP 503 with database_unreachable.Readiness probe in server/…/health.ts.
A WebSocket upgrade lacks valid company-scoped identityThe connection is rejected and live data is not exposed.Upgrade authorization in server/…/live-events-ws.ts.
A scheduled cleanup path runs during task drain or warm standbyMaintenance can admit new work or leave paid provider resources allocated.Scheduler admission checks and independent cleanup sweeps in server/…/index.ts.
Storage key is outside the requested company or contains traversalThe object operation is rejected.Company-prefix and path checks in server/…/service.ts.
Backup is stale, a backup fails, or an idle wake marker cannot be verifiedHealth reports a backup warning and idle readiness stays guarded.Backup health inspection and wake-checkpoint flow in server/…/database-backup-health.ts and server/…/index.ts.
The server metrics flag is invalid or the endpoint is called without instance-admin accessStartup rejects the bad setting or the metrics read is denied.Strict flag parsing and instance-admin authorization in server/…/server-metrics.ts and server/…/instance-server-metrics.ts.
A connected-app Skill changes its manifest after approvalThe Skill is no longer treated as the approved content.Manifest digest and approval state in packages/…/tool_access.ts.
A tool gateway session is missing, expired, or revokedTool listing and execution are rejected.Session-token checks in server/…/tool-gateway.ts.
A company has more than one active or next workload identity keyKey provisioning or rotation fails instead of creating ambiguous signing state.Partial unique indexes in packages/…/company_workload_identity_keys.ts.
A responsibility summary reserves budget without its matching invocation or cost kindThe accounting write is rejected and the summary cannot be charged as an unrelated run.Source and usage-kind checks in packages/…/accounting.ts and packages/…/cost_events.ts.
Shutdown closes the database before application cleanup finishesFinalization, recovery, or login-session cleanup can fail.Ordered shutdown in server/…/index.ts and server/…/shutdown.ts.

Key files

FileWhy to read it
server/…/index.tsShows startup composition, database selection, migrations, backups, WebSocket setup, recovery, controller leases, scheduling, deployment handoff, settled native process release, and shutdown.
server/…/app.tsShows middleware order, route mounting, UI serving, provider services, and application cleanup.
server/…/config.tsDefines deployment, database, storage, backup, and telemetry settings.
server/…/connection-management-authz.tsShows the shared company, actor, responsible-user, resource, and human-setup boundary for connection routes.
server/…/openapi.tsShows how schemas, responses, status codes, health contracts, server metrics, workload identity discovery, MCP Apps, MCP Skills, project routes, and security levels become the public API contract.
server/…/instance-server-metrics.ts and server/…/server-metrics.tsDefine the protected, opt-in process-load view used during scale and operations work.
packages/…/client.tsDefines database client creation and migration inspection and application.
packages/…/_journal.jsonDefines the ordered migration history used to compare the deployed schema with the application version.
packages/…/0465_better_auth_account_identity.sql through packages/…/0499_numerous_whizzer.sqlShow the current operational schema changes for identity, ownership, lookup, controller handoff, recovery checkpoints, tool content cleanup, plugin compatibility, responsibility reporting, accounting, workload identity, and responsibility-to-skill links.
packages/…/controller_instances.tsDefines boot leases and dispatch handoff state used during process replacement.
packages/…/automatic_run_brakes.ts and server/…/automatic-run-brake.tsDefine the durable company-level pause and controlled release of automatic recovery work.
packages/…/issue_recovery_full_passes.tsDefines durable full-pass and deferred-check state used to continue issue recovery across server replacement.
packages/…/tool_access.tsDefines durable tool access, provider task handles, MCP Apps views, connected-app Skills, gateway sessions, and related company scope.
packages/…/tool_connection_content_deletions.tsDefines retryable deletion requests for content owned by a tool connection.
packages/…/company_workload_identity_keys.tsDefines company signing-key state used by workload identity federation.
packages/…/responsibilities.tsDefines responsibility areas, responsibility metrics, and the single-target links that can include company skills.
packages/…/responsibility_reporting.ts and packages/…/responsibility_dashboard_versions.tsDefine the durable responsibility reporting, metric, and history records introduced by the current schema tail.
packages/…/responsibility_summary_invocations.ts and packages/…/accounting.tsDefine model-summary accounting and its budget reservation source.
packages/…/heartbeat_runs.ts and packages/…/heartbeat_run_events.tsDefine durable run state, controller ownership, ordered events, usage data, and recovery indexes.
packages/…/agent_wakeup_requests.tsDefines durable wake delivery, idempotency fences, and run lookup.
packages/…/backup-lib.tsDefines backup archive creation, retention, and restore behavior.
server/…/live-events-ws.ts and server/…/live-events.tsDefine live-event authorization, subscriptions, health checks, and process-local publication.
server/…/service.tsDefines provider-neutral storage behavior and company-safe object naming.
server/…/health.tsDefines public liveness, readiness, database probes, backup warnings, and development restart coordination.
server/…/instrumentation.ts and server/…/sentry.tsDefine optional tracing, error monitoring, compatibility checks, and shutdown flushing.
scripts/dev-runner.tsDefines local server supervision, migration preflight, UI fallback, and restart behavior.