Introducing a new way to review your PRs.

Read the announcement

paperclipai / paperclip

The open-source app everyone uses to manage agents at work

View on GitHub

30 documents · Updated Nov 20, 2026

docs / execution-environments

Execution environments and workspaces

Execution environments define where agent work runs and which provider rules apply. Execution workspaces define the code and runtime state for that work. Together they turn an environment choice into a usable workspace, bounded runtime access, safe cleanup, and operator-visible recovery. See agent work management for the wider work model, issue execution for run behavior, agent instructions for work guidance, adapter integrations for adapter boundaries, and native runner execution for the native runner lifecycle. Related recovery and reuse context is in runner evaluation and conformance, case handoffs and continuity, and reusable workflows.

Environment model

An environment is a named execution target with a driver, configuration, runtime settings, and ownership rules. A lease records the provider resource used by one run, so later environment edits or deletion do not redirect cleanup to another provider.

Environment selection checks that the target is active and allowed for the adapter. A connection or sandbox probe supplies readiness evidence without turning a test into retained work. A sandbox probe uses a fresh temporary lease and releases it before it reports success.

Environment configuration is validated at the boundary where it is used. SSH settings require a host, user, absolute workspace path, and valid network selection. Plugin-backed sandbox settings are checked by the active plugin worker. Sensitive values are stored as secret references and resolved only for the runtime, probe, or cleanup action that needs them. Sandbox provider references use the latest secret version, while old saved sandbox settings can still load after removed host-only fields are stripped.

References: server/…/environment-run-orchestrator.ts• L193-220 server/…/environment-config.ts• L41-139 server/…/environment-config.ts• L164-200 server/…/environment-config.ts• L452-560 server/…/environment-config.ts• L682-824

Capability snapshots

Each provider lease receives an effective capability snapshot. The snapshot is the intersection of the driver's static support, verified worker methods, provider declarations, and lease-specific narrowing. This gives every later execution step one policy view instead of letting a provider claim a feature that the current worker or lease cannot safely support.

Worker-property capabilities can use the verified baseline when the provider gives no declaration. Opt-in capabilities such as incremental output, concurrent transfers, duplex command streams, and Runner ingress stay disabled unless the provider declares them and the worker proves the required methods. Missing verification or failed configuration resolution removes authority. Execution reads the same snapshot before dispatch.

References: server/…/environment-runtime.ts• L152-180 server/…/environment-runtime.ts• L182-237 server/…/environment-runtime.ts• L286-350 server/…/environment-runtime.ts• L352-395 server/…/environment-config.ts• L425-457

Provider availability

Provider availability separates a temporary refusal from a setup error that needs a human fix. Lease acquisition happens before agent work, so bounded startup retry can use structured HTTP status, connection codes, provider deadlines, validated acquisition evidence, or transient database driver evidence. A definite non-transient status takes priority, and free-form error text never grants retry authority.

Acquisition evidence is tied to the company, environment, and run that created it. It can identify an unfinished creation, a provider that failed during sandbox start, or a database failure while recording the lease. An unfinished provider allocation keeps its own creation identity and cleanup record. A retry starts a new creation attempt. This prevents a retry from reusing an uncertain resource or losing responsibility for its cleanup.

The warm-run default also checks whether the provider can safely reuse leases. It applies only when the provider declares reuse and its live worker supports the required lifecycle actions.

References: server/…/environment-provider-unavailable-error.ts• L31-113 server/…/environment-provider-unavailable-error.ts• L115-162 server/…/environment-acquisition-diagnostics.ts• L6-23 server/…/environment-run-orchestrator.ts• L225-267 server/…/environment-runtime.ts• L2048-2057 server/…/environment-runtime.ts• L696-708

Provider outage waiting

When a provider remains unavailable after the fast startup retries, the run enters a parked wait instead of consuming retries continuously. The wait keeps the first outage time, backs off from one minute to two minutes and then five minutes, and stops after a 30-minute ceiling. The task then needs a later explicit retry if the provider is still unavailable. The task records the provider identity so its status can tell the operator what it is waiting for.

References: server/…/environment-provider-wait.ts• L1-16 server/…/environment-provider-wait.ts• L30-73 server/…/environment-provider-wait.test.ts• L22-49

Acquisition fault testing

Lease acquisition has a test-only fault seam for restart and recovery checks. A controlled file can inject bounded timeout or database-like failures before a provider call, while production ignores the control. The injected errors use the same structured evidence as real failures, so retry and cleanup paths are tested without a paid provider.

References: server/…/environment-run-orchestrator.ts• L241-264 server/…/environment-lease-fault-injection.ts• L1-75

Workspace model

An execution workspace is the durable record of the code location and runtime state used for issue work. It links a project workspace to its path, branch, provider, environment, runtime settings, related issues, delivery state, and cleanup state.

Workspace policy can choose a shared location, an isolated worktree, an adapter-managed location, or a managed remote workspace. Shared locations favor continuity. Isolated locations keep changes apart and provide a boundary for lower-trust work. Reuse is allowed only when repository, path, branch, ownership, and workspace identity agree.

References: packages/…/workspace-runtime.ts• L261-288 server/…/execution-workspace-policy.ts• L107-233 server/…/workspace-runtime.ts• L2767-2858

Workspace realization

Workspace realization turns the selected code location into the effective workspace inside the selected environment. The run keeps a stable description of its authoritative root, transport, paths, provider, and code movement rules. Additional project workspaces remain read-only inputs.

The run follows this order because the adapter must not use a workspace before its environment, lease, and ownership are known:

flowchart LR
    A[Issue run] --> B[Select environment]
    B --> C[Acquire or resume lease]
    C --> D[Realize workspace]
    D --> E[Resolve execution target]
    E --> F[Prepare and run adapter]
    F --> G[Release retain or clean up lease]

Local targets use the local workspace. SSH targets use a remote path that already exists on the host. Sandbox targets create the remote workspace first, then the adapter transfers the prepared tree. Plugin-backed Runner work uses its native task bridge and does not fall back to host execution. A managed remote workspace is qualified for a native sandbox and prepared at its authoritative remote root before the adapter runs.

The host provision command runs only where the workspace tree is available. It is skipped for local and sandbox runs when the command belongs to the host-side preparation stage. Sandbox adapters deliver the prepared tree during synchronization. SSH provisioning runs on the remote host that owns the workspace path.

References: server/…/environment-run-orchestrator.ts• L302-481 server/…/environment-run-orchestrator.ts• L514-609 server/…/environment-run-orchestrator.ts• L611-739 server/…/environment-run-orchestrator.ts• L766-838

Runtime provisioning

Runtime provisioning prepares the tools and dependencies needed by workspace commands. Workspace realization establishes the code location first. Provisioning then runs in the target that owns that location, so a remote command does not assume an empty or unrelated directory.

Runtime configuration supports long-running services and one-shot jobs. Services can be shared or temporary. A healthy service with the same reuse identity can continue instead of restarting. Runtime control remains exclusive per execution workspace, while stale owners can be recovered.

For sandbox work, the adapter delivers the prepared tree after the remote workspace exists. The host provision command therefore does not run in the empty sandbox. SSH provisioning runs on the remote host that owns the workspace path. Plugin Runner work provisions through the native task bridge rather than a host fallback.

References: server/…/environment-run-orchestrator.ts• L683-745 server/…/environment-runtime.ts• L595-701 server/…/environment-task-runtime.ts• L250-290

Runtime exposure and readiness

Managed runtime exposure makes a workspace service reachable through a controlled endpoint. The system reserves the endpoint before startup and publishes the mapping only after the service passes its readiness contract. A live port alone is not proof that the workspace is correct.

Readiness checks workspace data, seed results, representative data, login handoff, and instance, company, and workspace identity. A workspace can move through these states:

StateMeaning
provisioningRestore or seed work is in progress.
validatingRestore finished, but data or identity checks are incomplete.
readyProtected readiness signals agree.
degradedA ready workspace lost a signal and needs repair.
repairingA managed repair is active.
failedProvisioning or repair reached a terminal failure.

References: server/…/workspace-runtime.ts• L4484-4521 server/…/workspace-runtime.ts• L5592-5639 server/…/workspace-runtime.ts• L6150-6408 server/…/workspace-readiness.ts• L179-295

Platform qualification

Platform qualification records the OS, architecture, and libc facts of the exact immutable image or verified snapshot used by a sandbox. The proof remains valid only while the environment and image identity match. A successful connection does not prove native runtime compatibility.

References: server/…/environment-probe.ts• L19-69 server/…/environment-probe.ts• L164-186

Provider leases and policy controls

An environment lease binds company, issue, run, execution workspace, provider resource, lifetime, status, and cleanup result. Its resource disposition decides whether the provider resource stays running, stops for later reuse, or is destroyed after the run.

Reusable leases require the same company, environment, workspace or projectless issue scope, agent, adapter, provider, runtime settings, and secret versions. An effective configuration fingerprint blocks reuse after provider, plugin, runtime, workspace migration, or secret-version drift.

The lease also preserves provider identity and the process owner that created or holds it. Recovery uses these saved facts rather than a later environment edit or a lease row that merely happens to be newer.

References: server/…/environment-runtime.ts• L861-1013 server/…/environment-runtime.ts• L1016-1100 server/…/environment-runtime.ts• L1671-1849 server/…/environment-task-runtime.ts• L285-290

Lease settlement

Lease settlement handles the gap between a terminal run and confirmed provider stop or destroy. It acts only after a shared recovery check proves the original holder is gone. It leaves the lease alone when workspace export, recovery, another live lease, or a native result still needs the resource.

If a dead run held a sandbox, settlement stops and retains it first. A later cleanup step destroys the retained orphan only after recovery obligations and other resource owners end. This keeps recovery possible without leaving warm resources indefinitely.

References: server/…/environment-run-lease-settlement.ts• L23-90 server/…/environment-retained-orphan-settlement.ts• L162-268

Environment cleanup

Environment cleanup is deferred maintenance for provider resources whose creation, release, or database write was interrupted. It uses the provider identity and saved lease data, so an environment change or deletion does not strand the resource.

Cleanup retries use bounded timing and safe metadata parsing. A temporary provider-worker outage is checked before an attempt, which preserves the finite retry budget. When the database is unavailable, an in-process orphan record waits for a later flush and then becomes normal pending cleanup.

A teardown claim fences other cleanup for a limited time. If its process dies, settlement returns the lease to the cleanup sweep without changing whether the resource still needs a stop, destroy, or workspace-preservation action.

References: server/…/environment-cleanup-state.ts• L6-95 server/…/environment-runtime.ts• L721-760 server/…/environment-runtime.ts• L1360-1457 server/…/environment-runtime.ts• L2439-2657

Plugin task execution

Plugin task execution gives a plugin-backed environment a controlled path for Runner work. The server accepts a schema-checked operation only with an authorized company and persisted lease. It then checks the pinned plugin, worker support, lease state, project scope, managed workspace binding, and Runner identity before dispatch.

The task identity stays bound to the same workspace and lease chain. Preparation and submission carry checked launch and project-auth evidence. A timeout is ambiguous, so retry uses the same operation boundary until the provider reports its state. Stop and completion require confirmed remote stop. If confirmation is missing, the lease remains a cleanup obligation.

Warm tasks can resume only when the provider proves the prior task state and the saved Runner identity matches. If a warm task is lost, the system confirms the old task stopped before it creates a replacement. The replacement keeps the same authorization boundary and updates the lease only after provider creation is recorded.

References: server/…/environment-task-runtime.ts• L39-100 server/…/environment-task-runtime.ts• L219-318 server/…/environment-task-runtime.ts• L341-461 server/…/environment-task-runtime.ts• L482-605 server/…/environment-task-runtime.ts• L608-627

Workspace operation observability

Workspace operations provide the durable view of provisioning, runtime control, repair, cleanup, and workspace commands. Each operation records its phase, status, progress, bounded output, log reference, and exit result. The workspace API can use this history instead of relying on process logs.

Managed runtime controls are exclusive per execution workspace. A live owner refreshes its heartbeat. A dead or stale owner is reconciled with a compare-and-swap update. Lifecycle controls have a finite ceiling, and long-running workspace jobs have a wider but still finite ceiling.

Provider execution traces add bounded timing and lifecycle facts to the same run. They record provider family, command label, exit result, timing, critical-path status, and cache status. They omit credentials, full commands, paths, and stream text. Incremental output reaches durable logs in order. A thrown provider execution still creates a failed span, while observability errors do not change execution control flow.

References: server/…/workspace-operations.ts• L24-73 server/…/workspace-operations.ts• L367-413 server/…/workspace-operations.ts• L438-519 server/…/workspace-operations.ts• L584-850 server/…/environment-execution-target.ts• L168-225 server/…/environment-execution-target.ts• L473-625

Change checklist

When these files or sections are changed, remember to consider these:

  • Keep capability declarations, verified worker methods, static driver support, and lease narrowing aligned. Unknown support must remove access.
  • Keep host-owned sandbox lifecycle flags aligned with plugin normalization. A provider must not silently replace a selected warm or per-turn mode.
  • Keep execution-target gates aligned with the capability snapshot. Native sync needs both directions. Duplex and Runner ingress must remain explicit grants.
  • Keep the reusable-lease support probe aligned with acquisition. A provider must not receive the warm default unless its declaration and live lifecycle methods support resume, release, and destroy.
  • Keep provider-unavailable classification tied to structured status, error codes, acquisition diagnostics, or database driver evidence. Preserve the company, environment, and run scope of acquisition diagnostics. A definite non-transient status must override deeper transient evidence. Do not turn message text into retry authority. Keep the later parked wait tied to the failed environment identity and its original wait time.
  • Keep environment secret references aligned across persistence, probe, runtime, cleanup, and reuse fingerprints. Cleanup must work from the recorded lease after environment deletion.
  • Keep lease fingerprints and reuse scope aligned with provider, workspace, agent, adapter, runtime, migration, and secret-version changes.
  • Keep remote workspace authority, provider bindings, authoritative roots, and adapter transport selection aligned before allowing commands to run.
  • Keep provider identity, owner proofs, compare-and-swap checks, and run terminal state aligned before changing orphan settlement.
  • Keep retained-sandbox holds aligned with recovery actions, open runs, exports, stop receipts, and provider-resource sharing.
  • Keep cleanup retry timing, worker readiness checks, deferred buffers, and teardown claims aligned. A dead claimant must not refund or lose an attempt twice.
  • Keep readiness checks separate from transport health. A responding port must not publish an unverified workspace.
  • Keep plugin task launch digests, project-auth proofs, workspace bindings, Runner identities, and replacement ownership aligned.
  • Keep runtime-control admission, owner heartbeats, stale recovery, compare-and-swap updates, and operation time budgets aligned.
  • Keep trace fields bounded and non-secret. Observability must not alter provider execution or expose command arguments, paths, or stream text.

Important failure modes

TriggerConsequenceGuard
Capability proof is missing or cannot be readThe runtime may use an unverified sync, session, or command channelIntersect static support, verified methods, provider declarations, and lease narrowing. Fail closed on resolution errors.
A host-owned lifecycle flag is dropped during plugin normalizationA warm Runner can fall back to per-turn behavior without an explicit configuration changeReapply host-owned flags after provider normalization.
Warm-run support is inferred from configuration onlyA run can request a reusable lease that the live worker cannot resume or clean upProbe the provider declaration and live worker lifecycle methods before applying the warm default.
Provider returns an unstructured outage messageA temporary outage may be retried incorrectly or a permanent error may consume the startup budgetClassify only structured provider, acquisition, or database evidence.
A definite non-transient status wraps transient-looking evidenceA permanent setup error can be retried as if the provider were briefly unavailableTreat the definite non-transient status as authoritative.
Acquisition evidence belongs to another run or is from an unrelated stepThe current run can retry without proof that its own provider creation was transientAccept only evidence scoped to the current company, environment, and run.
A provider outage loses its saved environment or wait startThe task can wait for the wrong provider or reset its outage budget on every wakePreserve the failed environment identity and original wait timestamp. Apply the bounded backoff and ceiling.
A reusable lease is matched after runtime, migration, or secret driftA later run can inherit stale provider stateRequire the full reuse scope and effective configuration fingerprint.
A secret is resolved only from the current environment during cleanupAn environment update or deletion can strand the provider resourceResolve cleanup secrets from the durable orphan record and its saved schema.
Remote workspace authority or provider binding changes during setupCommands can run against the wrong root or through an unsafe host fallbackRecheck the lease, workspace, provider binding, authoritative root, and native task bridge before dispatch.
A lease holder dies between run completion and releaseThe provider resource can remain active or be cleaned while recovery still needs itProve the stamped holder is gone, then settle ownership with an exact-row compare-and-swap.
A retained sandbox has no current owner but an open recovery action remainsRecovery loses access to the stopped workspace sourceHold the sandbox until the recovery action and retention window end.
A port answers before workspace identity is verifiedAn incorrect clone can be published as healthyRequire the protected readiness contract before exposure or reuse.
A teardown claimant dies while its fence is activeCleanup waits for the fence or loses the retry attemptSettle the exact claim and return it to the pending-cleanup sweep once the claimant is proven gone.
A Runner operation times out after dispatchA retry can create duplicate work or attach to a changed taskRetry the same operation boundary and reconcile the persisted task identity first.
A runtime-control owner dies or stops heartbeatingThe workspace can remain blocked by an abandoned operationReconcile dead or stale owners with compare-and-swap and preserve the operation evidence.
A provider execution throws before returning a resultFailure timing and cause can be missing from tracesEmit a bounded failed execution span while keeping raw command and output out of the span.

Key files

FileWhy to read it
server/…/environment-config.tsShows how environment settings are validated, persisted, probed, and resolved with secret references.
server/…/environment-run-orchestrator.tsShows the run lifecycle from environment selection through lease use, workspace realization, target resolution, and release.
server/…/environment-runtime.tsConnects provider leases to capabilities, reuse identity, runtime actions, teardown, and deferred cleanup.
server/…/environment-execution-target.tsShows how lease capabilities become local, sandbox, plugin Runner, or SSH targets and bounded execution traces.
server/…/environment-provider-unavailable-error.tsDefines structured provider and acquisition failure classification for bounded startup retry.
server/…/environment-acquisition-diagnostics.tsDefines the scoped evidence that links provider acquisition failures to the run that owns them.
server/…/environment-provider-wait.tsDefines the parked wait and bounded backoff used after fast provider retries.
server/…/environment-lease-fault-injection.tsDefines test-only acquisition faults used to exercise retry and cleanup recovery.
server/…/environment-run-lease-settlement.tsExplains recovery when a run or its lease holder disappears before release.
server/…/environment-retained-orphan-settlement.tsExplains when a retained sandbox remains a recovery source and when it becomes destroyable.
server/…/environment-cleanup-state.tsDefines cleanup retry timing, malformed-metadata handling, and recoverable creation rules.
server/…/environment-task-runtime.tsDefines authorization, identity checks, dispatch, resume, stop confirmation, replacement, and cleanup for plugin Runner tasks.
server/…/workspace-operations.tsDefines durable operation records, logs, progress, ownership heartbeats, time limits, and stale-operation recovery.
server/…/execution-workspace-policy.tsDefines workspace modes, selection strategy, and trust-related workspace policy.
server/…/workspace-runtime.tsCovers workspace realization, runtime provisioning, service lifecycle, readiness, exposure, and port ownership.
packages/…/workspace-runtime.tsDefines the shared shape of workspace runtime and service state.
packages/…/environment_leases.tsDefines durable provider lease identity, status, expiry, and cleanup state.
server/…/environment-provider-unavailable-error.test.tsShows the contract for structured transient evidence, scoped acquisition diagnostics, and non-retryable errors.