Runner evaluation and conformance
Runner evaluation and conformance provide a common way to show that native runner work follows Paperclip's protocol and meaning. The system links capability definitions, protocol histories, semantic results, workflow outcomes, provider identity, and trace records so engineers can compare execution behavior without relying only on provider output. See agent work management for the parent work model and native runner execution for the execution path.
Evaluation scope
The evaluation system checks execution at several levels. Protocol fixtures check the wire contract. Capability cases check which actions are allowed and what they mean. Workflow cases check that a complete run keeps its state, continuation, presentation, and trace evidence aligned.
Capability catalog
The native runner capability catalog is the provider-neutral authority for actions and their placement. It keeps three views separate: protocol families describe wire evidence, capability placement describes who owns an operation, and behavior groups organize the scenario corpus. A behavior group does not grant a tool. A protocol family does not define tool meaning.
The transport-neutral semantic action catalog adds an immutable declaration for each native action. Each declaration states the operation, effect, placement, required claims, allowed task modes, and input and output shapes. The catalog does not grant discovery or invocation authority. A run-scoped authorization layer must still admit the action.
The catalog reconciles the control-plane set, live catalog, generated provider contract, legacy aliases, and scenario links. The generated contract publishes the normative inventories, evaluation groups, primary dispositions, and current coverage counts used by this view. Coverage records connect operations and scenarios to contract owners, deterministic owners, and evidence. This makes coverage a trace aid rather than only a count.
References: packages/…/semantic-action-catalog.ts• L90-140 packages/…/semantic-action-types.ts• L91-108 packages/…/reconciliation.test.ts• L19-26 packages/…/reconciliation.test.ts• L75-130 packages/…/operation-groups-doc.ts• L102-124 packages/…/operation-groups-doc.ts• L281-289 packages/…/capability-contract.ts• L1-46
Contract layers
The checks use separate contracts for wire shape, action meaning, observed behavior, and scored results. A protocol schema answers whether data has the right form. A semantic contract answers whether an action has the right effect. An observation and scorecard answer whether a run met its workflow expectations.
Completion uses a provider-neutral result contract. It separates the user-facing answer from the controller's continuation reason and from evidence about verification and attention. Provider input can use known aliases, legacy field values, or an explicit absent continuation for compatibility. Normalization must still produce the strict canonical result, where disposition controls attention and continuation. Unsupported contract versions fail closed so the runner does not guess what changed.
References: packages/…/replay-contract.ts• L23-63 packages/…/semantic-conformance.ts• L13-67 packages/…/workflow-contracts.ts• L3-28 packages/…/eval-scorecard.v2.schema.json• L5-17 packages/…/completion-result.ts• L147-214 packages/…/completion-result.ts• L397-464 packages/…/completion-result.test.ts• L61-81 packages/…/completion-result.test.ts• L168-205
Conformance suites
Conformance suites compare runner boundaries against stable contracts. They use fixed inputs and normalized outputs so provider-specific message formats do not hide differences in Paperclip behavior.
Protocol conformance
Protocol conformance checks identity, version, event order, duplicate delivery, result binding, and terminal meaning. A shared fixture describes one run through its commands, events, requests, capabilities, and result. The validator checks both the record shapes and the links between them.
The control-plane harness also checks source gaps, changed content under an existing event identity, forged run bindings, repeated completion, lease authority, and durable ownership before command admission. Credential updates act as content-free notifications. Their revision and binding identify the change without placing secret material in the event.
The protocol contract checks keep ACPX message families on one declared version and bind candidate admission to the run and session that use it. They also keep question adapter fixtures and cross-language conformance input and output aligned, while closed envelopes reject unsupported fields and drift. The ACPX checks also keep runtime context objects closed, preserve provider-specific thinking-level identity, and restrict rich activity to display-only payloads. The command contract binds external-provider operations to the run, session, turn, assignment, and input digest, then restricts each action to its expected input shape. It also makes provider-started turn signaling an explicit opt-in rather than an assumed state.
References: packages/…/replay-contract.ts• L114-177 packages/…/replay-contract.ts• L193-376 packages/…/control-plane-port.ts• L85-184 packages/…/auth-credentials.test.ts• L19-41 packages/…/durable-prp-control-plane.test.ts• L49-137 packages/…/acpx-sidecar-contract.test.mjs• L50-157 packages/…/acpx-sidecar-contract.test.mjs• L179-279 packages/…/command-v3.schema.json• L1-545 packages/…/protocol-contract.test.mjs• L28-37 packages/…/protocol-contract.test.mjs• L68-152 packages/…/protocol-contract.test.mjs• L670-686
Semantic conformance
Semantic conformance checks whether different action adapters produce the same provider-neutral result for the same operation and input. An observation records authorization, projected state, effects, audit evidence, and a normalized receipt. The suite compares the observation with the expected result and with other adapters.
Completion and continuation are part of this meaning. A provider may omit or set continuation to null at its boundary, but the controller can fill that wait only from current task state with the same run identity and an eligible saved monitor or other authorized continuation. It must not invent a wait from stale or incomplete state. Reports identify the operation, adapter, and changed paths when results differ.
References: packages/…/semantic-conformance.ts• L69-88 packages/…/semantic-conformance.ts• L142-200 packages/…/capability-semantic-conformance.ts• L44-149 packages/…/completion-continuation.test.ts• L8-36 packages/…/completion-result.ts• L401-430 packages/…/completion-result.test.ts• L40-66 packages/…/completion-result.test.ts• L145-205
Capability evaluation matrix
The capability evaluation matrix turns catalog trace data into offline semantic cases. It covers typed tool effects, allow and deny policy, private control-plane actions, multi-step behavior, and restraint cases where no action is correct.
Optional actions run with and without their required grants. The suite compares the fake-agent and Codex tool surfaces and checks a bounded provider sample. Catalog reconciliation checks that these surfaces use the same claims and task modes.
References: packages/…/capability-eval-suite.ts• L16-32 packages/…/capability-eval-suite.ts• L102-128 packages/…/capability-eval-suite.ts• L150-225 packages/…/operation-groups-doc.ts• L147-188
Native provider conformance
Native provider conformance checks the boundary where provider sessions become durable runner events. Controlled Codex, ACPX, OpenCode, and external-runner cases make provider faults repeatable without paid model calls.
The suites protect identity and authority across provider turns, tool calls, completion, cancellation, interruption, recovery, launch profiles, and terminal projection. The OpenCode fixture also gives repeatable cases for authentication, assigned MCP use, runtime requests, streaming order, duplicate and late delivery, long history, managed operations, trace export, terminal projection, completion feedback, and secret-safe diagnostics. Its simulated provider can produce native questions and permissions in both current and legacy forms, which lets the adapter be checked against both event styles.
Native process cases use a private host-control fixture when the runner must listen on a host channel. The fixture tracks the native process identity, bounds control frames and replies, serializes requests, and confirms the child exit before it reports completion. Cleanup rechecks the original process generation before it signals a detached process group.
References: packages/…/installation-integrity.test.ts• L604-618 packages/…/recovery-identity.test.ts• L125-173 packages/…/runtime-sandbox.test.ts• L145-225 packages/…/acpx-sidecar-contract.test.mjs• L81-190 packages/…/fake-opencode-server.mjs• L1-16 packages/…/fake-opencode-server.mjs• L121-220 packages/…/fake-opencode-server.mjs• L223-307 packages/…/fake-opencode-server.mjs• L310-445 packages/…/fake-opencode-server.mjs• L456-616 packages/…/fake-opencode-server.mjs• L618-758 packages/…/fake-opencode-server.mjs• L760-1146 packages/…/private-host-control.py• L1-127 packages/…/private-host-control.ts• L19-143 packages/…/stdio-mcp.test.ts• L20-63 packages/…/runnerd-muse-driver.test.ts• L72-147
Replay fixtures
Replay fixtures are fixed protocol histories used to test validation, reduction, and recovery without a live provider. They let an engineer reproduce a run from known events and compare the resulting state with a stable expected result.
Fixture validation
Replay validation checks required versions, schema shape, run identity, semantic call phases, command order, result binding, and terminal binding. It accepts permitted additive fields but rejects unsupported versions and changed content under an existing event identity.
References: packages/…/replay-contract.ts• L50-80 packages/…/replay-contract.ts• L379-414 packages/…/conformance-fixture.schema.json• L4-45
Snapshot reduction
Replay reduces a valid ordered history into a session snapshot. An invalid history returns structured issues without a partial snapshot. This prevents incomplete or conflicting evidence from becoming a valid run.
References: packages/…/replay.ts• L7-24 packages/…/replay-loader.ts• L14-31 packages/…/happy-path.json• L1-18
Native execution bundles
Native execution bundles join one runner attempt's identity, inputs, runtime context, provider profile, completion contract, semantic results, transcript, usage, terminal state, and ordered events. The evaluator can judge the attempt as one evidence set instead of trusting a summary field alone.
The versioned native execution boundary preserves saved provider and driver identities across recovery while accepting closed historical input schemas. It binds the task, workspace, session policy, completion contract, interaction responses, credential references, and runtime-context digest. The model envelope carries task context without authority or credential bindings. Current provider options must match their driver and qualified profile, while historical profiles can decode for recovery without becoming current launch authority. Runtime-context drift, provider mismatch, unsupported profiles, and unsafe limits fail closed.
The seeded native execution fixture provides a deterministic evidence set for a denied semantic effect. Its case, configuration, attempt, runner, event, semantic call, denial receipt, terminal result, transcript, and artifact digests share one seeded identity. The fixture also records seeded provenance and preserves additive fields without changing the core evidence links.
References: packages/…/native-execution.ts• L10-25 packages/…/native-execution.ts• L165-274 packages/…/native-execution.ts• L358-397 packages/…/native-execution.ts• L414-588 packages/…/native-execution.ts• L691-764 packages/…/native-execution.ts• L945-994 packages/…/runtime-context.ts• L11-34 packages/…/runtime-context.ts• L107-179 packages/…/native-execution.test.ts• L244-253 packages/…/native-execution.test.ts• L375-525 packages/…/native-execution-seeded.json• L1-30 packages/…/native-execution-seeded.json• L32-261
Workflow evaluation
Workflow evaluation checks behavior across a run instead of checking one action at a time. Sanitized provider fixtures feed a common observation model, while live campaigns use qualified provider candidates under explicit limits.
Workflow catalog
The workflow catalog defines provider-neutral flows for response output, activity, verification, governed interaction, steering, planning, review, delegation, completion recovery, restart recovery, cancellation, and trace lineage. Each workflow states its steps, expected issue and run outcomes, required operations, required or forbidden protocol events, visible activity, and trace expectations.
References: packages/…/workflow-catalog.ts• L8-21 packages/…/workflow-catalog.ts• L150-212 packages/…/workflow-contracts.ts• L30-64
Workflow harness
The workflow harness builds a matrix from workflows and applicable provider candidates. It normalizes provider events and checks protocol events, activity order, lifecycle state, terminal evidence, recovery ownership, semantic receipts, and trace lineage.
The deterministic matrix includes Codex, OpenCode, and ACPX. Missing workflow evidence is a candidate failure. It is not a pass and does not receive a misleading successful score.
References: packages/…/workflow-harness.ts• L1-35 packages/…/workflow-harness.ts• L75-225 packages/…/workflow-evals.test.ts• L44-107
Live workflow campaigns
Live workflow evaluation runs selected cases against qualified provider and model candidates. The schedule uses stable seeds, weekly rotation, fixed candidate slots, and bounded time, attempt, token, and cost limits.
Candidate manifests exclude secrets and unqualified candidates. Retry applies only to retryable infrastructure failures. Candidate failures remain failures rather than being hidden by retry or scoring rules.
References: packages/…/live-workflow-matrix.ts• L10-25 packages/…/live-workflow-matrix.ts• L69-221 packages/…/workflow-evals.test.ts• L254-337 packages/…/workflow-evals.test.ts• L339-413
Scorecards and reports
The evaluator scores lifecycle integrity, semantic outcome, continuation, presentation, trace completeness, and related behavior dimensions. Lifecycle and other hard safety gates can force the overall score to zero. Skipped and infrastructure-failed executions are not scored.
Reports preserve per-case results, coverage, aggregate outcomes, comparisons, and alerts. A report comparison is compatible only when its evaluation bundle identity matches.
References: packages/…/workflow-scoring.ts• L14-29 packages/…/workflow-scoring.ts• L89-167 packages/…/workflow-report.ts• L18-43 packages/…/workflow-evals.test.ts• L442-502
Traceability artifacts
Traceability connects observed behavior to its source case, expected semantics, evidence IDs, regression anchor, and reproducibility inputs. These links help an engineer tell whether a change affected a contract, provider behavior, or an evaluation fixture.
Stress finding manifest
The stress finding manifest maps known findings to workflows, regression tests, or an explicit exclusion. It is a build gate that rejects unknown workflows, missing test anchors, invalid paths, and exclusions without reasons.
References: packages/…/workflow-traceability.ts• L3-19 packages/…/workflow-traceability.ts• L29-73 packages/…/stress-workflow-traceability.json• L1-18
Evaluation bundles
An evaluation bundle declares the inputs that make one run comparable. It includes the provider and transport, model, launch context, prompt policy, grants, runner identity, control-plane adapter, and deterministic faults. Its content-based ID changes when a declared input changes.
Bundles hold declarations and digests rather than credentials. Secret-shaped keys and values are rejected so evaluation evidence does not become a second secret store.
References: packages/…/eval-bundle.ts• L5-18 packages/…/eval-bundle.ts• L83-133 packages/…/eval-bundle.ts• L201-309
Attempt journals
An attempt journal stores durable controller-owned evidence for one evaluation attempt. It is bounded so evidence growth stops provider dispatch before the attempt can consume unbounded storage.
References: packages/…/attempt-journal.ts• L3-22
Evaluation kernel
The evaluation kernel is the shared execution layer that runs a scenario against a candidate and returns a normalized attempt outcome. Runner-owned contracts define workflow meaning, observations, scores, reports, and trace links around this common layer.
The kernel uses one session boundary for local, remote, and mock execution. A harness driver or native session backend opens or recovers a session, emits normalized events, starts and controls turns, resolves runtime requests, snapshots durable state, and closes provider work. The same boundary also defines cancellation and recovery authority, so cleanup cannot publish new results after a turn is revoked. A failed provider terminal remains distinct from a missing completion proposal, and a pending side effect stops unsafe session replacement. Runtime request validation is tied to the request kind, so approval, user input, and elicitation cannot silently accept one another's payloads. Durable handoff commits control-plane state before provider cleanup.
The test transport supplies the same runtime context and prompt digest used by native execution, waits through the bounded startup notification prefix, and exposes a committable control-plane store for deterministic cases. This boundary supports offline matrix runs and controlled live campaigns. The deterministic path can run without network access or production provider startup. The live path adds qualified credentials, provider admission, durable session limits, and runtime deadlines without changing the scenario and candidate model.
References: packages/…/workflow-harness.ts• L1-35 packages/…/workflow-harness.ts• L75-225 packages/…/harness-driver.ts• L17-64 packages/…/harness-driver.ts• L196-320 packages/…/harness-driver.ts• L601-690 packages/…/native-session-backend.ts• L49-78 packages/…/native-session-backend.ts• L140-237 packages/…/native-session-backend.ts• L239-389 packages/…/runnerd-transport.ts• L14-30 packages/…/runnerd-transport.ts• L46-97 packages/…/private-host-control.ts• L55-143
Change checklist
When these files or sections are changed, remember to consider these:
- Change a canonical operation or its projection together with scenario and live reconciliation, claims, task modes, exposure, legacy alias folds, and capability evaluation cases.
- Change a semantic action declaration together with its operation ID type, strict input and output schemas, generated inventory, and the rule that authorization stays outside the catalog.
- Change a native execution fixture together with its seeded provenance, runtime-context digest, provider profile identity, completion contract, semantic call and receipt links, terminal state, transcript and artifact evidence, and event order.
- Change provider profile fixtures or their history together with current-candidate admission, historical recovery, profile lineage, and profile-specific capability claims. A qualification identity must not become launch authority only because it is present in a fixture.
- Change replay or workflow contracts together with version checks, event identity, terminal evidence, matrix expectations, and report schema validation.
- Change ACPX sidecar or protocol contract rules together with shared family versions, closed envelopes, candidate admission identity, runtime context, question adapter mapping, external-provider operation bindings and action inputs, provider-started turn signaling, and conformance fixture pairing.
- Change native test transport or host-control behavior together with process identity checks, bounded control frames, startup notifications, exit confirmation, cleanup ownership, and runtime-context digests.
- Change live schedule rules together with candidate qualification, rotation coverage, selection validation, retry behavior, runtime deadlines, and cost limits.
- Change traceability findings together with their workflow IDs, regression paths, exclusion reasons, and coverage totals.
- Change evaluation bundle inputs together with bundle identity so report comparisons do not mix unlike runs.
- Change scoring or classification rules together with gate behavior and the treatment of skipped, infrastructure, and candidate failures.
- Change provider fixtures together with authentication, assigned MCP bindings, stream ordering, late-event handling, terminal receipts, runtime request variants, and secret-safe diagnostics.
- Change provider completion compatibility or canonical result rules together with normalization, bridge validation, semantic tool descriptions, yielded-only continuation handling, and the completion conformance cases.
- Change harness or native session lifecycle contracts together with runtime-request payload validation, cancellation authority, recovery failure classes, terminal projection, snapshot identity, and cleanup behavior.
- Change the capability source inventories together with generated capability counts, evaluation groups, primary dispositions, and legacy alias indexes.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| A catalog projection diverges from the canonical operation set | An action can appear covered on one surface while its claims or mode differ on another. | Reconcile operation membership, claims, task modes, placement, aliases, and generated contracts. |
| A semantic action declaration is treated as permission | A caller can discover or invoke an action without the run's authority. | Apply run-scoped authorization after catalog lookup. |
| A yielded completion omits an eligible continuation or uses one for a completed result | The controller can lose a required wake or treat a finished turn as waiting. | Accept omission only at the provider boundary, then resolve it from current run-owned state and enforce strict yielded-only canonical validation. |
| A replay contains a changed duplicate or source gap | Recovery can apply conflicting or incomplete history. | Enforce canonical duplicate equality and contiguous source evidence. |
| A workflow fixture lacks lifecycle or terminal evidence | Partial provider activity can be marked complete. | Classify incomplete fixture evidence as a candidate failure. |
| A provider profile or installation changes under an old identity | Evaluation can run an unqualified provider or change behavior during recovery. | Admit only the exact qualified profile and preserve historical identity for recovery decode. |
| A candidate admission record is copied to another run, session, platform, or expiry window | A sidecar can start with authority that belongs to a different evaluation. | Validate the closed admission record against provider identity, run identity, session identity, platform, and expiry. |
| An external-provider command has a wrong binding, digest, action input, or extra field | A provider operation can cross an evaluation boundary or drift from the action contract. | Validate the closed command payload and its action-specific input against the shared schema. |
| An ACPX family version differs from the schema identity or a closed envelope gains an unsupported field | Provider messages fail validation or protocol drift passes unnoticed. | Require one shared version and reject unknown commands, fields, and payload shapes. |
| A conformance input and output use different run identity | A result can be compared with evidence from another execution. | Validate the fixture pair and require one stable run identity. |
| A large provider frame is compacted without preserving its outcome or neighbors | Trace meaning or event order can change while the stream still parses. | Bound text with omission metadata and re-check canonical event projection. |
| A runtime request is resolved with the wrong payload shape | The provider can receive an empty or misread answer while the control plane records success. | Validate the resolution against the request kind and persist the canonical outcome. |
| A native child exits before the host-control fixture confirms its identity or exit result | The test can report a false success or signal an unrelated reused process during cleanup. | Require bounded startup and exit receipts, match the process generation, and signal only an owned process group. |
| A candidate or schedule selector is unknown | A live run can use an unintended pairing or no valid work. | Validate candidate and case IDs and require a non-empty selection. |
| A retryable infrastructure failure is scored as candidate behavior | Provider outage can distort quality results. | Retry only allowed infrastructure failures and leave exhausted infrastructure unscored. |
| A bundle input changes without a new bundle ID | Historical report comparisons mix different runs. | Derive the ID from canonical declared inputs. |
| A stress finding loses its workflow or test anchor | Known behavior can leave the regression set. | Require valid anchors, exclusion reasons, and full workflow coverage. |
| Evaluation evidence contains credential-shaped data | Reports or journals can leak secrets. | Reject secret-shaped bundle data and keep journals bounded. |
| A provider completion uses a legacy alias without canonical normalization | A valid provider report can fail strict evaluation or carry the wrong continuation meaning. | Accept only known compatibility forms, normalize them, and apply canonical disposition rules before scoring. |
| A session cleanup or recovery path publishes after authority is revoked | A cancelled or replaced turn can alter durable state or duplicate a side effect. | Commit revocation first, keep cleanup passive, and stop replacement when a side effect has no settled result. |
Key files
| File | Why to read it |
|---|---|
| packages/…/semantic-action-catalog.ts | Defines the immutable native action declarations. |
| packages/…/operation-groups-doc.ts | Shows how protocol families, capability placement, behavior groups, aliases, and drift checks remain separate. |
| packages/…/capability-contract.ts | Publishes the generated capability inventories and evaluation coverage contract. |
| packages/…/semantic-conformance.ts | Defines the normalized semantic result used to compare adapters. |
| packages/…/replay-contract.ts | Defines replay validation, binding checks, and version handling. |
| packages/…/completion-result.ts | Defines provider-compatible completion input and strict canonical result rules. |
| packages/…/native-execution.ts | Defines versioned native launch input and recovery identity. |
| packages/…/native-session-backend.ts | Defines the native session lifecycle, cancellation authority, and recovery outcomes. |
| packages/…/native-execution-seeded.json | Shows a deterministic native attempt with a denied effect and linked evidence. |
| packages/…/command-v3.schema.json | Defines command identity, external-provider operation bindings, action inputs, and provider-started turn opt-in. |
| packages/…/harness-driver.ts | Defines the shared harness session and runtime-request boundary. |
| packages/…/workflow-catalog.ts | States provider-neutral workflow expectations. |
| packages/…/workflow-harness.ts | Builds deterministic workflow observations from provider fixtures. |
| packages/…/workflow-evals.test.ts | Shows matrix, schedule, report, score, schema, and traceability guarantees. |
| packages/…/eval-bundle.ts | Defines reproducibility inputs and secret-safe bundle identity. |
| packages/…/workflow-traceability.ts | Validates links from stress findings to workflows and tests. |
| packages/…/acpx-sidecar-contract.test.mjs | Shows ACPX envelope, identity, version, runtime-context, provider-option, and profile contract checks. |
| packages/…/protocol-contract.test.mjs | Shows protocol schema, replay, question adapter, external operation, and conformance-pair checks. |
| packages/…/fake-opencode-server.mjs | Provides controlled OpenCode behavior for authentication, requests, streams, MCP use, diagnostics, recovery, and terminal evaluation. |
| packages/…/private-host-control.ts and packages/…/private-host-control.py | Provide bounded host control with process-generation checks for native process tests. |
| packages/…/runnerd-transport.ts | Supplies qualified runner transport, startup expectations, runtime context, and a committable store for deterministic tests. |