Evaluation harness and evidence
The evaluation harness provides a controlled path for inspecting proposed variants, limiting their possible effects, replaying tool calls, and producing evidence that another machine can check. It records what the harness knows about a run without treating a score, a signature, or a hash as proof that the evaluated work is correct. Candidate execution and promotion are currently unavailable because the repository has no verified operating-system containment backend.
System purpose
Evaluation boundary
The harness separates evaluation from trust. It can inspect a variant and its task set, but it refuses to load or run candidate code until an external containment layer is available. Static warnings can show likely unsafe patterns, yet no warning-free result proves that code is safe or that a variant deserves promotion.
The refusal happens before the harness reads gate configuration, copies files, loads candidate modules, or writes a gate receipt. Caller flags and JavaScript wrappers cannot override this boundary. This keeps an incomplete safety mechanism from being mistaken for a sandbox.
References: scripts/…/gate.js• L1-18 scripts/…/gate.js• L174-181 scripts/eval-harness.js• L92-96
Evaluation flow
The evaluation flow turns a declared task and variant into either safe inspection data or a blocked run. When execution becomes available, the same boundary is intended to connect controlled inputs to a checker while keeping the checker and evidence outside candidate control.
flowchart LR
T[Task set] --> I[Static inspection]
V[Variant] --> I
I --> B{Containment available?}
B -->|No| X[Refuse execution]
B -->|Yes| R[Controlled evaluation]
R --> C[Capsule events]
X --> C
C --> P[Projection and receipt]
P --> Q[Offline verification]
The harness records blocked or inspected activity so later automation can distinguish “not evaluated” from “evaluated and passed.” The child page Evaluation gates, capsules, and replay describes the gate, capsule, and replay contracts in more detail.
References: scripts/…/index.js• L3-13 examples/…/README.md• L9-19 docs/…/eval-harness-frameworks.md• L14-33
Controlled variants
Variant inspection
A variant is a proposed implementation paired with a manifest, an entry file, an effect class, and a content digest. A task set supplies the named inputs and expected outputs used by a future checker. Inspection confirms that the variant and task set have a stable identity before any evaluation decision uses them.
Inspection rejects missing or malformed manifests, unsafe entry paths, symbolic links, duplicate task IDs, and task records without the required input and expected value. It also scans source text for a small set of tripwires such as hidden network access, process creation, checker probing, parent-directory escape, and effect-class expansion. These checks are signals for review, not containment.
References: scripts/…/gate.js• L97-147 scripts/…/gate.js• L150-172 examples/…/taskset.json• L1-8
Effect boundaries
An effect class is a declared level of side effect. The same vocabulary is used for journal entries, variants, and tools so that a run can state what it may affect and replay can apply one consistent limit.
| Class | Meaning | Harness treatment |
|---|---|---|
SE0 | Read-only evaluation | Allowed throughout the local harness |
SE1 | Reversible local writes | Used for local evidence and capsule data |
SE2 | Sandboxed process or file changes without live network writes | Requires a real containment backend for candidate execution |
SE3 | Append-only remote evidence publication | Refused during replay |
SE4 | Economic, external, counterparty, or secret-handling effects | Refused during replay and should not be enabled for untrusted work |
These classes describe policy. They are not operating-system permissions. A declaration can expose a policy mismatch, while only an external containment system can enforce what code can access.
References: scripts/…/envelope.js• L16-27 scripts/…/envelope.js• L189-203 schemas/capsule-envelope.schema.json• L31-38
Replayable work
Tool replay
Replay makes a tool call repeatable without calling the live tool again. In record mode, a trusted implementation produces a response and the harness stores it as a fixture. In replay mode, the harness returns only the fixture that matches the tool name and argument content.
Each fixture has hashes for its arguments and response. A missing, changed, or corrupt fixture stops the replay instead of falling back to a live call. Tools with SE3 or SE4 effects remain forbidden in replay even when a fixture exists, because replay must not turn an external effect into an apparently safe local action.
References: scripts/…/replay.js• L3-16 scripts/…/replay.js• L33-83 scripts/…/replay.js• L117-143
Replay contracts
Every replayable tool declares its effect class and whether its result is deterministic or not. The declaration lets the harness reject unknown tools and effects before work starts. The implementation supplied to record mode remains trusted caller code; the replay wrapper does not sandbox arbitrary code.
The exported effect-fence entry point is retired. Loading it refuses before supplied code runs, so it cannot be used as evidence of process isolation. Use the replay contract for repeatable data and a reviewed OS boundary for untrusted execution.
References: scripts/…/replay.js• L86-107 scripts/…/replay.js• L123-143 scripts/…/effect-fence.js• L1-5
Evidence records
Capsule journal
A capsule is the local record of a harness run. It groups events into five lineages: plan, attempt, interaction, environment, and strategy. Each event has a bounded envelope with run identity, effect class, and an allowlisted payload so the record carries useful facts without accepting raw secrets or unrestricted reasoning text.
The journal is append-only and hash-linked. Each entry points to the previous entry, and verification stops at the first invalid, reordered, or truncated entry. A deterministic projection summarizes the verified journal. An export contains the capsule metadata, journal, and projection rather than the workspace touched by the run.
References: scripts/…/envelope.js• L3-11 scripts/…/envelope.js• L18-27 scripts/…/capsule.js• L3-17 scripts/…/capsule.js• L244-295
Retrospective grouping
A retrospective report groups selected, verified capsule snapshots for offline comparison. It is report-only: it counts declared entries by task family, harness version, lineage, and effect class without scoring work, copying payloads, invoking tools, or producing promotion evidence.
The grouping process accepts a bounded set of capsule directories and removes exact duplicate identities only when their verified projections agree. It refuses mixed task families and conflicting checkpoints. The report includes source hashes so another process can identify the snapshots used for the summary.
References: scripts/…/retrospective.js• L3-5 scripts/…/retrospective.js• L58-99 scripts/eval-harness.js• L65-73
Evidence minimization
The envelope is a record boundary, not a transcript store. Its default-deny payload policy drops or rejects fields outside the known set, and secret canaries refuse common credential shapes before an entry is written. This prevents a receipt bundle from becoming an accidental store for credentials or private reasoning.
Canonical JSON gives equivalent records the same bytes and hash. That property supports stable projections, fixture keys, and receipt checks, but it does not recover data that was never recorded or prove that an operator did not replace an entire local log.
References: scripts/…/envelope.js• L67-88 scripts/…/envelope.js• L137-158 scripts/…/canonical.js• L3-45
Verifiable evidence
Receipt scope
A receipt binds the evidence to a capsule and to the artifact under review. It records the capsule root, journal digest, entry count, projection hash, artifact digest, and optional gate-receipt digest. A detached signature can identify the party that vouched for those bytes.
Receipt verification reads the supplied bundle and checks its schema, receipt hash, journal integrity, capsule identity, projection, artifact, and gate receipt. It does not repair missing files or regenerate a missing projection. Verification can therefore run on an exported bundle without access to the original source store.
References: scripts/…/receipt.js• L3-14 scripts/…/receipt.js• L31-63 scripts/…/receipt.js• L97-165
Evidence limits
Hashes and signatures establish that the checked bytes match the named bytes. They do not prove that the evaluation was correct, that a candidate was safely contained, or that a promotion decision was justified. A receipt with no gate receipt is evidence of the capsule and artifact relationship only; it is not a promotion result.
The current example demonstrates inspection, refusal, fixture replay, tamper detection, and offline receipt verification. It deliberately emits no candidate score, gate receipt, or promotion verdict because candidate execution is disabled.
References: scripts/…/receipt.js• L7-14 examples/…/README.md• L9-19 docs/…/eval-harness-frameworks.md• L297-307
Operational interfaces
Command interface
The CLI exposes capsule verification, projection, export, report-only grouping, gate refusal, receipt build, receipt verification, and the local example. Successful verification returns exit code 0; failed verification and unavailable gate execution return exit code 1; malformed command use returns exit code 2.
Use the CLI to move evidence between local stages. Use capsule grouping only for selected verified snapshots that share a task family. Treat a gate command that returns gate.isolation_required as an explicit blocked state, not as a failed candidate score.
References: scripts/eval-harness.js• L7-17 scripts/eval-harness.js• L65-96 scripts/eval-harness.js• L129-143
Change checklist
When these files or sections are changed, remember to consider these:
- Changes to envelope fields or effect classes must stay aligned with the JSON schema, capsule validation, replay policy, and any stored evidence that uses their hashes.
- Changes to canonical JSON or hash inputs can change fixture keys, journal entry hashes, projections, retrospective source hashes, and receipts. Preserve or deliberately version those contracts.
- Changes to variant or task-set validation must keep source digests, entry paths, tripwire results, and future checker inputs tied to the same content.
- Changes to retrospective grouping must preserve same-family inputs, duplicate identity handling, harness-version grouping, and report-only status.
- Changes to receipt fields or verification order must preserve the ability to verify exported bundles without the producer's source store.
- Any new execution path must prove OS containment before candidate code, gate configuration, or receipt creation can run. Do not use trust flags or runtime wrappers as substitutes.
- Any new tool must declare its effect class and replay behavior. External or money-touching effects must remain outside permissive replay.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| Candidate execution is requested without a verified containment backend | The harness refuses before loading code or writing a gate receipt | gate.isolation_required is unconditional; treat the run as unavailable |
| A variant exceeds the allowed effect class or contains a static tripwire | Inspection reports a policy hit and cannot support a clean evaluation claim | Review the hit and keep the execution boundary closed |
| A replay fixture is missing, changed, or corrupt | Replay stops instead of calling the live tool or returning untrusted data | Content-address fixtures and fail closed on lookup or hash mismatch |
| Selected retrospective capsules use different task families | No report is produced | Group only verified snapshots from one task family |
| Duplicate capsule identities have different projections | The retrospective report refuses the ambiguous snapshots | Resolve the conflicting checkpoint before grouping |
| A journal entry is changed, reordered, or truncated | Capsule verification fails at the affected entry | Verify the journal before projection, export, grouping, or receipt build |
| A receipt names an advanced journal or mismatched artifact | Verification reports a stale checkpoint or artifact failure | Verify the exact capsule bundle and artifact named by the receipt |
| A payload contains an unknown field or common secret shape | The event is rejected before it enters the journal | Use the allowlisted envelope and remove sensitive values |
Key files
| File | Why to read it |
|---|---|
| scripts/…/index.js | Entry point for the reusable harness areas, including retrospective reporting |
| scripts/…/gate.js | Defines variant inspection and the current disabled execution boundary |
| scripts/…/envelope.js | Defines event identity, effect classes, payload limits, and secret checks |
| scripts/…/capsule.js | Defines the journal, verification, projection, and export model |
| scripts/…/retrospective.js | Defines report-only grouping of verified capsule snapshots |
| scripts/…/replay.js | Defines fixture-backed tool replay and effect refusal |
| scripts/…/receipt.js | Defines receipt creation and offline verification |
| scripts/eval-harness.js | Shows the supported command and exit-code interface |
| schemas/capsule-envelope.schema.json | Defines the portable envelope contract shared by records and consumers |
| examples/…/README.md | Shows the supported local flow and its deliberate limits |