Introducing a new way to review your PRs.

Read the announcement

affaan-m / ecc

The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.

View on GitHub

30 documents · Updated Nov 20, 2026

docs / evaluation-harness-and-evidence

Evaluation harness and evidence

The evaluation harness provides a controlled path for inspecting proposed variants, limiting their possible effects, replaying tool calls, and producing evidence that another machine can check. It records what the harness knows about a run without treating a score, a signature, or a hash as proof that the evaluated work is correct. Candidate execution and promotion are currently unavailable because the repository has no verified operating-system containment backend.

System purpose

Evaluation boundary

The harness separates evaluation from trust. It can inspect a variant and its task set, but it refuses to load or run candidate code until an external containment layer is available. Static warnings can show likely unsafe patterns, yet no warning-free result proves that code is safe or that a variant deserves promotion.

The refusal happens before the harness reads gate configuration, copies files, loads candidate modules, or writes a gate receipt. Caller flags and JavaScript wrappers cannot override this boundary. This keeps an incomplete safety mechanism from being mistaken for a sandbox.

References: scripts/…/gate.js• L1-18 scripts/…/gate.js• L174-181 scripts/eval-harness.js• L92-96

Evaluation flow

The evaluation flow turns a declared task and variant into either safe inspection data or a blocked run. When execution becomes available, the same boundary is intended to connect controlled inputs to a checker while keeping the checker and evidence outside candidate control.

flowchart LR
    T[Task set] --> I[Static inspection]
    V[Variant] --> I
    I --> B{Containment available?}
    B -->|No| X[Refuse execution]
    B -->|Yes| R[Controlled evaluation]
    R --> C[Capsule events]
    X --> C
    C --> P[Projection and receipt]
    P --> Q[Offline verification]

The harness records blocked or inspected activity so later automation can distinguish “not evaluated” from “evaluated and passed.” The child page Evaluation gates, capsules, and replay describes the gate, capsule, and replay contracts in more detail.

References: scripts/…/index.js• L3-13 examples/…/README.md• L9-19 docs/…/eval-harness-frameworks.md• L14-33

Controlled variants

Variant inspection

A variant is a proposed implementation paired with a manifest, an entry file, an effect class, and a content digest. A task set supplies the named inputs and expected outputs used by a future checker. Inspection confirms that the variant and task set have a stable identity before any evaluation decision uses them.

Inspection rejects missing or malformed manifests, unsafe entry paths, symbolic links, duplicate task IDs, and task records without the required input and expected value. It also scans source text for a small set of tripwires such as hidden network access, process creation, checker probing, parent-directory escape, and effect-class expansion. These checks are signals for review, not containment.

References: scripts/…/gate.js• L97-147 scripts/…/gate.js• L150-172 examples/…/taskset.json• L1-8

Effect boundaries

An effect class is a declared level of side effect. The same vocabulary is used for journal entries, variants, and tools so that a run can state what it may affect and replay can apply one consistent limit.

ClassMeaningHarness treatment
SE0Read-only evaluationAllowed throughout the local harness
SE1Reversible local writesUsed for local evidence and capsule data
SE2Sandboxed process or file changes without live network writesRequires a real containment backend for candidate execution
SE3Append-only remote evidence publicationRefused during replay
SE4Economic, external, counterparty, or secret-handling effectsRefused during replay and should not be enabled for untrusted work

These classes describe policy. They are not operating-system permissions. A declaration can expose a policy mismatch, while only an external containment system can enforce what code can access.

References: scripts/…/envelope.js• L16-27 scripts/…/envelope.js• L189-203 schemas/capsule-envelope.schema.json• L31-38

Replayable work

Tool replay

Replay makes a tool call repeatable without calling the live tool again. In record mode, a trusted implementation produces a response and the harness stores it as a fixture. In replay mode, the harness returns only the fixture that matches the tool name and argument content.

Each fixture has hashes for its arguments and response. A missing, changed, or corrupt fixture stops the replay instead of falling back to a live call. Tools with SE3 or SE4 effects remain forbidden in replay even when a fixture exists, because replay must not turn an external effect into an apparently safe local action.

References: scripts/…/replay.js• L3-16 scripts/…/replay.js• L33-83 scripts/…/replay.js• L117-143

Replay contracts

Every replayable tool declares its effect class and whether its result is deterministic or not. The declaration lets the harness reject unknown tools and effects before work starts. The implementation supplied to record mode remains trusted caller code; the replay wrapper does not sandbox arbitrary code.

The exported effect-fence entry point is retired. Loading it refuses before supplied code runs, so it cannot be used as evidence of process isolation. Use the replay contract for repeatable data and a reviewed OS boundary for untrusted execution.

References: scripts/…/replay.js• L86-107 scripts/…/replay.js• L123-143 scripts/…/effect-fence.js• L1-5

Evidence records

Capsule journal

A capsule is the local record of a harness run. It groups events into five lineages: plan, attempt, interaction, environment, and strategy. Each event has a bounded envelope with run identity, effect class, and an allowlisted payload so the record carries useful facts without accepting raw secrets or unrestricted reasoning text.

The journal is append-only and hash-linked. Each entry points to the previous entry, and verification stops at the first invalid, reordered, or truncated entry. A deterministic projection summarizes the verified journal. An export contains the capsule metadata, journal, and projection rather than the workspace touched by the run.

References: scripts/…/envelope.js• L3-11 scripts/…/envelope.js• L18-27 scripts/…/capsule.js• L3-17 scripts/…/capsule.js• L244-295

Retrospective grouping

A retrospective report groups selected, verified capsule snapshots for offline comparison. It is report-only: it counts declared entries by task family, harness version, lineage, and effect class without scoring work, copying payloads, invoking tools, or producing promotion evidence.

The grouping process accepts a bounded set of capsule directories and removes exact duplicate identities only when their verified projections agree. It refuses mixed task families and conflicting checkpoints. The report includes source hashes so another process can identify the snapshots used for the summary.

References: scripts/…/retrospective.js• L3-5 scripts/…/retrospective.js• L58-99 scripts/eval-harness.js• L65-73

Evidence minimization

The envelope is a record boundary, not a transcript store. Its default-deny payload policy drops or rejects fields outside the known set, and secret canaries refuse common credential shapes before an entry is written. This prevents a receipt bundle from becoming an accidental store for credentials or private reasoning.

Canonical JSON gives equivalent records the same bytes and hash. That property supports stable projections, fixture keys, and receipt checks, but it does not recover data that was never recorded or prove that an operator did not replace an entire local log.

References: scripts/…/envelope.js• L67-88 scripts/…/envelope.js• L137-158 scripts/…/canonical.js• L3-45

Verifiable evidence

Receipt scope

A receipt binds the evidence to a capsule and to the artifact under review. It records the capsule root, journal digest, entry count, projection hash, artifact digest, and optional gate-receipt digest. A detached signature can identify the party that vouched for those bytes.

Receipt verification reads the supplied bundle and checks its schema, receipt hash, journal integrity, capsule identity, projection, artifact, and gate receipt. It does not repair missing files or regenerate a missing projection. Verification can therefore run on an exported bundle without access to the original source store.

References: scripts/…/receipt.js• L3-14 scripts/…/receipt.js• L31-63 scripts/…/receipt.js• L97-165

Evidence limits

Hashes and signatures establish that the checked bytes match the named bytes. They do not prove that the evaluation was correct, that a candidate was safely contained, or that a promotion decision was justified. A receipt with no gate receipt is evidence of the capsule and artifact relationship only; it is not a promotion result.

The current example demonstrates inspection, refusal, fixture replay, tamper detection, and offline receipt verification. It deliberately emits no candidate score, gate receipt, or promotion verdict because candidate execution is disabled.

References: scripts/…/receipt.js• L7-14 examples/…/README.md• L9-19 docs/…/eval-harness-frameworks.md• L297-307

Operational interfaces

Command interface

The CLI exposes capsule verification, projection, export, report-only grouping, gate refusal, receipt build, receipt verification, and the local example. Successful verification returns exit code 0; failed verification and unavailable gate execution return exit code 1; malformed command use returns exit code 2.

Use the CLI to move evidence between local stages. Use capsule grouping only for selected verified snapshots that share a task family. Treat a gate command that returns gate.isolation_required as an explicit blocked state, not as a failed candidate score.

References: scripts/eval-harness.js• L7-17 scripts/eval-harness.js• L65-96 scripts/eval-harness.js• L129-143

Change checklist

When these files or sections are changed, remember to consider these:

  • Changes to envelope fields or effect classes must stay aligned with the JSON schema, capsule validation, replay policy, and any stored evidence that uses their hashes.
  • Changes to canonical JSON or hash inputs can change fixture keys, journal entry hashes, projections, retrospective source hashes, and receipts. Preserve or deliberately version those contracts.
  • Changes to variant or task-set validation must keep source digests, entry paths, tripwire results, and future checker inputs tied to the same content.
  • Changes to retrospective grouping must preserve same-family inputs, duplicate identity handling, harness-version grouping, and report-only status.
  • Changes to receipt fields or verification order must preserve the ability to verify exported bundles without the producer's source store.
  • Any new execution path must prove OS containment before candidate code, gate configuration, or receipt creation can run. Do not use trust flags or runtime wrappers as substitutes.
  • Any new tool must declare its effect class and replay behavior. External or money-touching effects must remain outside permissive replay.

Important failure modes

TriggerConsequenceGuard
Candidate execution is requested without a verified containment backendThe harness refuses before loading code or writing a gate receiptgate.isolation_required is unconditional; treat the run as unavailable
A variant exceeds the allowed effect class or contains a static tripwireInspection reports a policy hit and cannot support a clean evaluation claimReview the hit and keep the execution boundary closed
A replay fixture is missing, changed, or corruptReplay stops instead of calling the live tool or returning untrusted dataContent-address fixtures and fail closed on lookup or hash mismatch
Selected retrospective capsules use different task familiesNo report is producedGroup only verified snapshots from one task family
Duplicate capsule identities have different projectionsThe retrospective report refuses the ambiguous snapshotsResolve the conflicting checkpoint before grouping
A journal entry is changed, reordered, or truncatedCapsule verification fails at the affected entryVerify the journal before projection, export, grouping, or receipt build
A receipt names an advanced journal or mismatched artifactVerification reports a stale checkpoint or artifact failureVerify the exact capsule bundle and artifact named by the receipt
A payload contains an unknown field or common secret shapeThe event is rejected before it enters the journalUse the allowlisted envelope and remove sensitive values

Key files

FileWhy to read it
scripts/…/index.jsEntry point for the reusable harness areas, including retrospective reporting
scripts/…/gate.jsDefines variant inspection and the current disabled execution boundary
scripts/…/envelope.jsDefines event identity, effect classes, payload limits, and secret checks
scripts/…/capsule.jsDefines the journal, verification, projection, and export model
scripts/…/retrospective.jsDefines report-only grouping of verified capsule snapshots
scripts/…/replay.jsDefines fixture-backed tool replay and effect refusal
scripts/…/receipt.jsDefines receipt creation and offline verification
scripts/eval-harness.jsShows the supported command and exit-code interface
schemas/capsule-envelope.schema.jsonDefines the portable envelope contract shared by records and consumers
examples/…/README.mdShows the supported local flow and its deliberate limits