Introducing a new way to review your PRs.

Read the announcement

affaan-m / ecc

The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.

View on GitHub

30 documents · Updated Nov 20, 2026

docs / evaluation-gates-capsules-and-replay

Evaluation gates, capsules, and replay

The evaluation harness gives an agent run a stable input set, a bounded record of what happened, and evidence that another machine can check later. It separates source inspection from candidate execution, keeps side effects explicit, and makes local tool results replayable without contacting live systems. For the wider evidence model, see Evaluation harness and evidence.

Evaluation model

Evaluation inputs

An evaluation starts with a task set and one or more variants. A task set names a version and family, then gives each task a unique id, an input, and an expected value. A variant names its source, declares an effect class, and points to an entry file inside its own tree. The harness hashes both inputs so later records can identify the exact data and source that were inspected. examples/…/taskset.json, scripts/…/gate.js

The example keeps twelve slugify tasks. Three are marked held out. This marks data for a possible evaluation split; it does not claim that a candidate ran or earned a score.

References: examples/…/taskset.json• L1-19 scripts/…/gate.js• L109-147

Evaluation stages

The harness treats inspection, execution, replay, and evidence as separate stages. Inspection can load a variant and task set without loading candidate code. Execution is a separate gate that currently stops before it reads the gate configuration or creates work files because no verified operating-system containment backend exists. Replay and receipt work remain available for local, bounded evidence.

The main relationship between these stages is shown below.

flowchart LR
  I[Canonical task set] --> V[Variant inspection]
  V --> D[Source and task digests]
  D --> G[Execution gate]
  G -->|verified containment required| R[Candidate results]
  G -->|current result| B[Refusal record]
  R --> C[Capsule]
  B --> C
  C --> P[Projection]
  P --> E[Receipt and export bundle]
  T[Declared tools] --> F[Fixture record or replay]
  F --> C

References: scripts/eval-harness.js• L7-16 scripts/…/gate.js• L1-18 examples/…/run-example.js• L51-65

Capsules and envelopes

Canonical records

Canonical JSON is the shared byte form for hashes in journals, projections, fixtures, and receipts. It sorts object keys at every level, omits undefined object values, emits compact UTF-8 JSON, and then applies SHA-256. Two producers with the same data therefore produce the same bytes and digest.

This rule is a cross-system contract. A change to canonicalization changes fixture keys and evidence hashes even when the visible data looks the same.

References: scripts/…/canonical.js• L3-10 scripts/…/canonical.js• L14-45

Capsule journal

A capsule is the local record for one run. It binds a run_id, capsule_id, harness version, task family, and creation time to an append-only journal. Each journal entry has one of five lineages: plan, attempt, interaction, environment, or strategy. The lineages let later readers relate a decision to the plan, work, tool use, environment, or strategy that produced it.

Each entry points to the previous entry with parent_hash. The first entry points to a zero hash. Verification checks the entry order, metadata identity, canonical bytes, and entry hashes. A changed, missing, reordered, or partial entry makes verification fail at the first affected position. A projection summarizes counts, the last sequence number, the root hash, and the journal digest.

Capsule export is deliberately small. It writes only capsule.json, journal.ndjson, and projection.json; it does not copy the workspace that the run touched.

References: scripts/…/capsule.js• L5-17 scripts/…/capsule.js• L124-171 scripts/…/capsule.js• L244-366 scripts/…/capsule.js• L383-395

Envelope boundary

The envelope is the record format for one capsule entry. It carries run identity, sequence, time, lineage, entry kind, effect class, the predecessor link, the entry hash, and a small payload. The effect classes range from SE0 for read-only work to SE4 for economic or external effects.

Payload fields use a default-deny allowlist. The contract keeps hashes, statuses, counts, scores, variants, fixture keys, and similar summaries. It rejects invalid types, unknown fields in strict mode, secret-like values, credentials, and raw reasoning text before the entry is written. The JSON schema mirrors this boundary for consumers outside the library.

References: scripts/…/envelope.js• L4-11 scripts/…/envelope.js• L16-31 scripts/…/envelope.js• L137-158 scripts/…/envelope.js• L161-235 schemas/capsule-envelope.schema.json• L7-42

Gate evaluation

Variant inspection

Variant inspection answers which source tree and task data are being considered. It requires a regular-file tree with a variant.json manifest. The manifest supplies a safe name, an effect class, and an optional relative entry path that defaults to run.js. The harness computes a digest from sorted relative paths and file bytes.

Static tripwires then look for signs such as hidden network access, process spawning, sandbox weakening, checker probing, parent-directory escape, or an effect class above the configured limit. These findings are warnings from source inspection. A clean scan is not proof of containment and does not prove that a candidate is correct.

References: scripts/…/gate.js• L55-106 scripts/…/gate.js• L109-132 scripts/…/gate.js• L150-171

Gate execution

The gate run command is the intended boundary for running a baseline and a candidate against a task set. Its current contract is refusal: requireSupportedIsolation() raises gate.isolation_required before configuration, capsule, work-directory, or candidate-code access. The same refusal applies to the library gate, the retired child runner, and the retired effect-fence preload.

No trusted_local option, command flag, copied directory, or JavaScript interception can replace operating-system containment. Because execution is unavailable, the example produces no candidate score, gate receipt, or promotion verdict. It records the refusal as capsule evidence instead.

References: scripts/eval-harness.js• L84-88 scripts/…/gate.js• L174-181 scripts/…/gate.js• L239-256 scripts/…/gate-child.js• L1-5 scripts/…/effect-fence.js• L1-5 examples/…/README.md• L9-19

Variant example

The example compares a baseline, an honest candidate, and a reward-hack source tree through inspection only. The baseline and candidate have different source digests. The reward-hack tree produces visible hidden_network and checker_probe findings. The harness never executes any of these variants, so the example demonstrates variant identification and risk reporting rather than model scoring.

The preserved gate configuration shows the intended evaluation inputs: a task set, baseline, candidate, an SE1 effect limit, and thresholds for smoke tasks, pass rate, regressions, and timeout. Running that configuration through gate run still stops at the isolation boundary.

References: examples/…/run-example.js• L42-75 examples/…/gate.config.json• L1-12 examples/…/README.md• L24-33

Replay and security fences

Fixture replay

Replay replaces live tool calls with content-addressed local fixtures. Each tool must declare its effect class and whether it is deterministic or nondeterministic. In record mode the implementation runs once and stores the response under a hash of the tool name and arguments. In replay mode the implementation is never called; the store returns the matching response or fails closed.

The fixture record includes hashes for the arguments and response. A missing, corrupt, mismatched, or tampered fixture is an error. This keeps a replayed result tied to the exact call that produced it.

References: scripts/…/replay.js• L3-16 scripts/…/replay.js• L33-83 scripts/…/replay.js• L86-145

Effect fences

Effect classes limit which tools a replayer may use. The configured maximum applies in both modes. Replay also refuses every SE3 and SE4 tool even when a fixture exists because remote publication and external or economic effects are not replay-safe. An undeclared tool is refused as unknown.

The example records a local read_inventory call and replays it from a fixture. A call with new arguments fails with tool.fixture_missing. A place_order tool is refused with tool.effect_forbidden because it is SE4. The interaction entries keep only the tool name, status, and content hashes.

References: scripts/…/replay.js• L117-143 examples/…/run-example.js• L77-105 scripts/…/envelope.js• L21-27

Security fences

Security controls fail closed at multiple boundaries. Envelope writes deny unapproved payload data and secret canaries. Variant inspection refuses links and special files. Gate execution refuses until a verified OS boundary exists. The retired preload refuses before supplied code can run.

These controls reduce accidental leakage and unsafe execution. They do not attest that a caller has not replaced an entire capsule or that a clean static scan is safe. A hash chain protects the checked journal history; it does not provide external witnessing by itself.

References: scripts/…/envelope.js• L73-109 scripts/…/capsule.js• L15-17 scripts/…/gate.js• L73-95 scripts/…/replay.js• L147-152

Receipts and replayable results

Receipt contract

A receipt binds an evaluation artifact and optional gate evidence to a verified capsule snapshot. It names the capsule root, entry count, journal digest, projection hash, artifact digest, gate-receipt digest, and gate verdict. Its own hash covers the receipt content. A caller may add a detached signature through a signer function; the receipt library does not create or store keys.

A signature proves who vouched for the receipt bytes. It does not prove that a candidate was correct or that an unavailable gate was completed.

References: scripts/…/receipt.js• L3-14 scripts/…/receipt.js• L24-63

Receipt verification

Receipt verification checks the receipt schema and hash first. It then checks the journal, metadata, capsule prefix, raw journal digest, persisted projection, and any supplied artifact or gate receipt. It can run against an exported capsule bundle without access to the original source store.

Verification distinguishes a truncated journal from a journal that advanced after the receipt. The latter is a stale checkpoint, not a valid receipt for the current capsule. Verification does not repair a missing or altered projection; it reports the failed check.

The example builds a receipt for the candidate artifact, exports the capsule bundle, and verifies the bundle offline. It then changes one journal value. Capsule verification identifies the affected entry, and receipt verification reports journal_integrity.

References: scripts/…/receipt.js• L81-165 scripts/…/capsule.js• L377-395 examples/…/run-example.js• L107-135

Command and library interfaces

Command interface

The CLI is scripts/eval-harness.js. It prints JSON results and uses exit code 0 for a verified result, 1 for failed verification or an unavailable gate, and 2 for a usage error. It validates option values before reading receipt inputs or writing a projection.

CommandPurpose
node scripts/eval-harness.js capsule verify <dir>Verify capsule metadata and journal integrity.
node scripts/eval-harness.js capsule project <dir>Build the deterministic capsule projection.
node scripts/eval-harness.js capsule export <dir> <out-dir>Export the three-file capsule bundle.
node scripts/eval-harness.js gate run <config>Request gate evaluation; currently refuses with gate.isolation_required.
node scripts/eval-harness.js receipt build <capsule-dir>Build and optionally write an offline receipt.
node scripts/eval-harness.js receipt verify <receipt.json> <capsule-dir>Verify a receipt against a capsule or bundle.
node scripts/eval-harness.js exampleRun the offline demonstration.

References: scripts/eval-harness.js• L7-16 scripts/eval-harness.js• L64-126 scripts/eval-harness.js• L128-145

Library interface

The library exposes the same contracts without the CLI. canonical supplies stable bytes and hashes. envelope validates entry shape and payload safety. capsule creates, opens, appends, verifies, projects, and exports journals. gate inspects variants and task sets while refusing execution. replay records or replays declared tools. receipt builds and verifies offline evidence.

Use the library when a caller must add capsule entries during a workflow or connect replay events to a larger run record. Use the CLI when a human or automation step needs a stable JSON result and process exit status.

References: scripts/…/index.js• L3-21 scripts/…/capsule.js• L124-242 scripts/…/replay.js• L86-145 scripts/…/receipt.js• L97-180

Change checklist

When these files or sections are changed, remember to consider these:

  • Update the JSON schema and every envelope consumer when lineages, effect classes, payload fields, or identity rules change.
  • Treat any canonical JSON change as a compatibility change for journal hashes, fixture keys, projection hashes, and receipt hashes.
  • Keep variant digest rules aligned with manifest entry checks and task-set digests. A source tree must not be inspected under one digest and identified under another.
  • Preserve refusal before configuration or candidate access until a verified OS containment backend exists. Do not restore trust flags or JavaScript-only fences as substitutes.
  • Keep replay limits aligned with effect classes. A fixture must not make SE3 or SE4 behavior replayable.
  • Keep receipt creation and verification aligned with the persisted projection, journal prefix, artifact digest, and optional gate-receipt digest.

Important failure modes

TriggerConsequenceGuard
A journal entry is changed, reordered, or cut offCapsule verification fails at the first affected entry.Canonical entry hashes, sequence checks, predecessor links, and terminating-newline checks.
Capsule metadata identity differs from a journal entryThe capsule and any receipt are rejected as mismatched.Metadata validation binds run_id, capsule_id, harness version, and task family.
A payload contains an unknown field or secret-like valueThe entry is not written.Default-deny payload allowlist and secret canary scan.
A variant contains a tripwire or exceeds the effect limitStatic findings are reported; a clean result still does not prove containment.Source digest, regular-file inspection, tripwire scan, and explicit effect-class comparison.
Gate execution is requested without verified OS containmentNo candidate code, work directory, or gate receipt is created.gate.isolation_required refusal before gate input access.
Replay has no matching or valid fixtureThe live tool is not used and replay fails closed.Content-addressed fixture lookup with argument and response hash checks.
Replay requests an SE3 or SE4 toolThe tool call is refused even if a fixture exists.Maximum effect class and replay-specific remote-effect ban.
The journal advances after a receipt is builtThe receipt becomes a stale checkpoint.Receipt verification compares the recorded entry count and journal prefix.

Key files

FileWhy to read it
scripts/eval-harness.jsDefines the user-facing command groups, output shape, and exit behavior.
scripts/…/index.jsShows the library contract exposed to other code.
scripts/…/canonical.jsDefines the shared byte and hash rules.
scripts/…/envelope.jsDefines entry identity, payload limits, lineages, effect classes, and secret checks.
schemas/capsule-envelope.schema.jsonStates the portable envelope shape for external consumers.
scripts/…/capsule.jsDefines journal integrity, projections, and minimal bundle export.
scripts/…/gate.jsDefines variant and task-set inspection plus the execution refusal boundary.
scripts/…/replay.jsDefines declared tool calls, fixture storage, and fail-closed replay.
scripts/…/receipt.jsDefines offline receipt creation and verification.
examples/…/run-example.jsConnects inspection, refusal evidence, replay, receipt export, and tamper detection in one flow.
examples/…/gate.config.jsonShows the intended variant gate inputs and thresholds.
examples/…/taskset.jsonShows the canonical task-set shape and held-out task marker.