Agent evaluation platform
The agent-eval application is a separate research and quality system for measuring how coding agents use Storybook. It runs repeatable tasks in isolated project sandboxes, checks the agent's work or records it for later study, and publishes results for local and CI review. It sits under developer workflows and complements project setup and migration, testing fixtures and sandboxes, repository automation, and ecosystem quality assurance.
Evaluation system
The platform separates an agent task from the project used to run it and from the way its result is judged. This lets one task run against several agents, Storybook information sources, content variants, or package versions without changing the task itself.
Evaluation lifecycle
An evaluation starts with a fixture that gives the agent a task and a starting project. The harness prepares a sandbox, runs the agent, saves the project and transcript, then either checks explicit workflow rules or sends the saved artifacts to offline analysis.
The main lifecycle is:
flowchart LR
F[Task fixture] --> S[Sandbox setup]
S --> A[Agent workflow]
A --> R[Saved transcript and project]
R --> G[Pass or fail checks]
R --> M[Offline metrics]
M --> C[Baseline or case comparison]
G --> P[Results playground and CI gate]
C --> P
References: agent-eval/README.md• L5-9 agent-eval/README.md• L138-151 agent-eval/…/experiment.ts• L193-265
Agent integrations
The platform compares how agents work with different Storybook support channels. A case can provide a Storybook MCP server, plugin skills, other MCP services, prompt guidance, or no support as a control. Linux sandboxes can also provide a compatible node_repl MCP service for Codex browser workflows when the native Codex service is not available. The recorded case keeps these choices with each result so later comparisons do not depend on current configuration.
The default suite covers Claude Code and Codex configurations. A case can pin the model and effort so a change in an agent's default does not change the experiment without notice. The evaluation project resolves the local Storybook MCP source through its type configuration, which keeps agent-facing support aligned with the checkout under test.
References: agent-eval/…/templates.ts• L29-33 agent-eval/…/cases.ts• L36-68 agent-eval/…/experiment.ts• L76-109 agent-eval/…/node-repl-mock.mjs• L1-14 agent-eval/…/node-repl-mock.mjs• L32-90 agent-eval/tsconfig.json• L15-25
Evaluation scenarios
The scenario library tests the agent behaviors that matter to Storybook users. Each scenario combines a task prompt with checks over the final project, shell activity, transcript, Storybook workflow calls, or test results. The workflow line also checks that agents use the supported Storybook workflow and produce a complete display review when the task changes or browses visual UI.
Workflow evaluations
The active 8xx line is the main product workflow suite. It checks work such as creating or editing components, writing stories for new, existing, fetching, and monorepo components, fixing tests and accessibility issues, using documentation and browsing tools, and running story tests. Visual-change cases also check story discovery before review publication, review coverage for the changed or newly created stories, and a review link in the final response. A browse request must publish the current stories with an empty change list. The 82x scenarios extend this line to Storybook project setup, upgrades, and outdated projects.
The 9xx line is a smaller MCP-focused set for the published latest Storybook release. It covers gaps such as asynchronous mocks, story drift, tool parameters, preview by path or ID, and Vitest command use. Its component-creation test case requires story creation, a focused story test, and a focused preview. This checks the complete agent path rather than only the resulting files. The line is selected separately because it measures the stable release rather than the checkout under test. The default run uses one core 8xx scenario; broader 8xx, 82x, or 9xx lines require explicit selection.
References: agent-eval/README.md• L32-83 agent-eval/README.md• L181-205 agent-eval/…/EVAL.ts• L17-64 agent-eval/…/EVAL.ts• L11-40 agent-eval/…/EVAL.ts• L4-18
Agentic-reference evaluations
The 70x line is a research suite for measuring the value of Storybook content in AI-assisted UI work. It does not reduce each run to a pass or fail. Instead, it compares a control case with cases that serve different documentation and story content through a design-system Storybook MCP.
The current workflow families are:
| Workflow | Question it represents |
|---|---|
701-new-ui-flow | How the agent creates new UI |
702-rework-ui-flow | How the agent reworks existing UI |
703-fix-bug-flow | How the agent fixes a UI bug |
704-fix-a11y-flow | How the agent addresses an accessibility issue |
706-new-ui-scheduled-flow | How a scheduled new-UI task behaves |
Cases vary the amount and type of documentation or stories. The control case has no agent support, while treatment cases can provide empty, docs-focused, stories-focused, or complete Storybook content. The research line captures the output for later metric and judge analysis instead of using the hand-crafted workflow gate.
References: agent-eval/…/cases.ts• L18-28 agent-eval/…/cases.ts• L94-157 agent-eval/…/cases.ts• L183-203 agent-eval/README.md• L389-423
Workflow outcome checks
The workflow cases judge both the result and the way the agent reaches it. Story tests must finish with the target stories passing and must use focused story input when the scenario requires it. Accessibility cases must observe the issue, keep the coverage that exposes it, and report unresolved visual contrast concerns instead of hiding them. Story-writing cases must model network data in stories without changing the component's fetch contract. These checks make the scenarios assess safe Storybook work rather than a file-only patch.
The review checks use story discovery as the source of story IDs. They require the final review to include the stories created or affected by the task. Shared-token changes therefore review consumer stories, while a monorepo case runs the checks in the Storybook package that owns the stories. The plugin path also checks that the development server remains available after verification.
References: agent-eval/…/EVAL.ts• L19-73 agent-eval/…/EVAL.ts• L10-36 agent-eval/…/EVAL.ts• L14-69 agent-eval/…/EVAL.ts• L17-48 agent-eval/…/EVAL.ts• L19-64 agent-eval/…/EVAL.ts• L4-18 agent-eval/…/test-utils.ts• L200-289 agent-eval/…/test-utils.ts• L334-431
Baselines and comparisons
The research line needs to separate the agent's change from the state of the external application before the agent ran. It records the external repository and design-system pin used by each run, then compares the run with a pristine copy of that pinned tree.
Baseline measurements
A baseline is a saved measurement of one pinned external repository before an agent changes it. The platform reuses one baseline for all compatible runs that use the same pin and metric definition. This keeps expensive whole-tree measurements out of every individual run.
Baseline records are invalidated when the metric version changes. Metrics that need a whole design-system node census store that census beside the baseline so the baseline and its supporting data remain tied to the same rules.
References: agent-eval/…/analyze-results.ts• L23-26 agent-eval/…/baseline.ts• L1-10 agent-eval/…/baseline.ts• L140-177 agent-eval/…/baseline.ts• L200-232
Comparable samples
The analyzer groups runs only when they measure the same case, workflow, and configuration. A fixture or external pin change creates a new measurement group. Older runs remain available as superseded data instead of being mixed with the current sample.
This rule lets a plan collect one sample over several batches and still produce one comparison. It also prevents a changed prompt, Storybook content pin, or metric definition from silently changing a historical result.
References: agent-eval/…/comparability.ts• L1-5 agent-eval/…/comparability.ts• L49-84 agent-eval/…/comparability.ts• L113-155 agent-eval/…/analyze-results.ts• L424-476
Collection plans
Plans turn a research question into a repeatable set of cases, workflows, and repetitions. They answer one comparison question at a time and make collection resumable when a run is interrupted or a resource limit stops a batch.
Plan groups
The plans progress from broad questions to narrower ones. Early groups measure how much content helps and whether docs or stories matter. Later groups isolate JSDoc, documentation types, story types, accessibility, and brand or animation content.
| Group | Main comparison |
|---|---|
levels | Control, empty, stories API reference, and full content |
docs-vs-stories | Documentation-focused versus story-focused content |
jsdoc | Full content versus removed JSDoc |
doc-types | Different documentation facets |
story-types | Different story forms |
doc-types-lower-roi | Accessibility and brand-animation facets |
Each plan targets a fixed number of runs per case and workflow. A dry run shows which cells already have enough comparable data and which cells need more collection before any paid work starts.
References: agent-eval/…/README.md• L1-19 agent-eval/…/README.md• L21-45 agent-eval/…/run-plan.ts• L13-21 agent-eval/…/run-plan.ts• L206-247
Collection safeguards
Collection batches cap concurrent sandboxes and keep a report of gaps, resource signals, and stopped batches. The plan runner does not treat an interrupted or infrastructure-only run as useful sample data unless the operator explicitly acknowledges that choice.
The agentic-reference dispatcher also supports a dry-run spend guard. An expected evaluation count can stop a dispatch before any agent call when the selected cases resolve to a different size.
References: agent-eval/…/README.md• L48-61 agent-eval/…/run-plan.ts• L251-347 agent-eval/…/run-agentic-ref.ts• L94-148
Fixture templates and sandboxes
Templates provide shared starter projects so scenario fixtures can focus on the task and its expected behavior. The sandbox setup merges a template with fixture files, adds the selected agent support, and prepares the Storybook packages that the evaluation is meant to measure.
Template roles
The platform uses three broad project shapes:
| Template | Purpose |
|---|---|
reshaped-storybook | A design-system project with Storybook, MSW, and story test support |
vite-app | A small React and Vite app for initialization and upgrade tasks |
monorepo | A workspace project where the runnable Storybook lives in a package leaf |
The fixture keeps its own prompt, evaluation rules, and metadata. Shared app files stay in the template so changes to a common starting state apply to all scenarios that use it.
References: agent-eval/README.md• L344-382 agent-eval/…/templates.ts• L118-193 agent-eval/…/templates.ts• L300-376
Checkout fidelity
By default, a sandbox uses the Storybook packages built from the checkout under test. This is important because the result must describe the branch being evaluated rather than a similarly versioned package from the registry. A separate latest mode installs published packages to test the stable-release behavior.
Lifecycle scenarios that install Storybook themselves use a local registry so the initializer or upgrade command can still install the checkout build. The harness records the checkout and working-tree state with each result. Shared package preparation is cached across concurrent runs so each sandbox gets the same package set without unsafe overlapping package builds.
References: agent-eval/README.md• L86-136 agent-eval/README.md• L153-159 agent-eval/…/templates.ts• L455-505 agent-eval/…/templates.ts• L617-705 agent-eval/…/templates.ts• L708-767
Agent support files
Sandbox setup gives each run the support files that match its agent and integration. MCP cases receive the Storybook addon and server instructions. Plugin cases receive the agent's Storybook skills. The evaluation also records the selected agent, model, and integration so outcome checks can apply only where they are meaningful.
The shared evaluation helpers read transcripts and saved run data through one interface. They normalize MCP calls and approved storybook tools CLI calls into the same workflow events. Shell parsing covers the command forms agents use, including chained commands, shell wrappers, substitutions, and heredoc payloads. It does not credit raw network requests or quoted text as workflow calls. This lets a case require a workflow step without accepting an ad hoc request that only resembles the supported tool.
References: agent-eval/…/templates.ts• L118-227 agent-eval/…/templates.ts• L905-955 agent-eval/…/templates.ts• L1051-1129 agent-eval/…/test-utils.ts• L28-159 agent-eval/…/shell-parse.ts• L43-107 agent-eval/…/shell-parse.ts• L128-183 agent-eval/…/shell-parse.ts• L258-335 agent-eval/…/shell-parse.ts• L475-515 agent-eval/…/shell-parse.test.ts• L132-182 agent-eval/…/shell-parse.test.ts• L367-453
Result analysis and delivery
The platform keeps raw run artifacts separate from derived analysis. This allows metric definitions to change without spending on new agent runs and supports both local investigation and CI review.
Offline analysis
Offline analysis reads the saved transcript, result metadata, project tree, and fixture definition. It computes per-run metrics, applies baseline deltas when needed, caches the result, and writes grouped summaries for comparable samples.
For agentic-reference data, the comparison command builds a dataset from usable cells, checks that every selected case has enough runs, and produces statistical estimates, false-discovery-rate verdicts, curves, and reports. The output identifies the control, treatment cases, workflows, sample threshold, and provenance.
The command surface separates collection, analysis, comparison, result export, and the local playground. It also provides dry-run commands for paid collection and focused judging. Transcript usage is summarized for Claude Code and Codex, with an estimated cost when the model has a known price. The workspace recognizes agent-eval as an independent project with its own check target, while the TypeScript project covers evaluation definitions, plans, libraries, and scripts and excludes saved results and template trees from type checking.
References: agent-eval/…/analyze-results.ts• L1-26 agent-eval/…/types.ts• L7-16 agent-eval/…/types.ts• L112-151 agent-eval/…/compare-results.ts• L68-167 agent-eval/…/compare-results.ts• L187-244 agent-eval/…/usage.ts• L1-130 agent-eval/package.json• L10-34 agent-eval/project.json• L1-7 agent-eval/…/test-utils.ts• L100-159 agent-eval/…/test-utils.ts• L551-686
Usage accounting
Usage accounting turns saved agent transcripts into token totals and an estimated dollar cost. It handles the different usage events emitted by Claude Code and Codex, avoids counting repeated streamed messages twice, and leaves the cost unknown when a model has no pricing entry.
References: agent-eval/…/usage.ts• L45-130
CI delivery
The GitHub Actions workflow runs the default smoke evaluation on pull requests marked by a maintainer and runs the broader line on the weekly schedule or a manual dispatch. It archives results, deploys the results playground to Vercel, and updates the pull request gate when the evaluated head is still current.
The agentic-reference workflow is dispatch-only because its runs are research work with a larger spend. It accepts explicit case, workflow, repetition, and dry-run controls, then archives the captured results for later analysis.
References: .github/…/agent-eval.yml• L3-63 .github/…/agent-eval.yml• L174-230 .github/…/agent-eval.yml• L269-347 .github/…/agentic-ref-eval.yml• L1-45 .github/…/agentic-ref-eval.yml• L103-158
Change checklist
When these files or sections are changed, remember to consider these:
- When adding or renaming a workflow fixture, update the active registry and any collection plans that select it. A fixture on disk is not enough for the paid runner to select it.
- When changing a case's prompt, MCP source, external pin, or agent settings, treat existing samples as a new measurement and collect enough runs for every comparison cell.
- When changing post-analysis metrics, bump the metric version and rebuild the matching committed baselines and node censuses.
- When changing model names or transcript parsing, update the pricing table and confirm that usage totals and cost estimates remain comparable across Claude Code and Codex.
- When changing a shared template or checkout package mapping, check every fixture that uses that template and ensure the sandbox still measures the intended checkout rather than a registry release.
- When changing workflow outcome rules, review the matching fixture prompts and support helpers. A new required step can change scores for both MCP and plugin integrations.
- When changing story-review rules, keep story discovery, changed-file mode, consumer fallback, review coverage, and final-response link checks aligned.
- When changing the local Storybook package set or checkout registry, keep package compilation, tarball resolution, and install-source checks aligned.
- When changing CI labels, result paths, or playground routes, keep the eval gate, artifact upload, Vercel deployment, and pull request summary contracts aligned.
- When changing the Linux
node_replservice, keep its MCP tool names, input shapes, and persistent-kernel behavior aligned with the Codex browser workflow. - When changing shell parsing or supported CLI forms, keep the parsed workflow events aligned with the real Storybook CLI and with the scenario checks that use them.
- When changing the evaluation command set or its TypeScript project boundary, keep local scripts, CI entry points, source aliases, and excluded generated data aligned.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| A case or workflow name is not in the active registry | Selection fails or a paid run does not include the intended scenario | Resolve selections against the registry and run the plan in dry mode first |
| A fixture or pin changes after data collection | Old runs measure a different question and must not join the current sample | Comparability groups them as superseded data |
| A metric definition changes without a new version | A stale baseline can produce a false delta | Baseline reuse requires a matching metric version |
| A model is missing from the pricing table | Token totals remain valid but the reported cost is absent | Add the model's price before using cost data for spend checks |
| A run stops before saving a project tree | The transcript exists but there is no usable measurement | Analysis skips the run and plan collection reports the sample gap |
| A visual-change case publishes guessed or incomplete story IDs | The review can look valid while omitting the changed surface | Require discovery before review and check the target story IDs |
| A browse request sends changed files instead of an empty list | The review is treated as a change review rather than a view of current stories | Browse-specific checks require changedFiles: [] |
| A fetching-story case changes the component or uses an unsupported mock | The scenario no longer measures story-level data handling | Keep the component fetch contract and require the supported MSW path |
| An agent uses a shell form that the workflow parser cannot recognize | The action occurs but the evaluation records a missing workflow step | Keep parser support aligned with the CLI and test wrappers, substitutions, and heredoc inputs |
| Checkout packages resolve from the public registry | The evaluation measures published code instead of the branch under test | Sandbox setup points dependencies to checkout packages and checks the install source |
A Codex browser workflow cannot reach the node_repl MCP service in a Linux sandbox | Browser-assisted steps fail before they can assess the Storybook workflow | Provide the evaluation's compatible node_repl service and keep its MCP tool surface aligned with the Codex workflow |
| Required agent, sandbox, or Vercel credentials are missing | CI skips work, fails before collection, or cannot publish the playground | CI validates secrets before the paid run and checks that result files exist |
| A source file falls outside the TypeScript project boundary or a local alias points to the wrong source | Type checks pass over an incomplete or different evaluation surface | Keep include, exclude, and local path mappings aligned with the scripts and checkout |
Key files
| File | Why to read it |
|---|---|
| agent-eval/README.md | End-to-end operating model, scenario families, commands, CI behavior, and result use |
| agent-eval/package.json | Command entry points for evaluation, plan collection, result analysis, the playground, focused judges, tests, and type checks |
| agent-eval/…/usage.ts | Transcript usage totals and model-based cost estimates |
| agent-eval/tsconfig.json | Type-checking boundary and local source mappings for the evaluation application |
| agent-eval/…/cases.ts | Active workflows and the control and treatment case model |
| agent-eval/…/experiment.ts | Relationship between a case, sandbox setup, agent support, and stored run metadata |
| agent-eval/…/node-repl-mock.mjs | Linux-compatible MCP support for Codex browser workflows that need a Node-backed runtime |
| agent-eval/…/templates.ts | Shared project shapes, agent support, checkout package selection, and the boundary between fixture content and sandbox preparation |
| agent-eval/…/test-utils.ts | Shared scenario checks for transcripts, workflow calls, story reviews, tests, and integration-specific outcomes |
| agent-eval/…/shell-parse.ts | Common view of supported Storybook CLI workflow calls across shell forms and agent integrations |
| agent-eval/…/shell-parse.test.ts | Contract checks that keep shell parsing aligned with the Storybook CLI argument rules and agent command forms |
| agent-eval/…/EVAL.ts | Example of a workflow evaluation that checks agent actions, story tests, documentation use, and final Storybook review results |
| agent-eval/…/EVAL.ts | Example of a focused workflow evaluation that requires story creation, targeted tests, and a preview |
| agent-eval/…/EVAL.ts | Example of a lifecycle evaluation for Storybook initialization |
| agent-eval/…/README.md | Research question order, sample targets, and collection rules |
| agent-eval/…/run-plan.ts | Plan resolution, batching, reuse, and gap reporting |
| agent-eval/…/analyze-results.ts | Offline metric and baseline analysis flow |
| agent-eval/…/compare-results.ts | Statistical comparison and report production |
| agent-eval/…/baseline.ts | Baseline reuse and invalidation rules |
| .github/…/agent-eval.yml | Pull request and scheduled evaluation delivery |
| .github/…/agentic-ref-eval.yml | Manual collection of the research case matrix |
| agent-eval/project.json | Workspace registration for the separate evaluation project and its check target |