Introducing a new way to review your PRs.

Read the announcement

storybookjs / storybook

Storybook is the industry standard workshop for building, documenting, and testing UI components in isolation

View on GitHub

20 documents · Updated Nov 20, 2026

docs / agent-evaluation-platform

Agent evaluation platform

The agent-eval application is a separate research and quality system for measuring how coding agents use Storybook. It runs repeatable tasks in isolated project sandboxes, checks the agent's work or records it for later study, and publishes results for local and CI review. It sits under developer workflows and complements project setup and migration, testing fixtures and sandboxes, repository automation, and ecosystem quality assurance.

Evaluation system

The platform separates an agent task from the project used to run it and from the way its result is judged. This lets one task run against several agents, Storybook information sources, content variants, or package versions without changing the task itself.

Evaluation lifecycle

An evaluation starts with a fixture that gives the agent a task and a starting project. The harness prepares a sandbox, runs the agent, saves the project and transcript, then either checks explicit workflow rules or sends the saved artifacts to offline analysis.

The main lifecycle is:

flowchart LR
    F[Task fixture] --> S[Sandbox setup]
    S --> A[Agent workflow]
    A --> R[Saved transcript and project]
    R --> G[Pass or fail checks]
    R --> M[Offline metrics]
    M --> C[Baseline or case comparison]
    G --> P[Results playground and CI gate]
    C --> P

References: agent-eval/README.md• L5-9 agent-eval/README.md• L138-151 agent-eval/…/experiment.ts• L193-265

Agent integrations

The platform compares how agents work with different Storybook support channels. A case can provide a Storybook MCP server, plugin skills, other MCP services, prompt guidance, or no support as a control. Linux sandboxes can also provide a compatible node_repl MCP service for Codex browser workflows when the native Codex service is not available. The recorded case keeps these choices with each result so later comparisons do not depend on current configuration.

The default suite covers Claude Code and Codex configurations. A case can pin the model and effort so a change in an agent's default does not change the experiment without notice. The evaluation project resolves the local Storybook MCP source through its type configuration, which keeps agent-facing support aligned with the checkout under test.

References: agent-eval/…/templates.ts• L29-33 agent-eval/…/cases.ts• L36-68 agent-eval/…/experiment.ts• L76-109 agent-eval/…/node-repl-mock.mjs• L1-14 agent-eval/…/node-repl-mock.mjs• L32-90 agent-eval/tsconfig.json• L15-25

Evaluation scenarios

The scenario library tests the agent behaviors that matter to Storybook users. Each scenario combines a task prompt with checks over the final project, shell activity, transcript, Storybook workflow calls, or test results. The workflow line also checks that agents use the supported Storybook workflow and produce a complete display review when the task changes or browses visual UI.

Workflow evaluations

The active 8xx line is the main product workflow suite. It checks work such as creating or editing components, writing stories for new, existing, fetching, and monorepo components, fixing tests and accessibility issues, using documentation and browsing tools, and running story tests. Visual-change cases also check story discovery before review publication, review coverage for the changed or newly created stories, and a review link in the final response. A browse request must publish the current stories with an empty change list. The 82x scenarios extend this line to Storybook project setup, upgrades, and outdated projects.

The 9xx line is a smaller MCP-focused set for the published latest Storybook release. It covers gaps such as asynchronous mocks, story drift, tool parameters, preview by path or ID, and Vitest command use. Its component-creation test case requires story creation, a focused story test, and a focused preview. This checks the complete agent path rather than only the resulting files. The line is selected separately because it measures the stable release rather than the checkout under test. The default run uses one core 8xx scenario; broader 8xx, 82x, or 9xx lines require explicit selection.

References: agent-eval/README.md• L32-83 agent-eval/README.md• L181-205 agent-eval/…/EVAL.ts• L17-64 agent-eval/…/EVAL.ts• L11-40 agent-eval/…/EVAL.ts• L4-18

Agentic-reference evaluations

The 70x line is a research suite for measuring the value of Storybook content in AI-assisted UI work. It does not reduce each run to a pass or fail. Instead, it compares a control case with cases that serve different documentation and story content through a design-system Storybook MCP.

The current workflow families are:

WorkflowQuestion it represents
701-new-ui-flowHow the agent creates new UI
702-rework-ui-flowHow the agent reworks existing UI
703-fix-bug-flowHow the agent fixes a UI bug
704-fix-a11y-flowHow the agent addresses an accessibility issue
706-new-ui-scheduled-flowHow a scheduled new-UI task behaves

Cases vary the amount and type of documentation or stories. The control case has no agent support, while treatment cases can provide empty, docs-focused, stories-focused, or complete Storybook content. The research line captures the output for later metric and judge analysis instead of using the hand-crafted workflow gate.

References: agent-eval/…/cases.ts• L18-28 agent-eval/…/cases.ts• L94-157 agent-eval/…/cases.ts• L183-203 agent-eval/README.md• L389-423

Workflow outcome checks

The workflow cases judge both the result and the way the agent reaches it. Story tests must finish with the target stories passing and must use focused story input when the scenario requires it. Accessibility cases must observe the issue, keep the coverage that exposes it, and report unresolved visual contrast concerns instead of hiding them. Story-writing cases must model network data in stories without changing the component's fetch contract. These checks make the scenarios assess safe Storybook work rather than a file-only patch.

The review checks use story discovery as the source of story IDs. They require the final review to include the stories created or affected by the task. Shared-token changes therefore review consumer stories, while a monorepo case runs the checks in the Storybook package that owns the stories. The plugin path also checks that the development server remains available after verification.

References: agent-eval/…/EVAL.ts• L19-73 agent-eval/…/EVAL.ts• L10-36 agent-eval/…/EVAL.ts• L14-69 agent-eval/…/EVAL.ts• L17-48 agent-eval/…/EVAL.ts• L19-64 agent-eval/…/EVAL.ts• L4-18 agent-eval/…/test-utils.ts• L200-289 agent-eval/…/test-utils.ts• L334-431

Baselines and comparisons

The research line needs to separate the agent's change from the state of the external application before the agent ran. It records the external repository and design-system pin used by each run, then compares the run with a pristine copy of that pinned tree.

Baseline measurements

A baseline is a saved measurement of one pinned external repository before an agent changes it. The platform reuses one baseline for all compatible runs that use the same pin and metric definition. This keeps expensive whole-tree measurements out of every individual run.

Baseline records are invalidated when the metric version changes. Metrics that need a whole design-system node census store that census beside the baseline so the baseline and its supporting data remain tied to the same rules.

References: agent-eval/…/analyze-results.ts• L23-26 agent-eval/…/baseline.ts• L1-10 agent-eval/…/baseline.ts• L140-177 agent-eval/…/baseline.ts• L200-232

Comparable samples

The analyzer groups runs only when they measure the same case, workflow, and configuration. A fixture or external pin change creates a new measurement group. Older runs remain available as superseded data instead of being mixed with the current sample.

This rule lets a plan collect one sample over several batches and still produce one comparison. It also prevents a changed prompt, Storybook content pin, or metric definition from silently changing a historical result.

References: agent-eval/…/comparability.ts• L1-5 agent-eval/…/comparability.ts• L49-84 agent-eval/…/comparability.ts• L113-155 agent-eval/…/analyze-results.ts• L424-476

Collection plans

Plans turn a research question into a repeatable set of cases, workflows, and repetitions. They answer one comparison question at a time and make collection resumable when a run is interrupted or a resource limit stops a batch.

Plan groups

The plans progress from broad questions to narrower ones. Early groups measure how much content helps and whether docs or stories matter. Later groups isolate JSDoc, documentation types, story types, accessibility, and brand or animation content.

GroupMain comparison
levelsControl, empty, stories API reference, and full content
docs-vs-storiesDocumentation-focused versus story-focused content
jsdocFull content versus removed JSDoc
doc-typesDifferent documentation facets
story-typesDifferent story forms
doc-types-lower-roiAccessibility and brand-animation facets

Each plan targets a fixed number of runs per case and workflow. A dry run shows which cells already have enough comparable data and which cells need more collection before any paid work starts.

References: agent-eval/…/README.md• L1-19 agent-eval/…/README.md• L21-45 agent-eval/…/run-plan.ts• L13-21 agent-eval/…/run-plan.ts• L206-247

Collection safeguards

Collection batches cap concurrent sandboxes and keep a report of gaps, resource signals, and stopped batches. The plan runner does not treat an interrupted or infrastructure-only run as useful sample data unless the operator explicitly acknowledges that choice.

The agentic-reference dispatcher also supports a dry-run spend guard. An expected evaluation count can stop a dispatch before any agent call when the selected cases resolve to a different size.

References: agent-eval/…/README.md• L48-61 agent-eval/…/run-plan.ts• L251-347 agent-eval/…/run-agentic-ref.ts• L94-148

Fixture templates and sandboxes

Templates provide shared starter projects so scenario fixtures can focus on the task and its expected behavior. The sandbox setup merges a template with fixture files, adds the selected agent support, and prepares the Storybook packages that the evaluation is meant to measure.

Template roles

The platform uses three broad project shapes:

TemplatePurpose
reshaped-storybookA design-system project with Storybook, MSW, and story test support
vite-appA small React and Vite app for initialization and upgrade tasks
monorepoA workspace project where the runnable Storybook lives in a package leaf

The fixture keeps its own prompt, evaluation rules, and metadata. Shared app files stay in the template so changes to a common starting state apply to all scenarios that use it.

References: agent-eval/README.md• L344-382 agent-eval/…/templates.ts• L118-193 agent-eval/…/templates.ts• L300-376

Checkout fidelity

By default, a sandbox uses the Storybook packages built from the checkout under test. This is important because the result must describe the branch being evaluated rather than a similarly versioned package from the registry. A separate latest mode installs published packages to test the stable-release behavior.

Lifecycle scenarios that install Storybook themselves use a local registry so the initializer or upgrade command can still install the checkout build. The harness records the checkout and working-tree state with each result. Shared package preparation is cached across concurrent runs so each sandbox gets the same package set without unsafe overlapping package builds.

References: agent-eval/README.md• L86-136 agent-eval/README.md• L153-159 agent-eval/…/templates.ts• L455-505 agent-eval/…/templates.ts• L617-705 agent-eval/…/templates.ts• L708-767

Agent support files

Sandbox setup gives each run the support files that match its agent and integration. MCP cases receive the Storybook addon and server instructions. Plugin cases receive the agent's Storybook skills. The evaluation also records the selected agent, model, and integration so outcome checks can apply only where they are meaningful.

The shared evaluation helpers read transcripts and saved run data through one interface. They normalize MCP calls and approved storybook tools CLI calls into the same workflow events. Shell parsing covers the command forms agents use, including chained commands, shell wrappers, substitutions, and heredoc payloads. It does not credit raw network requests or quoted text as workflow calls. This lets a case require a workflow step without accepting an ad hoc request that only resembles the supported tool.

References: agent-eval/…/templates.ts• L118-227 agent-eval/…/templates.ts• L905-955 agent-eval/…/templates.ts• L1051-1129 agent-eval/…/test-utils.ts• L28-159 agent-eval/…/shell-parse.ts• L43-107 agent-eval/…/shell-parse.ts• L128-183 agent-eval/…/shell-parse.ts• L258-335 agent-eval/…/shell-parse.ts• L475-515 agent-eval/…/shell-parse.test.ts• L132-182 agent-eval/…/shell-parse.test.ts• L367-453

Result analysis and delivery

The platform keeps raw run artifacts separate from derived analysis. This allows metric definitions to change without spending on new agent runs and supports both local investigation and CI review.

Offline analysis

Offline analysis reads the saved transcript, result metadata, project tree, and fixture definition. It computes per-run metrics, applies baseline deltas when needed, caches the result, and writes grouped summaries for comparable samples.

For agentic-reference data, the comparison command builds a dataset from usable cells, checks that every selected case has enough runs, and produces statistical estimates, false-discovery-rate verdicts, curves, and reports. The output identifies the control, treatment cases, workflows, sample threshold, and provenance.

The command surface separates collection, analysis, comparison, result export, and the local playground. It also provides dry-run commands for paid collection and focused judging. Transcript usage is summarized for Claude Code and Codex, with an estimated cost when the model has a known price. The workspace recognizes agent-eval as an independent project with its own check target, while the TypeScript project covers evaluation definitions, plans, libraries, and scripts and excludes saved results and template trees from type checking.

References: agent-eval/…/analyze-results.ts• L1-26 agent-eval/…/types.ts• L7-16 agent-eval/…/types.ts• L112-151 agent-eval/…/compare-results.ts• L68-167 agent-eval/…/compare-results.ts• L187-244 agent-eval/…/usage.ts• L1-130 agent-eval/package.json• L10-34 agent-eval/project.json• L1-7 agent-eval/…/test-utils.ts• L100-159 agent-eval/…/test-utils.ts• L551-686

Usage accounting

Usage accounting turns saved agent transcripts into token totals and an estimated dollar cost. It handles the different usage events emitted by Claude Code and Codex, avoids counting repeated streamed messages twice, and leaves the cost unknown when a model has no pricing entry.

References: agent-eval/…/usage.ts• L45-130

CI delivery

The GitHub Actions workflow runs the default smoke evaluation on pull requests marked by a maintainer and runs the broader line on the weekly schedule or a manual dispatch. It archives results, deploys the results playground to Vercel, and updates the pull request gate when the evaluated head is still current.

The agentic-reference workflow is dispatch-only because its runs are research work with a larger spend. It accepts explicit case, workflow, repetition, and dry-run controls, then archives the captured results for later analysis.

References: .github/…/agent-eval.yml• L3-63 .github/…/agent-eval.yml• L174-230 .github/…/agent-eval.yml• L269-347 .github/…/agentic-ref-eval.yml• L1-45 .github/…/agentic-ref-eval.yml• L103-158

Change checklist

When these files or sections are changed, remember to consider these:

  • When adding or renaming a workflow fixture, update the active registry and any collection plans that select it. A fixture on disk is not enough for the paid runner to select it.
  • When changing a case's prompt, MCP source, external pin, or agent settings, treat existing samples as a new measurement and collect enough runs for every comparison cell.
  • When changing post-analysis metrics, bump the metric version and rebuild the matching committed baselines and node censuses.
  • When changing model names or transcript parsing, update the pricing table and confirm that usage totals and cost estimates remain comparable across Claude Code and Codex.
  • When changing a shared template or checkout package mapping, check every fixture that uses that template and ensure the sandbox still measures the intended checkout rather than a registry release.
  • When changing workflow outcome rules, review the matching fixture prompts and support helpers. A new required step can change scores for both MCP and plugin integrations.
  • When changing story-review rules, keep story discovery, changed-file mode, consumer fallback, review coverage, and final-response link checks aligned.
  • When changing the local Storybook package set or checkout registry, keep package compilation, tarball resolution, and install-source checks aligned.
  • When changing CI labels, result paths, or playground routes, keep the eval gate, artifact upload, Vercel deployment, and pull request summary contracts aligned.
  • When changing the Linux node_repl service, keep its MCP tool names, input shapes, and persistent-kernel behavior aligned with the Codex browser workflow.
  • When changing shell parsing or supported CLI forms, keep the parsed workflow events aligned with the real Storybook CLI and with the scenario checks that use them.
  • When changing the evaluation command set or its TypeScript project boundary, keep local scripts, CI entry points, source aliases, and excluded generated data aligned.

Important failure modes

TriggerConsequenceGuard
A case or workflow name is not in the active registrySelection fails or a paid run does not include the intended scenarioResolve selections against the registry and run the plan in dry mode first
A fixture or pin changes after data collectionOld runs measure a different question and must not join the current sampleComparability groups them as superseded data
A metric definition changes without a new versionA stale baseline can produce a false deltaBaseline reuse requires a matching metric version
A model is missing from the pricing tableToken totals remain valid but the reported cost is absentAdd the model's price before using cost data for spend checks
A run stops before saving a project treeThe transcript exists but there is no usable measurementAnalysis skips the run and plan collection reports the sample gap
A visual-change case publishes guessed or incomplete story IDsThe review can look valid while omitting the changed surfaceRequire discovery before review and check the target story IDs
A browse request sends changed files instead of an empty listThe review is treated as a change review rather than a view of current storiesBrowse-specific checks require changedFiles: []
A fetching-story case changes the component or uses an unsupported mockThe scenario no longer measures story-level data handlingKeep the component fetch contract and require the supported MSW path
An agent uses a shell form that the workflow parser cannot recognizeThe action occurs but the evaluation records a missing workflow stepKeep parser support aligned with the CLI and test wrappers, substitutions, and heredoc inputs
Checkout packages resolve from the public registryThe evaluation measures published code instead of the branch under testSandbox setup points dependencies to checkout packages and checks the install source
A Codex browser workflow cannot reach the node_repl MCP service in a Linux sandboxBrowser-assisted steps fail before they can assess the Storybook workflowProvide the evaluation's compatible node_repl service and keep its MCP tool surface aligned with the Codex workflow
Required agent, sandbox, or Vercel credentials are missingCI skips work, fails before collection, or cannot publish the playgroundCI validates secrets before the paid run and checks that result files exist
A source file falls outside the TypeScript project boundary or a local alias points to the wrong sourceType checks pass over an incomplete or different evaluation surfaceKeep include, exclude, and local path mappings aligned with the scripts and checkout

Key files

FileWhy to read it
agent-eval/README.mdEnd-to-end operating model, scenario families, commands, CI behavior, and result use
agent-eval/package.jsonCommand entry points for evaluation, plan collection, result analysis, the playground, focused judges, tests, and type checks
agent-eval/…/usage.tsTranscript usage totals and model-based cost estimates
agent-eval/tsconfig.jsonType-checking boundary and local source mappings for the evaluation application
agent-eval/…/cases.tsActive workflows and the control and treatment case model
agent-eval/…/experiment.tsRelationship between a case, sandbox setup, agent support, and stored run metadata
agent-eval/…/node-repl-mock.mjsLinux-compatible MCP support for Codex browser workflows that need a Node-backed runtime
agent-eval/…/templates.tsShared project shapes, agent support, checkout package selection, and the boundary between fixture content and sandbox preparation
agent-eval/…/test-utils.tsShared scenario checks for transcripts, workflow calls, story reviews, tests, and integration-specific outcomes
agent-eval/…/shell-parse.tsCommon view of supported Storybook CLI workflow calls across shell forms and agent integrations
agent-eval/…/shell-parse.test.tsContract checks that keep shell parsing aligned with the Storybook CLI argument rules and agent command forms
agent-eval/…/EVAL.tsExample of a workflow evaluation that checks agent actions, story tests, documentation use, and final Storybook review results
agent-eval/…/EVAL.tsExample of a focused workflow evaluation that requires story creation, targeted tests, and a preview
agent-eval/…/EVAL.tsExample of a lifecycle evaluation for Storybook initialization
agent-eval/…/README.mdResearch question order, sample targets, and collection rules
agent-eval/…/run-plan.tsPlan resolution, batching, reuse, and gap reporting
agent-eval/…/analyze-results.tsOffline metric and baseline analysis flow
agent-eval/…/compare-results.tsStatistical comparison and report production
agent-eval/…/baseline.tsBaseline reuse and invalidation rules
.github/…/agent-eval.ymlPull request and scheduled evaluation delivery
.github/…/agentic-ref-eval.ymlManual collection of the research case matrix
agent-eval/project.jsonWorkspace registration for the separate evaluation project and its check target