Introducing a new way to review your PRs.

Read the announcement

storybookjs / storybook

Storybook is the industry standard workshop for building, documenting, and testing UI components in isolation

View on GitHub

20 documents · Updated Nov 20, 2026

docs / ecosystem-quality-assurance

Ecosystem quality assurance

Ecosystem QA checks how a Storybook version behaves in real external projects. It upgrades a catalogued project in a controlled run, records check results and evidence, removes secrets from public output, then turns repeat findings into safe issue or comment proposals.

System purpose

The system acts as a repeatable compatibility sweep for projects outside the main repository. It does not decide that a behavior is a bug by itself; an agent applies the shared checklist, while the harness records enough context for people to review the result.

The application complements developer workflows by testing the upgrade path in real projects. It is distinct from agent evaluation, which measures agent behavior on setup tasks rather than compatibility across an ecosystem.

Catalog scope

The catalog defines the projects that a sweep may inspect and the facts needed to run each project. Each entry names a repository revision, project class, package manager, Storybook directory, and Windows support; catalog ids are unique. The initial catalog contains the owned storybookjs/mealdrop project and uses Yarn.

References: ecosystem-qa/catalog.json• L1-11 ecosystem-qa/…/schema.ts• L5-34

Check model

Checks cover the upgrade, development server, build, and test stages. A must tier runs in every sweep, while extended adds broader compatibility checks. The same check set becomes the agent checklist and the required shape of its answer, so each selected check is reported once and findings attach only to failed checks.

The checks move through the project lifecycle. Earlier stages leave the installed, running, or built project ready for the next stage. Results can be pass, fail, n/a, blocked, or skip, which lets the report separate a real failure from an inapplicable or unavailable check. The reference checklist also includes manual upgrade checks outside the agent registry, such as the clear migration path required when @storybook/test-runner is installed for a Storybook 11 upgrade.

References: ecosystem-qa/checks.ts• L1-16 ecosystem-qa/checks.ts• L18-30 ecosystem-qa/checks.ts• L232-246 ecosystem-qa/…/basic-qa.md• L92-107 ecosystem-qa/…/schema.ts• L48-105

Quality assurance run

One run compares a target Storybook version with a catalogued project on one operating system and one agent engine. The run creates an isolated project copy, installs its dependencies, asks the agent to perform the selected checks, and writes a stable result record even when the agent fails or returns invalid output.

The main run path follows this sequence.

flowchart LR
    C[Catalog entry] --> K[Clone project revision]
    K --> S[Install and upgrade in sandbox]
    S --> A[Agent runs checklist]
    A --> E[Evidence and session output]
    A --> R[Validated result.json]
    E --> X[Redacted public artifact]
    R --> X

Run isolation

The run uses a sandbox to limit the agent while still allowing the project to install, build, serve, and capture evidence. Docker is the broad CI option; native Linux and macOS sandboxes support local use. An unsandboxed run is restricted by project class and the reviewed Windows exception, while fake runs execute no project or agent work.

The run also keeps the agent session, logs, evidence, and an optional cycle bundle. These artifacts support review and later inspection without making the run itself the place where a compatibility decision is made.

References: ecosystem-qa/run.ts• L48-59 ecosystem-qa/run.ts• L152-184 ecosystem-qa/run.ts• L213-288 ecosystem-qa/run.ts• L312-410 ecosystem-qa/…/sandbox-policy.ts• L1-120

Run result

result.json is the contract between a run and the later collection stages. It records the request, installed version, source and result revisions, checks, findings, model usage, timing, machine state, artifact paths, and overall status. A result can therefore be compared across projects, operating systems, engines, and Storybook versions without reopening the live agent session.

The result separates an agent or output failure from a check failure. A completed run may report findings while still having status ok; a timeout, agent error, or invalid structured answer is recorded as a run status instead.

References: ecosystem-qa/run.ts• L422-464 ecosystem-qa/…/schema.ts• L121-157

Evidence and privacy

Evidence gives a finding a reviewable link to what the agent saw. A finding can include logs, a screenshot, an animation, reproduction steps, the expected and actual behavior, and a severity hint. File names are constrained to the evidence area so the report can reference artifacts without allowing paths to escape it.

The complete output directory becomes a public artifact, so redaction is a final boundary rather than an optional display step. The system removes credential-shaped data from the run output before it prints or publishes the artifact, and the filing stage copies only redacted evidence to the shared asset branch.

References: ecosystem-qa/…/schema.ts• L43-70 ecosystem-qa/run.ts• L422-464 ecosystem-qa/run.ts• L485-495 ecosystem-qa/file.ts• L118-149 ecosystem-qa/…/redact.ts• L1-200

Finding collection

Collection combines results from the selected projects, operating systems, and engines into proposals for one Storybook repository. It accepts local results or downloaded CI artifacts, ignores results for another target version, groups equivalent findings by fingerprint, and checks GitHub for prior reports.

The collection flow turns separate observations into one of three proposal kinds:

ProposalMeaning
IssueA finding has no open matching report and needs a new actionable issue.
CommentA matching open issue exists and needs new run evidence or context.
SuppressedA matching known issue already marks the finding as understood.

This grouping keeps repeated observations from creating a new report for every project or engine. It also preserves the source runs and the cycle information needed to reproduce the result.

References: ecosystem-qa/collect.ts• L10-17 ecosystem-qa/collect.ts• L47-67 ecosystem-qa/…/schema.ts• L159-220 ecosystem-qa/…/proposals.ts• L1-220

Compatibility proposal filing

Filing converts proposals into controlled repository actions. Dry mode writes the proposed issue body and evidence plan to the job summary without changing GitHub; live mode checks that the destination has the required milestone and labels before it files anything.

For a new issue, the workflow can publish the upgraded project state and evidence separately from the issue. The upgraded state goes to a protected QA ref rather than a normal branch, which prevents repository workflows from running agent changes before review. Evidence goes to the qa-assets branch, and the issue links the material needed to inspect it.

Filing is designed to be safe to repeat. Existing fingerprints, identical comments, and already-published evidence are skipped. One proposal may fail without blocking the others, but the final job result reports the failures.

References: ecosystem-qa/file.ts• L25-32 ecosystem-qa/file.ts• L93-149 ecosystem-qa/file.ts• L199-241 ecosystem-qa/file.ts• L243-268 ecosystem-qa/…/filing.ts• L1-300

Operating modes

The application supports local review and scheduled or manually dispatched CI sweeps. Local runs let an engineer inspect proposals before filing. CI runs distribute project and platform combinations, upload each result as an artifact, then collect and file the combined proposals.

The package exposes separate commands for collection, filing, running a sweep, type checking, and tests. A fake run supplies a safe path through collection and dry filing when no agent key or external project run is available.

At the workspace level, ecosystem-qa has an implicit dependency on agent-eval. Its check target has no target-level prerequisite, so this relationship records project impact without making one check run before the other.

References: ecosystem-qa/package.json• L1-30 ecosystem-qa/README.md• L28-80 ecosystem-qa/README.md• L82-97 ecosystem-qa/project.json• L1-8

Change checklist

When these files or sections are changed, remember to consider these:

  • Keep the catalog fields, run request, and result record aligned. A new project or platform option must survive validation and remain usable by collection and filing.
  • Keep the check list, prompt rendering, and structured output rules aligned. A check id or tier change can otherwise produce missing, duplicate, or unfiled findings.
  • Keep the reference checklist's manual rows aligned with the agent registry. A removed or unsupported Storybook feature must still have a clear review path when it is not agent-runnable.
  • Keep evidence path rules aligned across result validation, redaction, and asset publishing. A path accepted by one stage must still resolve inside the intended evidence area.
  • Keep proposal fingerprints and GitHub search state aligned. Changes to grouping or issue matching can create duplicate reports or suppress a new regression.
  • Keep protected cycle refs and asset publishing separate from normal branch workflows. A change to either destination can expose agent changes or credentials to repository automation.

Important failure modes

TriggerConsequenceGuard
A selected check is missing or appears more than once in agent outputThe run is rejected as invalid structured outputThe output schema requires each selected check exactly once and findings only for failed checks
The clone contains a .sandcastle directoryThe agent could receive the project’s environment valuesThe run refuses the repository before starting the agent
Public output contains a credential-shaped valueLogs or evidence can expose a provider keyThe complete output tree is redacted before publication
The target milestone or label is missingIssues could be filed with incomplete routingFiling checks all prerequisites before any issue is created
A cycle is pushed as a normal branchRepository workflows could run unreviewed agent changesFiling pushes the cycle to a protected QA ref
A finding is repeated across projects or enginesThe issue tracker receives duplicate reportsCollection groups by fingerprint and filing checks existing issues and comments
An upgraded project still uses @storybook/test-runnerThe Storybook 11 upgrade stops before compatibility checks can runThe upgrade blocks the project and offers migration to the Vitest addon or continued use of Storybook 10

Key files

FileWhy to read it
ecosystem-qa/README.mdExplains the product goal, local and CI flows, artifacts, and dry filing behavior.
ecosystem-qa/catalog.jsonShows the catalog input that selects real projects for a sweep.
ecosystem-qa/run.tsShows the end-to-end run lifecycle from clone and sandbox through result and redaction.
ecosystem-qa/checks.tsDefines the compatibility stages and the checks an agent can run.
ecosystem-qa/…/basic-qa.mdDefines the wider manual and agent QA checklist, including upgrade paths that the registry does not run automatically.
ecosystem-qa/…/schema.tsDefines the contracts for catalog entries, results, findings, and proposals.
ecosystem-qa/collect.tsShows how run artifacts become grouped proposals.
ecosystem-qa/file.tsShows dry and live filing, asset publication, cycle publication, and repeat-safe behavior.
ecosystem-qa/package.jsonLists the supported command entry points for operating the application.
ecosystem-qa/project.jsonShows the workspace dependency link to agent evaluation and the project check target.