Ecosystem quality assurance
Ecosystem QA checks how a Storybook version behaves in real external projects. It upgrades a catalogued project in a controlled run, records check results and evidence, removes secrets from public output, then turns repeat findings into safe issue or comment proposals.
System purpose
The system acts as a repeatable compatibility sweep for projects outside the main repository. It does not decide that a behavior is a bug by itself; an agent applies the shared checklist, while the harness records enough context for people to review the result.
The application complements developer workflows by testing the upgrade path in real projects. It is distinct from agent evaluation, which measures agent behavior on setup tasks rather than compatibility across an ecosystem.
Catalog scope
The catalog defines the projects that a sweep may inspect and the facts needed to run each project. Each entry names a repository revision, project class, package manager, Storybook directory, and Windows support; catalog ids are unique. The initial catalog contains the owned storybookjs/mealdrop project and uses Yarn.
References: ecosystem-qa/catalog.json• L1-11 ecosystem-qa/…/schema.ts• L5-34
Check model
Checks cover the upgrade, development server, build, and test stages. A must tier runs in every sweep, while extended adds broader compatibility checks. The same check set becomes the agent checklist and the required shape of its answer, so each selected check is reported once and findings attach only to failed checks.
The checks move through the project lifecycle. Earlier stages leave the installed, running, or built project ready for the next stage. Results can be pass, fail, n/a, blocked, or skip, which lets the report separate a real failure from an inapplicable or unavailable check. The reference checklist also includes manual upgrade checks outside the agent registry, such as the clear migration path required when @storybook/test-runner is installed for a Storybook 11 upgrade.
References: ecosystem-qa/checks.ts• L1-16 ecosystem-qa/checks.ts• L18-30 ecosystem-qa/checks.ts• L232-246 ecosystem-qa/…/basic-qa.md• L92-107 ecosystem-qa/…/schema.ts• L48-105
Quality assurance run
One run compares a target Storybook version with a catalogued project on one operating system and one agent engine. The run creates an isolated project copy, installs its dependencies, asks the agent to perform the selected checks, and writes a stable result record even when the agent fails or returns invalid output.
The main run path follows this sequence.
flowchart LR
C[Catalog entry] --> K[Clone project revision]
K --> S[Install and upgrade in sandbox]
S --> A[Agent runs checklist]
A --> E[Evidence and session output]
A --> R[Validated result.json]
E --> X[Redacted public artifact]
R --> X
Run isolation
The run uses a sandbox to limit the agent while still allowing the project to install, build, serve, and capture evidence. Docker is the broad CI option; native Linux and macOS sandboxes support local use. An unsandboxed run is restricted by project class and the reviewed Windows exception, while fake runs execute no project or agent work.
The run also keeps the agent session, logs, evidence, and an optional cycle bundle. These artifacts support review and later inspection without making the run itself the place where a compatibility decision is made.
References: ecosystem-qa/run.ts• L48-59 ecosystem-qa/run.ts• L152-184 ecosystem-qa/run.ts• L213-288 ecosystem-qa/run.ts• L312-410 ecosystem-qa/…/sandbox-policy.ts• L1-120
Run result
result.json is the contract between a run and the later collection stages. It records the request, installed version, source and result revisions, checks, findings, model usage, timing, machine state, artifact paths, and overall status. A result can therefore be compared across projects, operating systems, engines, and Storybook versions without reopening the live agent session.
The result separates an agent or output failure from a check failure. A completed run may report findings while still having status ok; a timeout, agent error, or invalid structured answer is recorded as a run status instead.
References: ecosystem-qa/run.ts• L422-464 ecosystem-qa/…/schema.ts• L121-157
Evidence and privacy
Evidence gives a finding a reviewable link to what the agent saw. A finding can include logs, a screenshot, an animation, reproduction steps, the expected and actual behavior, and a severity hint. File names are constrained to the evidence area so the report can reference artifacts without allowing paths to escape it.
The complete output directory becomes a public artifact, so redaction is a final boundary rather than an optional display step. The system removes credential-shaped data from the run output before it prints or publishes the artifact, and the filing stage copies only redacted evidence to the shared asset branch.
References: ecosystem-qa/…/schema.ts• L43-70 ecosystem-qa/run.ts• L422-464 ecosystem-qa/run.ts• L485-495 ecosystem-qa/file.ts• L118-149 ecosystem-qa/…/redact.ts• L1-200
Finding collection
Collection combines results from the selected projects, operating systems, and engines into proposals for one Storybook repository. It accepts local results or downloaded CI artifacts, ignores results for another target version, groups equivalent findings by fingerprint, and checks GitHub for prior reports.
The collection flow turns separate observations into one of three proposal kinds:
| Proposal | Meaning |
|---|---|
| Issue | A finding has no open matching report and needs a new actionable issue. |
| Comment | A matching open issue exists and needs new run evidence or context. |
| Suppressed | A matching known issue already marks the finding as understood. |
This grouping keeps repeated observations from creating a new report for every project or engine. It also preserves the source runs and the cycle information needed to reproduce the result.
References: ecosystem-qa/collect.ts• L10-17 ecosystem-qa/collect.ts• L47-67 ecosystem-qa/…/schema.ts• L159-220 ecosystem-qa/…/proposals.ts• L1-220
Compatibility proposal filing
Filing converts proposals into controlled repository actions. Dry mode writes the proposed issue body and evidence plan to the job summary without changing GitHub; live mode checks that the destination has the required milestone and labels before it files anything.
For a new issue, the workflow can publish the upgraded project state and evidence separately from the issue. The upgraded state goes to a protected QA ref rather than a normal branch, which prevents repository workflows from running agent changes before review. Evidence goes to the qa-assets branch, and the issue links the material needed to inspect it.
Filing is designed to be safe to repeat. Existing fingerprints, identical comments, and already-published evidence are skipped. One proposal may fail without blocking the others, but the final job result reports the failures.
References: ecosystem-qa/file.ts• L25-32 ecosystem-qa/file.ts• L93-149 ecosystem-qa/file.ts• L199-241 ecosystem-qa/file.ts• L243-268 ecosystem-qa/…/filing.ts• L1-300
Operating modes
The application supports local review and scheduled or manually dispatched CI sweeps. Local runs let an engineer inspect proposals before filing. CI runs distribute project and platform combinations, upload each result as an artifact, then collect and file the combined proposals.
The package exposes separate commands for collection, filing, running a sweep, type checking, and tests. A fake run supplies a safe path through collection and dry filing when no agent key or external project run is available.
At the workspace level, ecosystem-qa has an implicit dependency on agent-eval. Its check target has no target-level prerequisite, so this relationship records project impact without making one check run before the other.
References: ecosystem-qa/package.json• L1-30 ecosystem-qa/README.md• L28-80 ecosystem-qa/README.md• L82-97 ecosystem-qa/project.json• L1-8
Related pages
- Developer workflows
- Project setup and migration
- Testing fixtures and sandboxes
- Repository automation
- Agent evaluation platform
Change checklist
When these files or sections are changed, remember to consider these:
- Keep the catalog fields, run request, and result record aligned. A new project or platform option must survive validation and remain usable by collection and filing.
- Keep the check list, prompt rendering, and structured output rules aligned. A check id or tier change can otherwise produce missing, duplicate, or unfiled findings.
- Keep the reference checklist's manual rows aligned with the agent registry. A removed or unsupported Storybook feature must still have a clear review path when it is not agent-runnable.
- Keep evidence path rules aligned across result validation, redaction, and asset publishing. A path accepted by one stage must still resolve inside the intended evidence area.
- Keep proposal fingerprints and GitHub search state aligned. Changes to grouping or issue matching can create duplicate reports or suppress a new regression.
- Keep protected cycle refs and asset publishing separate from normal branch workflows. A change to either destination can expose agent changes or credentials to repository automation.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| A selected check is missing or appears more than once in agent output | The run is rejected as invalid structured output | The output schema requires each selected check exactly once and findings only for failed checks |
The clone contains a .sandcastle directory | The agent could receive the project’s environment values | The run refuses the repository before starting the agent |
| Public output contains a credential-shaped value | Logs or evidence can expose a provider key | The complete output tree is redacted before publication |
| The target milestone or label is missing | Issues could be filed with incomplete routing | Filing checks all prerequisites before any issue is created |
| A cycle is pushed as a normal branch | Repository workflows could run unreviewed agent changes | Filing pushes the cycle to a protected QA ref |
| A finding is repeated across projects or engines | The issue tracker receives duplicate reports | Collection groups by fingerprint and filing checks existing issues and comments |
An upgraded project still uses @storybook/test-runner | The Storybook 11 upgrade stops before compatibility checks can run | The upgrade blocks the project and offers migration to the Vitest addon or continued use of Storybook 10 |
Key files
| File | Why to read it |
|---|---|
| ecosystem-qa/README.md | Explains the product goal, local and CI flows, artifacts, and dry filing behavior. |
| ecosystem-qa/catalog.json | Shows the catalog input that selects real projects for a sweep. |
| ecosystem-qa/run.ts | Shows the end-to-end run lifecycle from clone and sandbox through result and redaction. |
| ecosystem-qa/checks.ts | Defines the compatibility stages and the checks an agent can run. |
| ecosystem-qa/…/basic-qa.md | Defines the wider manual and agent QA checklist, including upgrade paths that the registry does not run automatically. |
| ecosystem-qa/…/schema.ts | Defines the contracts for catalog entries, results, findings, and proposals. |
| ecosystem-qa/collect.ts | Shows how run artifacts become grouped proposals. |
| ecosystem-qa/file.ts | Shows dry and live filing, asset publication, cycle publication, and repeat-safe behavior. |
| ecosystem-qa/package.json | Lists the supported command entry points for operating the application. |
| ecosystem-qa/project.json | Shows the workspace dependency link to agent evaluation and the project check target. |