Introducing a new way to review your PRs.

Read the announcement

pbakaus / impeccable

The design vocabulary for your coding agent. Steer with polish and bolder, review with critique and audit, catch AI slop before it ships.

View on GitHub

32 documents · Updated Nov 20, 2026

docs / automated-test-harness

Automated test harness

The automated test harness checks the product at its main user and delivery boundaries. It covers detector behavior, command-line use, hooks, agent workflows, browser live mode, editor packages, plugin loading, engine downloads, and release safety. Fast local suites give repeatable checks. Browser, remote, and provider-backed suites run only when their tools and services are available. See also Distribution and engineering tooling, Build, release, and validation, Plugin packaging and installation, Live editing sessions, and Agent hook workflow.

Suite model

The test system groups checks by product boundary. Each suite has a clear purpose and a selection rule so a change runs the smallest useful set of checks.

Suite groups

The default local run covers deterministic product checks, the engine oracle, detector and extension packaging, native UI bundle parity, live helper behavior, framework fixtures, token budgets, motion capture timing, and the committed plugin in a real Claude Code loader when that CLI is available. Browser sweeps, remote install checks, and provider-backed agent checks remain opt-in because they need more time or external services.

SuiteMain guaranteeRun mode
coreRepository contracts, build output, provider transforms, hook manifests, plugin guards, release guards, workflow security, token budgets, motion capture timing, and native UI bundle parityDefault
oracleStable command and engine behavior against recorded cases and expected filesDefault; skips without an engine binary
detectorBrowser extension packaging and tracked detector registry outputDefault when its inputs change
liveLive-mode reference behavior and helper contractsDefault
frameworkEngine live injection, CSP detection, and source wrapping across project shapesDefault; skips without an engine binary
plugin-e2eComponent discovery by the real Claude Code loaderDefault; skips without claude
live-e2eFull browser click-to-accept behavior across runtime fixturesOpt-in; Playwright
new-work-e2eBrowser decisions for new-work concept, direction, build-path, and question flowsOpt-in; Playwright and an engine binary
cli-remote-e2eRemote CLI install and update smokeOpt-in; behavior moved to the engine repository
live-e2e-accept-cleanupProvider-backed cleanup after live acceptanceOpt-in; Playwright and a provider key
skill-behaviorProvider-backed setup, routing, questions, and write orderOpt-in; language model keys
skill-workflowProvider-backed attended builds, visual evidence, handoff, and system documentationOpt-in; Playwright, engine, and language model keys

The suite registry is the source of truth. A registry check finds every test file, requires it to belong to a suite, prevents duplicate default membership, and confirms that editing a test file selects a suite that runs it. The registry derives self-file triggers from each command, so a changed test file cannot silently fall outside change-based CI.

References: scripts/test-suites.mjs• L4-13 scripts/test-suites.mjs• L28-117 scripts/test-suites.mjs• L118-235 tests/test-suites.test.mjs• L14-75

Test selection

Local commands expand default, all-local, and all into named suites. CI always runs the core checks, then maps the event and changed paths to deterministic jobs. A pull request normally runs only affected deterministic groups. A manual run enables opt-in groups. The nightly schedule adds the full live browser matrix without enabling paid provider runs.

The selection path turns an event and its changed files into workflow outputs. Jobs then install only the browser, engine, or provider tools that their selected checks need. The runner starts each command in its own process group. It applies time limits, stops a wedged group, and fails the suite if the run leaves a marked live server behind.

This diagram shows how a change becomes a set of verification jobs.

flowchart LR
  C[Event and changed paths] --> P[CI test plan]
  P --> D[Deterministic jobs]
  P --> E[Opt-in jobs]
  D --> O[Core, oracle, detector, live, framework]
  E --> B[Browser, remote, or provider checks]
  O --> V[Build and tracked-output checks]

References: scripts/run-tests.mjs• L21-55 scripts/run-tests.mjs• L65-119 scripts/ci-test-plan.mjs• L6-59 tests/ci-test-plan.test.mjs• L10-135 .github/…/ci.yml• L24-143

Shared test environments

The harness stages real project trees and tool processes in temporary locations. Tests can observe source writes, generated files, process output, and final state without changing the checkout, the developer's home directory, or a real agent installation.

Temporary workspaces

Temporary workspaces give CLI, engine, live, release, and agent tests a known file system. The agent harness limits file tools to workspace-relative paths and records reads, writes, questions, commands, and changed paths. This makes the order of an agent's actions part of the evidence, not only its final answer.

The engine locator accepts an explicit IMPECCABLE_BIN, a fetched platform binary, or a local release build. Oracle and browser suites can therefore use the same engine contract in CI and during local source work. Provider-backed runs use explicit credentials and bounded turns. Local credentials can come from a private ~/…/dev.env file or an override path. Child processes receive a filtered environment so provider keys do not spread into fixture servers or shell helpers.

References: tests/…/harness.mjs• L1-19 tests/…/harness.mjs• L37-61 tests/…/harness.mjs• L119-191 tests/…/dev-keys.mjs• L1-12 tests/…/dev-keys.mjs• L48-87 tests/…/engine-bin.mjs• L1-49 tests/…/session.mjs• L48-64

Framework fixtures

Framework fixtures are small project shapes with expected live-mode results. The static fixture pass stages each tree as a temporary Git project and checks injection, removal, content security policy classification, and source target selection. Fixtures with a runtime block also install dependencies, start a development server, and enter the browser suite.

The fixture contract lets one project shape support both narrow engine checks and a user-visible browser flow. It now covers a nested application below a repository root, Vite 8 with React, Vite 7 with a Vue single-file component, manual copy changes, duplicate targets, orphaned wrappers, and foreign sessions. The Vue case keeps reactive form state in source while the harness checks that a live session can select, cycle, and accept the form without losing its source structure. These cases protect root selection, source ownership, state preservation, and safe recovery without treating a successful browser render as proof that source edits are correct.

References: tests/framework-fixtures.test.mjs• L30-74 tests/framework-fixtures.test.mjs• L93-264 tests/…/README.md• L1-14 tests/…/README.md• L102-121 tests/…/README.md• L144-176 tests/…/fixture.json• L1-41 tests/…/fixture.json• L1-55 tests/…/fixture.json• L1-63 tests/…/App.vue• L1-14

Process cleanup

Live tests mark every server that they start. In-process cleanup handles normal exits and signals. A detached reaper handles a killed test process or a timeout that prevents normal teardown. The top-level runner adds a second check after each command, so a leaked server cannot poison a later test with a stale port or session.

References: tests/…/live-servers.mjs• L1-19 tests/…/live-servers.mjs• L45-83 tests/live-server-leak.test.mjs• L129-213 tests/…/lib.mjs• L31-39

Behavioral verification

The harness compares user-visible behavior with stable expected results. Recorded cases provide broad command coverage. End-to-end runs cover behavior that needs a browser, a real loader, or a real provider.

Oracle corpus

The oracle corpus is a behavior record for engine commands. Each case supplies a command, inputs, a staged workspace, and files to inspect. The runner compares output, errors, exit status, signals, and selected files after it normalizes paths and other run-specific values.

The corpus covers context and doctor checks, detector commands, CLI help and version output, hooks, live commands, design work, questions, image helpers, component-review planning, manual copy apply and discard, skill installation, local control commands, and motion-capture command boundaries. Recent cases pin loop help and unknown-verb failures, signed-out critique preference state, DESIGN.md sync settings and refresh behavior, setup status output, and motion-capture URL-scheme refusal. Detector records also include wider edge cases for layout, contrast, typography, page-level styles, overlays, mock frames, SVG text, moving evidence, and source forms. Hook records cover provider-shaped envelopes, duplicate events, session cache state, per-edit scans, Stop events, linked stylesheet ownership, the Hermes plugin's Stop decision path, and a VS Code patch-string response. Context records cover monorepo surface-brief targets from the caller's directory and workspace discovery in a tooling-heavy monorepo. Question records cover state recorded under another project root, missing state, hidden-tab heartbeats, answer collection, and stopping one root without removing another. Live records now cover locator text that resolves to a later file or search directory, dynamic rendered text, style-only locator hits, and insert or wrap fallback when source text is absent. Comparison records also pin the offset used for each component-diff region.

Oracle workspaces use real paths and stable placeholders for the workspace, repository, home directory, timestamps, process IDs, UUIDs, and other run-specific values. Multi-step cases keep one workspace alive across commands, so the record can prove state transitions rather than only one response.

Reviewed deltas are part of the contract. A golden may differ from the previous implementation only when the behavior has a recorded explanation and review entry. Recent records cover detector fixture growth, reviewed page-level and overlay decisions, evidence motion, live manual removal, installer detection, ignore and advisory boundaries, output changes caused by the native engine, question state lookup, monorepo brief lookup, directory scans that include .mjs and .cjs files, the exact additional-context shape for a VS Code patch event, and per-region component-diff offsets.

References: tests/…/lib.mjs• L1-25 tests/…/lib.mjs• L47-101 tests/…/lib.mjs• L248-305 tests/…/run.mjs• L1-32 tests/…/README.md• L1-35 tests/…/motion.mjs• L1-7 tests/…/critiques.mjs• L1-21 tests/…/health-design.mjs• L1-21 tests/…/setup.mjs• L1-5 tests/…/context.mjs• L1-25 tests/…/hooks.mjs• L1-18 tests/…/hooks.mjs• L98-126 tests/…/live-wrap.mjs• L1-11 tests/…/live-wrap.mjs• L89-100 tests/…/package.json• L1-5 tests/…/DELTAS.md• L7334-7351 tests/…/critiques-on-signed-out.json• L1-19 tests/…/health-design-on.json• L1-8 tests/…/setup-status-default.json• L1-7 tests/…/live-wrap-text-picks-later-file.json• L1-3 tests/…/comp-diff-json.json• L1-7

Detector and extension checks

Detector verification uses the engine's own tests and the oracle corpus for rule behavior. The repository-side detector suite checks the browser extension package and the tracked rule registry. Fixtures pair flag and pass cases across common source formats and project shapes, so a rule must identify a real issue without treating valid, hidden, disabled, or unpainted states as failures.

The fixture corpus now tests painted color palettes, text and control contrast, gradients, spacing at visible edges, em-dash use, heading structure, nested cards, mock frames, font loading, page-level colors, motion, SVG text, overlays, and evidence that changes after a scan. Paired cases also separate static source checks from browser-only judgments such as paint order, masks, covered panels, viewport geometry, loaded fonts, and the page canvas. The broken-image checks treat srcset and framework source bindings as valid image sources while still reporting empty or unrelated bindings. Source scans also ignore image-like text inside JavaScript and TypeScript literals. Directory scans keep their file set aligned with direct scans by walking JavaScript module files such as .mjs and .cjs.

The extension checks exercise startup in Chrome-style and Firefox-style hosts. They prove that concurrent scans share one core host, that a failed host can retry, that scanning waits for a successful readiness response, and that WASM startup errors are reported instead of becoming a false clean result. DevTools panel checks add the copy boundary: advisory findings stay advice, mapped advisory fixes remain available, and ordinary findings keep their repair suggestions. Popup checks add the active-tab boundary: partial reports remain visible but cannot be copied, navigation and failed scans invalidate old results, and delayed state or clipboard work cannot overwrite a newer report. Repeated scans keep one overlay, and browser-facing smoke pages cover clean and flagged CSP-constrained pages.

Account checks extend this boundary to sign-in and paid walks. They verify PKCE callbacks, issuer and endpoint pairing, scope negotiation, durable refresh-token state, worker restart recovery, sender checks, server switching, sign-out, journey notices, persona data, and safe image loading. Agent checks verify the native host link, developer-build admission, store-build removal, side-panel consent, per-site access, secret redaction, same-tab control, session stop, and host loss. Walk checks verify hand-off origin and socket pairing, store and Firefox capability differences, cross-origin write guards, sign-in pauses, page reads, takeover, cleanup, and the shared web-app protocol.

The shared report formatter is checked separately from its popup and DevTools callers. It keeps the scanned URL, finding count, selector, category, detail, page-level location, advisory section, and mapped skill suggestion consistent across both copy surfaces. Aggregate sweeps keep counted findings separate from advisory notes. A missing linked stylesheet is reported as an incomplete scan warning rather than as a false finding. Directory checks also distinguish configured template files from plain files and verify that ignored files, rules, and glob boundaries stay ignored in direct and page-mediated scans.

References: tests/…/detect.mjs• L45-220 crates/…/broken_image_source.rs• L1-39 crates/…/missing.html• L1-30 crates/…/broken_image.rs• L1-57 tests/…/ai-color-palette.html• L8-14 tests/…/evidence-after-scan.html• L8-29 tests/…/intro-unnamed-clears.html• L16-20 tests/…/mockup-structure.html• L5-35 tests/extension-build.test.mjs• L66-160 tests/extension-build.test.mjs• L213-282 tests/extension-build.test.mjs• L329-518 tests/extension-devtools.test.mjs• L56-156 tests/extension-popup.test.mjs• L92-287 tests/extension-popup.test.mjs• L360-861 tests/extension-account.test.mjs• L19-81 tests/extension-account.test.mjs• L152-280 tests/extension-account.test.mjs• L285-476 tests/extension-account.test.mjs• L722-803 tests/extension-agent.test.mjs• L16-36 tests/extension-agent.test.mjs• L149-340 tests/extension-walks.test.mjs• L46-124 tests/extension-walks.test.mjs• L236-280 tests/extension-walks.test.mjs• L446-600 tests/extension-report.test.mjs• L12-38 tests/…/README.md• L1-106 tests/…/deferred.html• L20-25 tests/…/detect-fixture-json-page-level-forms-dark-contents-html.json• L1-7 tests/…/detect-fixture-text-framework-next-modules.json• L1-7

Motion capture checks

Motion capture checks verify both command boundaries and browser evidence. The browser-free oracle cases pin help, usage, and unsafe URL handling. The native fixture suite runs a real capture and checks findings, frame phases, rendered frame files, reduced-motion behavior, and settled completion.

The shared capture harness models playback rate, current position, reverse playback, finished entrance animations, frozen timelines, and sub-millisecond animations. The smooth fixture supplies a clean view transition and checks that reduced motion makes it instant instead of sampling a false transition.

References: tests/motion-capture.test.mjs• L6-80 tests/…/motion.mjs• L1-7 tests/…/smooth.html• L1-4 crates/…/motion.rs• L11-65 crates/…/motion.js• L220-268

Live browser flow

The live end-to-end suite follows a real design session from a staged project to an accepted source change. It proves the browser handshake, page steering, element pick or insertion, agent output, variant cycling, tuning, acceptance, cleanup, and reload behavior.

The default agent is deterministic. Its variants have distinct rendered styles, so the test can prove that the selected variant mounted in the browser instead of trusting only a status label. A language-model agent can use the same interface for selected runs. The fake agent also supplies insert variants, component-preview variants, parameter data, and partial manual-edit results so the browser can verify safe progress.

The browser flow also covers a nested app root, Svelte component previews, state preservation, manual copy staging, element removal, annotations, insert mode, agent-targeted work, discard recovery, and exit. It checks source cleanliness after acceptance and confirms that temporary preview files do not become product source. A focused provider-backed regression also checks that accepted source cleanup reaches durable completion and removes stale browser wrappers. Live-server security checks add the boundary around this flow: in-workspace symlinks work, links outside the workspace are denied, broken links return not found, project ignores and page roots reach the live script, page-controlled poller fields do not reach the agent, and ref-scoped discard preserves sibling edits. Source contracts also check that generation checkpoints follow acknowledged sessions, manual edits carry source and element context, safe discard restores only the intended text or element, framework-owned wrappers yield to reload recovery, and accepted sessions release the foreground UI without losing cleanup.

This sequence shows the main live proof from project setup to clean source.

sequenceDiagram
  participant F as Fixture project
  participant S as Live server
  participant B as Browser
  participant A as Agent
  F->>S: Start and inject live mode
  B->>S: Complete handshake
  B->>A: Send steer or generate event
  A->>S: Write variants and report progress
  S-->>B: Stream state and variants
  B->>B: Cycle, tune, edit, and accept
  S->>F: Persist accepted source and clean preview state
  B->>B: Verify DOM, requests, console, and reload

The browser checks reject missing preview resources, stale wrappers, failed mounts, and console errors that an unchanged original page could hide. Helper tests also protect ignore matching, session persistence, UUID fallback, generation ordering, manual-edit staging, safe discard restoration, removal undo, framework-owned cleanup, and recovery after reload or hot-module replacement. Extra scenarios cover mount retry, republish, storage loss, missed completion events, event ordering, agent targeting, modal top-layer behavior, and accepted-session cleanup.

The session helper stages the fixture, installs its dependencies, starts the live server, injects the runtime, starts the framework server, and returns one teardown path. It supports applications below the repository root so root selection and app selection are tested as part of the user flow. The server reaper protects runs where normal teardown does not happen.

Static-page acceptance covers the case where no hot-module reload or auto-reload can repaint the page after acceptance. It checks that the accepted variant keeps its scoped style and tuned values, that the original and rejected variants stay hidden, and that a second selection remains safe after the first acceptance.

References: crates/…/live_server_security.rs• L1-5 crates/…/live_server_security.rs• L65-170 crates/…/live_server_security.rs• L173-260 tests/live-e2e.test.mjs• L1-19 tests/live-e2e.test.mjs• L78-180 tests/live-e2e.test.mjs• L618-740 tests/live-e2e-accept-cleanup-regression.test.mjs• L47-141 tests/…/agent.mjs• L85-142 tests/…/agent.mjs• L223-237 tests/…/session.mjs• L1-20 tests/…/session.mjs• L287-352 tests/…/ui.mjs• L1-68 tests/live-browser-source.test.mjs• L11-83 tests/live-browser-source.test.mjs• L352-489 tests/live-browser-source.test.mjs• L491-755 tests/live-browser-source.test.mjs• L853-1083 tests/live-browser-source.test.mjs• L1106-1248 tests/live-svelte-adapter-deepseek.test.mjs• L71-154 tests/live-static-accept.test.mjs• L1-147

New-work browser flow

The new-work end-to-end suite checks the browser decision page and its command-line round lifecycle. It uses a real Chromium session, a real engine binary, and deterministic fake image generation so concept choices, re-rolls, follow-up rounds, build-path changes, sketches, composition approval, tab closure, and stalled or dead servers can be checked without language-model calls.

The flow starts a question round from an isolated workspace, observes the page through a scripted user bot, and collects the answer through --wait. A re-roll keeps the round open until a new hand arrives. A terminal choice removes the round state. Heartbeats, deadlines, reloads, server loss, and recorded roots prove that the page does not claim a choice when the round cannot accept it.

This sequence shows how a new-work decision moves between the engine, browser, and workspace.

sequenceDiagram
  participant W as Workspace
  participant E as Engine
  participant P as Browser page
  participant U as User bot
  W->>E: Start question round
  E-->>P: Serve decision page
  U->>P: Pick, steer, re-roll, or close
  P->>E: Answer or heartbeat
  E-->>W: Record answer and round state
  E-->>P: Deliver next hand or bounded failure
  P->>W: Expose selected direction or approved comp

The same suite checks that synthetic images are repeatable for one prompt, vary for another prompt, and carry a clear synthetic marker. This keeps visual decision tests deterministic while still proving that the page can load and display the expected media forms.

References: tests/new-work-e2e.test.mjs• L1-20 tests/new-work-e2e.test.mjs• L96-134 tests/new-work-e2e.test.mjs• L139-351 tests/new-work-e2e.test.mjs• L587-695 tests/new-work-e2e.test.mjs• L704-805 tests/new-work-e2e.test.mjs• L990-1167 tests/new-work-e2e.test.mjs• L1173-1225

Agent workflow

Provider-backed skill tests check how an agent prepares to work, not how well it invents a page. The harness gives the model a bounded workspace and records whether it loads product and design context, selects the right reference, asks required questions, and writes only after the required decision.

The skill-behavior scenarios cover missing context, existing context, command routing, native platform guidance, update notices, read-only advice, launcher failure, safe fallback behavior, and the difference between a browser variant request and a plain refinement. Routing cases stop at protocol checkpoints. They do not claim that a page was built or reviewed. A separate activation evaluation checks whether the skill description causes a model to load the skill for relevant requests without false activation for unrelated work. The separate skill-workflow suite owns claims about completed builds, visual evidence, handoff, and system documentation. Reference-authoring checks also keep direction contracts, native-platform limits, reduced-motion guidance, content fingerprints, visual decision fallback, card identity, hand sizing, touch checks, and audit score tables aligned with the skill flow. The live reference checks keep the foreground poll loop, manual-edit handoff, provider-specific guidance, capability-based CSS guidance, action vocabulary, and the README's Apply subprocess route aligned with the runtime. The live reference checks also keep setup read-only by default, monorepo guidance target-driven, and manual-edit dispatch ordered after prefetch and before exit.

The workflow scope checks use a valid target change as the starting point. They then prove that protected page content stays intact while target text or styles change, and that unrelated links, text, missing regions, comments, formatting-only edits, unused CSS, or unused custom properties do not count as a valid artifact. This keeps completion evidence tied to the named target rather than to any file change.

The live fake agent follows the same poll, wrap, write, and accept contract as a provider agent. Its stable output gives the browser a reliable proof of rendered variants. The agent contract now also checks exact variant counts, preserved visible copy, material source changes, progressive parameter delivery, valid CSS and HTML shape, safe insert roots, and source edits that preserve typed data and coupled lookup keys. Manual Apply reports applied and failed entries so the workflow can test safe partial progress rather than treating a write as successful only when every edit lands.

Agent-target checks extend this contract to requests that start in the CLI and resolve in a browser overlay. They verify action vocabulary, input validation, authenticated live-server access, one-tab claims, lease recovery, busy and no-match roll calls, reconnect behavior, bounded waits, and timeout results. Late or foreign results cannot create an unannounced session, and the target request is not left pending when its serving browser disappears.

The Hermes plugin uses the same launcher boundary as the other agent integrations. Its tests verify trusted project selection, launcher fallback, write and patch event translation, per-file findings, Stop decisions, disabled hooks, and reuse of the launcher selected during the edit session. Native hook tests extend this boundary with session identity safety, session-scoped completion reminders, shipped-build ownership, baseline attribution, bounded output, and provider-specific Stop decisions.

Provider-backed live runs select OpenAI, Anthropic, or DeepSeek from explicit settings and available keys. They skip when no key is available. The DeepSeek Svelte adapter sweep exercises the full flow with a real provider, including steer, replace, discard, copy Apply, annotations, insert, refresh recovery, accept, cleanup, and exit.

References: tests/…/README.md• L1-29 tests/…/README.md• L38-68 tests/…/activation.eval.mjs• L1-41 tests/…/activation.eval.mjs• L201-318 tests/…/harness.mjs• L89-117 tests/…/harness.mjs• L254-352 tests/…/assertions.mjs• L1-109 tests/skill-reference.test.mjs• L9-202 tests/skill-workflow-scope.test.mjs• L1-49 crates/…/hook_tests.rs• L22-196 crates/…/hook_tests.rs• L219-322 crates/…/hook_tests.rs• L628-988 tests/…/agent.mjs• L57-79 tests/…/agent.mjs• L96-142 tests/…/llm-agent.mjs• L54-98 tests/…/llm-agent.mjs• L100-168 tests/…/llm-agent.mjs• L205-259 tests/live-e2e-llm-agent.test.mjs• L23-125 tests/live-e2e-llm-agent.test.mjs• L1523-1692 tests/live-svelte-adapter-deepseek.test.mjs• L57-154 tests/hermes-plugin.test.mjs• L112-267 tests/live-reference.test.mjs• L9-217 tests/live-agent-target.test.mjs• L1-1176 crates/…/vocabulary.rs• L1-80

CLI and package loading

CLI verification protects the path from an installed command to a usable engine. Launcher checks cover verified download, checksum refusal, cache placement, retry diagnostics, concurrent first runs, interruption cleanup, stale staging, and quiet cached execution on both shell platforms. The npm shim is tested as a separate fallback: it honors a preinstalled binary or platform package, refuses missing or bad checksum records, answers version requests without touching a binary, and leaves no unverified cache artifact.

The npm shim tests force each lookup arm in an isolated staged package. They check the precedence of an explicit binary and an installed platform package before the cache and download paths. Downloaded bytes must match the shipped manifest. A cached binary must also match its digest, size, and verification stamp before it runs. Missing or bad records, stale stamps, host errors, and mismatched payloads fail closed without leaving cache files, while version requests remain local.

Hook-build checks protect the provider manifest contract across Claude, Codex, Cursor, GitHub Copilot, Grok, Gemini, and Antigravity. They verify event shape, launcher path, existence guards, time limits, Windows command forms where supported, plugin-root paths, and routing for unknown providers. Hook commands call the shipped launcher without a Node runtime or an old script probe. Hermes is checked as a plugin payload rather than as a hook manifest. Oracle hook cases then exercise the installed boundary over per-edit scans, Stop events, provider-shaped envelopes, session cache state, duplicate events, linked stylesheet ignore rules, and the VS Code patch-string envelope.

The Windows hook smoke test runs the generated shared command from paths with spaces and non-ASCII characters. It checks both cmd.exe and available PowerShell forms, preserves a launcher failure exit code, and treats a missing launcher as a safe no-op.

Build checks bind shared skill source to provider-specific outputs. They verify skill placement, native agent files, degraded fallbacks, provider placeholders, launcher payload contents, and the absence of engine binaries or bundled detector entry points from the source payload. They also verify that native agent files and degraded references carry the right provider-specific fields while retaining the shared agent body. Transformer checks add a narrow contract for required metadata, allowlisted optional fields, user-invocable hints, copied references, and provider markers without rewriting executable code.

The build integration checks now follow agent output across native and degraded provider surfaces. They verify that shared bodies resolve placeholders and links everywhere they ship, that provider-only blocks compile correctly, that tool restrictions remain meaningful for each host, and that Windows launcher files keep the required line endings and version alignment. OpenAI plugin staging has its own boundary: it packages the Codex skill, agent files, metadata, and icon into the ChatGPT plugin shape while excluding Claude content and bundled hooks. The release step also keeps the plugin upload visible when a skill release completes.

Cursor and VS Code checks add package-level boundaries. Cursor staging rewrites paths for a relocated plugin and preserves the executable launcher, hooks, links, and version agreement. VS Code packages the skill as a chat skill without an editor runtime or project-install sidecar. Both checks run the relocated launcher from a path with spaces and confirm that it keeps the user's project as its working directory. Installer checks also cover detected-only behavior and interactive provider choice: an explicitly selected harness is installed even when another harness already exists, while a detected but unselected harness stays unchanged. Scope checks protect home-rooted installs, provider agent freshness, relocated bundle staging, and global hook paths.

References: crates/…/drift_ports_tests.rs• L1-15 crates/…/drift_ports_tests.rs• L167-355 crates/…/drift_ports_tests.rs• L450-631 crates/…/install_picker_tests.rs• L1-129 crates/…/hook_manifest_tests.rs• L48-170 crates/…/hook_manifest_tests.rs• L464-510 tests/launcher-download.test.mjs• L22-177 tests/launcher-download.test.mjs• L201-377 tests/launcher-download.test.mjs• L466-534 tests/cli-shim.test.mjs• L1-27 tests/cli-shim.test.mjs• L152-316 tests/check-engine-release.test.mjs• L1-159 tests/hook-build.test.mjs• L33-50 tests/hook-build.test.mjs• L72-347 tests/hook-build.test.mjs• L350-507 tests/hook-build-windows.test.mjs• L1-44 tests/openai-plugin.test.mjs• L15-103 tests/build.test.js• L128-256 tests/build.test.js• L381-471 tests/build.test.js• L474-781 tests/…/provider-blocks.test.js• L5-99 tests/…/providers.test.js• L18-130 tests/…/factory.test.js• L30-203 tests/cursor-plugin.test.mjs• L37-119 tests/plugin-paths.test.js• L32-123 tests/vscode-extension.test.mjs• L27-89 tests/…/skills.mjs• L23-33

Release and artifact validation

Release checks run publishing guards in a disposable Git repository with a local bare origin. Dry runs exercise version, changelog, artifact, tag, branch, engine-source, signing, component-manifest, and release-key checks without tagging, pushing, publishing, or building a real release. The release script tests cover engine, skill, CLI, and extension paths, including release-note conversion and the pre-tag refusal boundary.

Engine release checks use a local server with fake platform assets, sidecars, and npm packages. They prove that the checksum manifest records every served asset, that a changed or missing line is reported, and that the fetch path refuses malformed or mismatched data before it writes a binary.

Loop release checks apply the same fail-closed rule to versioned loop archives. They validate semantic-version pins, require HTTPS release locations except for controlled local test bases, follow only permitted signed redirects, compare fetched bytes with the recorded digest, and refuse to overwrite an existing pin with different bytes.

Other checks bind release artifacts together. Signing tests verify that the exact bundle, version, size, digest, artifact name, and trusted key are covered by the signature. Platform package tests verify each target package and its binary checksum. Workflow tests verify action pinning, read-only CI access, protected signing, same-run artifact downloads, signing order, publication dependencies, and the absence of unsigned assets from the published set. The release workflow test also checks that publication uses one release creation step after signed assets are available. The engine release check retries a transient asset failure, so a short service outage does not look like a missing release asset. The release guard suite also scans compiled source inclusions so a file embedded by the engine cannot escape the engine-source change check.

References: tests/release.test.mjs• L1-12 tests/release.test.mjs• L46-68 tests/release.test.mjs• L70-121 tests/release.test.mjs• L146-237 tests/release.test.mjs• L294-368 tests/release.test.mjs• L388-488 tests/check-engine-release.test.mjs• L20-159 tests/check-loop-release.test.mjs• L1-58 tests/bundle-signing.test.mjs• L12-85 tests/publish-platform-packages.test.mjs• L24-113 tests/release-engine-workflow.test.js• L7-79 tests/workflow-security.test.js• L9-37

Repository maintenance checks

The core suite also verifies the GitHub pull-request sheriff. This check turns pull-request state into a safe maintenance plan: it distinguishes contributor work from maintainer decisions, keeps blocker labels current, warns after a waiting period, and closes only after a trusted warning and the configured close window. The test uses fixed dates and synthetic GitHub data, so time-based policy stays repeatable.

The checks cover review threads, requested changes, merge conflicts, failing checks, policy labels, draft transitions, contributor responses, trusted warning markers, regular-contributor exemptions, and explicit maintainer handoffs. A command-level case also proves that the sheriff can hydrate old comments and label events beyond the initial API window before it evaluates a pull request. This protects the difference between incomplete history and a real absence of a warning or blocker.

References: tests/github-sheriff.test.mjs• L21-186 tests/github-sheriff.test.mjs• L204-354 tests/github-sheriff.test.mjs• L356-498 tests/github-sheriff.test.mjs• L500-562 tests/github-sheriff.test.mjs• L564-883 scripts/…/sheriff.mjs• L147-318 scripts/…/sheriff.mjs• L321-407 scripts/…/sheriff.mjs• L605-704

Change checklist

When these files or sections are changed, remember to consider these:

  • Add every new test file to a suite whose trigger selects the file and whose command runs it.
  • When an engine command changes, update its oracle cases and reviewed deltas with the binary behavior.
  • When CLI control commands change, keep their usage errors, signed-out state, persisted settings, and no-network behavior aligned with oracle cases and golden files.
  • When detector rules, scopes, extensions, or output formats change, update fixture flag/pass pairs and the paired JSON and text records.
  • When detector advisory or ignore behavior changes, update direct scans, linked-file scans, hook results, and aggregate sweeps together.
  • When a new fixture has linked styles, framework syntax, or browser-only assumptions, keep its source files, directory sweeps, and both output modes aligned.
  • When framework fixture behavior changes, keep the fixture manifest, source shape, install contract, runtime probes, and expected accepted source aligned.
  • When broken-image source parsing changes, keep HTML, framework bindings, srcset, and JavaScript or TypeScript literal cases distinct in both native and oracle coverage.
  • When a directory scan extension changes, recheck direct-file parity, regex-only file counts, scope scans, and accepted sweep deltas.
  • When a consent or overlay fixture hides site state, keep hidden-page scoring, true wall or gate classification, evidence, and restored page state aligned.
  • When extension bundle inputs change, check the rule registry, generated output, package paths, browser startup, account and native-agent boundaries, walk hand-off rules, popup state, shared copy formatting, DevTools behavior, and browser-specific hosts as one boundary.
  • When popup or report state changes, keep active-tab identity, scan completion, navigation invalidation, clipboard status, advisory grouping, and URL or selector formatting aligned across popup and DevTools.
  • When account sign-in or persona flows change, keep callback checks, issuer selection, token persistence, worker recovery, sender checks, server switching, and side-panel messages aligned.
  • When walk protocol or browser-agent behavior changes, keep hand-off validation, socket pairing, permissions, sign-in pauses, takeover, cleanup, and store or Firefox staging aligned.
  • When deferred extension checks change, keep the fixture's viewport position, image source, worker-restart path, current-document recovery, and cancellation behavior aligned.
  • When motion capture changes, keep the browser timing harness, motion fixtures, oracle command cases, report frame phases, reduced-motion output, and native findings aligned.
  • When live event, variant, parameter, manual-edit, targeting, modal, or cleanup shapes change, update the fake agent, session helper, browser assertions, and source-side checks together.
  • When agent-target requests change, keep CLI validation, live-server vocabulary, overlay claims, leases, roll calls, timeout verdicts, and generate-event fencing aligned.
  • When live-server routes or event ingestion changes, check workspace symlink confinement, project-ignore serialization, poller field filtering, and ref-scoped manual discard together.
  • When live locator or source-wrap behavior changes, keep text disambiguation, later-file search, dynamic text fallback, generated-file refusal, and HTML or component fixture coverage aligned.
  • When static-page acceptance changes, keep immediate wrapper removal, scoped styles, tuned values, hidden variant state, and the next-session interaction aligned.
  • When question state can move between a caller directory and a recorded project root, update start, wait, update, stop, heartbeat, cleanup, and the no-state diagnostic as one contract.
  • When a hook builder or provider path changes, check generated manifests, launcher guards, platform variants, version alignment, Hermes packaging, OpenAI plugin staging, and real loader discovery.
  • When Windows hook commands change, run the spaced-path shell checks and preserve both launcher failure propagation and missing-launcher no-op behavior.
  • When hook session ownership or baseline attribution changes, keep per-edit, Stop, continuation, provider envelope, cache, and output-budget cases aligned.
  • When a launcher or npm shim download path changes, preserve lookup precedence, per-run staging, checksum refusal, cache cleanup, retry diagnostics, and quiet cached execution on both shell platforms.
  • When provider-key loading changes, preserve environment-over-file precedence, private file checks, child-environment filtering, and provider skip behavior.
  • When skill text or its loading plan changes, keep file caps, use caps, and CLI context and hook output caps aligned with scripts/token-budget.json.
  • When surface-brief target rules change, check reads and writes from repository roots and workspace child directories without confusing a real nested directory with a workspace prefix.
  • When interactive install selection changes, check explicit picks, mixed installed and missing providers, detected-only behavior, scope, and provider-specific bridge files.
  • When skill or agent source changes, verify every emitted provider surface, degraded fallback, placeholder replacement, relative link, and protected-content workflow check.
  • When engine release assets or checksum manifests change, check every target asset, sidecar, npm package pin, loop archive pin, and fetch failure path together.
  • When release guards or release-source inclusion rules change, keep component manifests, changelog labels, signing sidecars, tag state, engine tags, embedded inputs, and dry-run refusal order aligned.
  • When a comparison report changes, keep region alignment, per-region offsets, scores, verdicts, and no-spec output consistent.
  • When a release input or workflow changes, check version sources, checksums, signing order, package publication, action pins, artifact permissions, and token scope together.
  • When pull-request policy changes, keep blocker classification, maintainer handoffs, trusted warning and close markers, label events, hydrated history, and dry-run or scheduled thresholds aligned.

Important failure modes

TriggerConsequenceGuard
A test file is not in the suite registryChanged-path CI can omit the new testRegistry tests compare discovered files, suite membership, and own-file triggers
An oracle golden changes without a reviewed deltaA behavior change can enter the engine unnoticedOracle replay fails on missing or unreviewed differences
A CLI control command changes its default or signed-out state without new recordsAutomation can misread usage, consent, or persisted settingsOracle cases compare stdout, stderr, exit status, and isolated state files
Detector behavior changes without matching JSON and text recordsCLI output can drift between output modesDetector oracle records compare fixture, directory, scope, and format results
Source image parsing treats a binding or code literal as markupFramework code can report false broken-image findings or miss real empty sourcesNative HTML and source tests compare valid srcset and bindings with empty and literal cases
A directory walker omits a supported module fileA directory scan can look clean while a direct file scan reports findingsOracle framework records compare file counts and findings for the .mjs and .cjs cases
A consent or overlay fixture hides site stateA scan can change the page or misclassify a real gateBrowser fixtures check opt-out scans, wall and gate refusal, evidence, and restoration
The extension host fails or its readiness response is ignoredA page can appear clean when no scan ranStartup tests require one shared host, readiness before delivery, retry after failure, and visible scan errors
A popup accepts a partial, old-tab, or pre-navigation reportUsers can copy incomplete findings or findings from another pagePopup tests gate copy on scan completion and active URL, then invalidate state on navigation and failure
An account callback accepts the wrong state, issuer, or callback hostA token exchange can attach to the wrong sign-in or serverAccount tests reject repeated or mismatched callback fields and unpaired metadata
An extension message or walk hand-off comes from an untrusted senderA page or foreign extension can start work or reach account actionsAccount, agent, and walk tests require the extension origin, approved page, and paired socket
A DevTools advisory is treated as a repair commandThe panel can prescribe an unsafe or unwanted changePanel and shared-report tests keep advisory presentation separate from mapped repair suggestions
A motion capture accepts an unsupported timing or leaves an animation runningReport frames can describe incomplete or unbounded motionMotion tests bound wall duration and require all animations to settle
Reduced-motion capture samples a sub-millisecond transitionA clean reduced-motion page can produce false motion evidenceCapture tests treat sub-millisecond motion as instant and check zero duration
A live preview request fails while the original page still rendersThe test can report success without showing the generated variantBrowser checks reject live-tree request failures and verify rendered style
A static page loses its accepted style after the first sessionUsers can see the wrong source state or a later session can affect accepted contentStatic-page browser checks inspect immediate visibility, scoped style, tuned values, and a second interaction
A live-server source path follows a symlink outside the workspaceA live session can expose a file that the project does not ownSecurity checks allow in-workspace targets and reject outside or broken links
A test process leaves a live server runningLater suites can attach to a stale port or sessionRun markers, the reaper, and the runner's leak check kill and report survivors
A question round loses its recorded root after the caller changes directoryWait or stop can report a dead round or act on the wrong projectOracle cases check root pointers, missing-state diagnostics, scoped stop, and answer collection
A routing test is treated as a completed workflowAn agent can pass context loading without building or reviewingSkill behavior stops at protocol checkpoints and the workflow suite owns completion claims
A protected page change is judged only by any file differenceAn agent can claim a valid artifact while changing unrelated contentScope checks require the named target to change and protected content to remain intact
An activation description fires for unrelated requestsThe agent loads the wrong skill and changes task routingActivation evaluation requires zero false activations and model-specific recall floors
A provider key file is readable by other users or leaks into a childCredentials are exposed or a fixture process gains unrelated provider accessKey tests require private files, preserve environment precedence, and strip credential names from child environments
An explicitly selected installer provider is treated as already satisfied by another providerA requested harness is not installed even though install reports no workPicker tests distinguish named selection from detected-only selection
A Hermes project is untrusted or its launcher is missingThe plugin can run in the wrong project or silently skip design checksPlugin tests require trusted-root selection, home fallback, and no-op behavior without a launcher
A hook manifest names a missing launcher or retains a legacy script pathThe provider silently skips the hook or runs an obsolete entry pointHook builders and manifest tests require the current launcher and preserve unrelated settings
A provider hook uses the wrong platform form or install rootHooks work for one host or scope and silently no-op elsewhereBuild, manifest, and shell-runner tests check launcher paths, guards, Windows forms, and provider routing
A Windows hook path contains spaces or a launcher returns failureThe hook can fail to start or hide a blocking resultWindows shell tests run spaced paths and require nonzero launcher status to remain nonzero
An OpenAI plugin includes hooks or reads the Claude payloadPlugin review can reject the package or load the wrong skillOpenAI staging tests require Codex-only content, ChatGPT metadata, and no hook directory
An agent-target result comes from a non-holder or arrives after timeoutThe wrong tab can answer a request or start a session the agent never sawLive target tests enforce leases, target ownership, request fencing, and delivered status
A launcher or npm shim uses an unverified binaryAn untrusted engine can enter the cache or run as a valid installLauncher and shim tests require a manifest digest, size-aware stamp, clean refusal, and lookup precedence
A download sidecar or checksum line is missing or mismatchedAn unverified engine can enter the cache or release packageLauncher, shim, fetch, and engine-release tests refuse the payload and leave no cache artifact
A loop archive pin is overwritten by different bytes or an insecure redirectRelease verification can trust the wrong archive or weaken transport securityLoop release tests validate semver, HTTPS, signed redirects, digest equality, and no-overwrite behavior
A transient release service failure is treated as a missing assetA complete release can fail verification during a short CDN outageEngine release checks retry failed asset requests before reporting a missing asset
A release guard misses an embedded engine inputEngine source can change after the engine tag without blocking a dependent releaseRelease tests scan compiled source inclusions and compare them with guarded paths
A relocated package keeps a project-relative launcher pathAn installed plugin can run an old project copy or fail to startCursor, plugin-path, and VS Code tests rewrite paths and run the launcher from spaces
Version sources, release notes, or artifacts disagreeA valid-looking tag can publish incomplete or stale contentDisposable release tests and pre-release guards refuse the run
A signing job can publish unsigned output or use broad token accessA release can bypass its trust boundaryEngine workflow tests require protected signing, ordered verification, and scoped permissions
A workflow action is not pinned or CI gets write accessA build can run untrusted action code or modify repository contentWorkflow security tests require full commit pins and scoped permissions
A skill file exceeds its reviewed cap or a use loads too much contextAn agent can receive an oversized or unreviewed promptToken budget tests check every file, common use, and real CLI output cap
A provider response changes copy, omits an edit, or corrupts typed source dataLive Apply can save incomplete or unsafe source changesAgent validation requires exact coverage, source evidence, coupled-key checks, and safe partial results
A framework-owned wrapper is changed during recoveryHot reload can lose app state or leave stale live markupBrowser source contracts defer structural cleanup, fence old sessions, and use reload recovery
A live locator picks a style-only or dynamic-text match as source contentWrap or insert can edit the wrong file or hide an agent fallbackOracle cases require source candidates, text disambiguation, and explicit ambiguity errors
A component-diff region uses only the page-wide shiftA moved page can make a stationary region look missingGolden reports record the better per-region offset and preserve the region score
Pull-request history is evaluated before old comments or label events are hydratedThe sheriff can warn or close on incomplete evidenceSheriff tests exercise full-history hydration and preserve trusted markers and handoff events
A stale marker from an untrusted commenter is acceptedThe sheriff can close a pull request without its required warningSheriff tests require trusted authors for warning and close markers
A maintainer decision is treated as contributor workA pull request can age or close while waiting for a maintainerSheriff checks keep policy decisions visible while suppressing contributor aging until responsibility returns

Key files

FileWhy to read it
scripts/test-suites.mjsDefines suite purpose, membership, triggers, and opt-in policy.
scripts/run-tests.mjsShows command expansion, process-group limits, timeouts, and live-server leak checks.
scripts/ci-test-plan.mjsShows how events and changed paths select CI jobs.
.github/…/ci.ymlShows the test matrix, tool setup, and opt-in job gates.
tests/test-suites.test.mjsProtects suite registration and change-selection coverage.
tests/…/README.mdExplains recorded command behavior, fixtures, normalization, and reviewed deltas.
tests/…/lib.mjsDefines staged oracle cases and observable result comparison.
tests/…/motion.mjsShows browser-free motion command boundary cases.
tests/motion-capture.test.mjsShows the isolated animation timing contract.
tests/…/smooth.htmlShows the clean transition and reduced-motion fixture.
crates/…/motion.rsShows real capture, report, frame, and reduced-motion verification.
tests/…/critiques.mjsShows isolated signed-out preference state and multi-step CLI verification.
tests/…/health-design.mjsShows DESIGN.md sync and refresh command cases.
tests/…/setup.mjsShows local setup status and usage-error cases.
tests/…/context.mjsShows context, monorepo target, surface-brief, question, and deterministic workspace cases.
tests/…/hooks.mjsShows provider-shaped hook events, session flows, configuration, duplicate-event behavior, and Hermes coverage.
tests/…/live-wrap.mjsShows source locator, text disambiguation, wrap, insert, and generated-file behavior across project shapes.
tests/…/critiques-on-signed-out.jsonFixes persisted critique-sync state and signed-out user messaging.
tests/…/health-design-on.jsonFixes DESIGN.md sync state and guidance.
tests/…/setup-status-default.jsonFixes default setup-status output.
tests/…/hook-harness-vscode-patch-string.jsonFixes the VS Code patch-string hook response shape.
tests/…/live-wrap-text-picks-later-file.jsonFixes the successful later-file source selection and live wrapper contract.
tests/…/live-insert-text-in-no-file.jsonFixes the ambiguous dynamic-text fallback and error shape.
tests/…/comp-diff-json.jsonFixes structured component-diff output, including region offsets.
tests/…/DELTAS.mdRecords reviewed changes to command output and engine behavior.
tests/framework-fixtures.test.mjsShows the static fixture contract for live injection, CSP, and source wrapping.
tests/…/README.mdExplains runtime fixture configuration, nested app roots, and browser scenarios.
tests/…/fixture.jsonShows the Vue runtime fixture's source target, variant path, and acceptance checks.
tests/…/dev-keys.mjsDefines private local credential lookup and child-environment filtering.
tests/…/dev-keys.test.mjsProtects credential parsing, precedence, file privacy, and child isolation.
tests/…/engine-bin.mjsDefines the shared engine lookup and test environment.
crates/…/broken_image_source.rsShows source-level broken-image parsing across markup, bindings, and code literals.
crates/…/broken_image.rsShows HTML-level image source and binding rules.
crates/…/live_server_security.rsShows live-server path, event, ignore, and manual-discard security boundaries.
crates/…/hook_tests.rsShows native hook session, provider, baseline, and completion contracts.
tests/extension-build.test.mjsShows cross-browser extension startup, scan lifecycle, cancellation, recovery, and deferred checks.
tests/extension-account.test.mjsShows extension sign-in, account session recovery, journey notices, and persona access contracts.
tests/extension-agent.test.mjsShows native-agent admission, site consent, secret protection, and session cleanup.
tests/extension-walks.test.mjsShows browser walk hand-off, permissions, sign-in safety, protocol, and package variants.
tests/extension-popup.test.mjsShows active-tab popup and side-panel state, scan completion, navigation, copy, and retry behavior.
tests/extension-report.test.mjsShows the shared report format and advisory grouping used by extension copy surfaces.
tests/extension-devtools.test.mjsShows DevTools finding presentation and copy behavior.
tests/…/README.mdDescribes manual Chrome and Firefox smoke coverage for clean, flagged, CSP, and worker-restart pages.
tests/live-e2e.test.mjsShows the user-visible live flow and its source, DOM, network, and recovery proofs.
tests/live-static-accept.test.mjsShows acceptance behavior on a static page without HMR or auto-reload.
tests/live-e2e-accept-cleanup-regression.test.mjsShows the focused provider-backed durable accept and stale-wrapper guard.
tests/new-work-e2e.test.mjsShows browser decision rounds, bounded question failures, and deterministic image checks.
tests/…/agent.mjsShows the deterministic agent contract used to verify generated variants and edits.
tests/…/llm-agent.mjsShows provider selection and validation of model-produced variants and source edits.
tests/…/session.mjsShows fixture staging, server boot, browser setup, and teardown.
tests/live-browser-source.test.mjsShows source-level contracts for generation, manual edit staging, agent targets, wrapper recovery, and accepted-session cleanup.
tests/live-reference.test.mjsShows authoring contracts for live routing, polling, provider blocks, CSS guidance, action vocabulary, and manual-edit handoff.
tests/live-agent-target.test.mjsShows CLI-to-browser targeting, overlay claims, lease recovery, bounded waits, and late-result fencing.
tests/live-svelte-adapter-deepseek.test.mjsShows the full provider-backed Svelte live workflow and source cleanliness checks.
tests/hermes-plugin.test.mjsShows Hermes hook registration, trust boundaries, event translation, and Stop decisions.
tests/build.test.jsShows cross-provider skill, agent, fallback, launcher-payload, placeholder, and line-ending generation contracts.
tests/…/factory.test.jsShows provider output metadata, references, placeholders, and script marker contracts.
tests/…/activation.eval.mjsShows model-based skill activation scoring and false-activation gates.
tests/…/harness.mjsShows staged skill behavior workspaces, bounded tools, trace capture, and turn limits.
tests/…/providers.mjsShows provider selection, key lookup, model defaults, and skip behavior.
tests/skill-reference.test.mjsShows authoring contracts that keep reference flows aligned with engine behavior.
tests/skill-workflow-scope.test.mjsShows target-change and protected-content checks for workflow evidence.
tests/token-budget.test.mjsShows skill-file, use, and real CLI output budget guarantees.
scripts/…/token-budget.jsDefines file and use budget enforcement.
scripts/token-budget.jsonRecords the reviewed file, use, and CLI output caps.
tests/hook-build.test.mjsShows provider hook manifest shapes, launcher guards, platform forms, and Hermes packaging.
tests/openai-plugin.test.mjsShows OpenAI plugin staging, Codex-only content, metadata, icon, and release handoff checks.
tests/hook-build-windows.test.mjsShows Windows shell execution, spaced plugin paths, failure propagation, and missing-launcher behavior.
crates/…/hook_manifest_tests.rsShows hook merge, migration, scope, and provider-path safety contracts.
crates/…/install_picker_tests.rsShows explicit provider selection and detected-only install behavior.
crates/…/drift_ports_tests.rsShows cross-platform bundle staging, agent freshness, and install/update scope parity.
tests/launcher-download.test.mjsShows verified launcher download, cache, failure, cleanup, and retry behavior.
tests/cli-shim.test.mjsShows npm fallback lookup order, version handling, checksum-protected cache behavior, and host diagnostics.
tests/check-engine-release.test.mjsShows release-sidecar, checksum-manifest, and transient-download verification for every engine target.
tests/check-loop-release.test.mjsShows loop archive pin validation, secure redirects, digest checks, and no-overwrite behavior.
tests/github-sheriff.test.mjsShows deterministic pull-request policy, history hydration, trusted markers, and maintainer handoffs.
scripts/…/sheriff.mjsDefines pull-request blocker, label, warning, and close decisions.
tests/cursor-plugin.test.mjsShows relocated Cursor plugin staging, path rewriting, hooks, links, and version checks.
tests/plugin-paths.test.jsShows plugin path rewrite and safety checks.
tests/vscode-extension.test.mjsShows the VS Code chat-skill package boundary and relocated launcher behavior.
tests/release.test.mjsShows release refusal, source-inclusion, component, and dry-run checks in a disposable Git setup.
tests/release-engine-workflow.test.jsShows protected engine signing, same-run artifact transfer, and publication order.
tests/bundle-signing.test.mjsShows the signed bundle and trusted-key contract.
tests/workflow-security.test.jsShows action pinning and the permission boundary between CI and publishing workflows.