Evaluation and regression suite
The Python test suite checks that research remains useful, safe, repeatable, and honest from topic planning to saved output. It combines focused contract tests, isolated source runs, command flows, recorded fixtures, security checks, and quality comparisons so a local change does not hide a wider product regression.
Read this page with Person research mode, Research planning and querying, Source retrieval and connectors, Research quality and ranking, Briefings and rendered output, and Runtime configuration and safety. Those pages describe the product flows and safety rules that this suite protects.
Suite model
Test boundary
The suite runs through pytest from the repository root. Project settings discover tests below tests/, use short failure output, and measure branch coverage for the product scripts under skills/last30days/scripts. The coverage floor is 81 percent. The normal suite is the broad regression gate. Evaluation and comparison commands answer whether a change altered research quality.
References: pyproject.toml• L57-88
Test layers
The tests follow product contracts rather than one file per code module. This protects handoffs between planning, source retrieval, identity resolution, ranking, output, setup, persistence, host delivery, and distribution.
| Layer | Main question |
|---|---|
| Focused behavior | Does a rule shape, parse, rank, enrich, or format data correctly? |
| Cross-system contract | Do planning, configuration, identity, security, persistence, host, and output boundaries agree? |
| Command flow | Do representative topics, flags, sources, files, and exit states behave as one user sees them? |
| Source integration | Does an external response become useful evidence with a truthful outcome? |
| Skill and release contract | Do instructions, support paths, flags, package files, and translations stay aligned? |
| Recorded evaluation | Do fixed research cases keep their quality floors and repeatable reports? |
| Revision comparison | How did the candidate engine change overlap, retention, relevance, coverage, and errors? |
References: tests/…/test_eval_harness.py• L10-140 tests/test_person_inputs.py• L67-166 tests/…/test_skill_md_support_paths.py• L1-55 tests/…/test_no_critical_scan_findings.py• L1-128
Hermetic test state
Shared fixtures make each test behave like a new command on a clean machine. They redirect home and configuration paths, remove credential-shaped environment values, disable ambient tools, clear probe and fetch memory, turn off browser-host markers, and replace selected provider paths with safe defaults. A test that needs a provider, credential, host signal, or stored file creates it inside the test.
The network guard rejects non-owned destinations. Loopback servers remain available for real transport checks. Most source tests use local responses, recorded exchanges, mocked processes, or fixed payloads. Live source exploration is opt-in.
References: tests/conftest.py• L16-79 tests/conftest.py• L82-265
Fixture types
Fixtures stabilize the part of a flow that the test must observe. Small fixtures represent reports, plans, source items, provider replies, browser stores, credentials, and child-process results. Local servers and process stand-ins exercise transport and cleanup. Recorded source fixtures capture a full source seam so a replay can fail if a new request appears or an entity-specific input is lost.
Evaluation manifests declare the topic, date, plan, allowed sources, and recorded responses for each case. The replay model now also covers a person tied to an employer. The person-employee case resolves a profile, reads an employer page, reads first-party posts, and keeps a same-name person out of the result. This lets the suite test identity, source provenance, time windows, and rendered person context without live services.
References: tests/…/harness.py• L24-106 tests/…/harness.py• L136-199 tests/…/manifest.json• L1-47 tests/…/test_person_employee_fixture.py• L29-109
Research flow coverage
Topic planning
Planning tests check that a human topic becomes a bounded research plan. They cover intent, freshness, source choice, query breadth, explicit source selection, category peers, capability searches, and comparison entities.
An external plan cannot silently broaden retrieval. The suite checks that unavailable planned sources are reported before retrieval, mixed plans keep only usable lanes, excluded sources stay excluded, and paid lanes run only when the user asks for them. Category and capability checks protect useful breadth without letting generic words select the wrong route.
References: tests/test_planner_v3.py• L6-86 tests/test_external_plan_source_contract.py• L23-185 tests/test_categories.py• L16-73
Person research
Person-run tests protect the identity contract that turns optional employer, profile, or email input into trusted context. The suite checks that identity anchors guide retrieval and ranking, namesakes stay out, and a name-only run keeps its old behavior.
Resolution tests also check the order of evidence. A supplied or verified profile can lead to an employer bio and first-party handles, while weak or rejected matches do not create new search targets. Person posts in the requested window become evidence. Older posts remain visible only as older activity in the person profile. A failed or incomplete read stays visible as a status instead of becoming a quiet result.
References: tests/test_person_inputs.py• L67-260 tests/test_person_identity.py• L189-322 tests/test_person_resolve_step.py• L106-239 tests/test_person_posts_lane.py• L94-270
Comparison fan-out
Comparison tests protect the separation between the main topic and its peers. Each entity receives its own plan, resolution data, report, cache scope, evidence envelope, and source budget. A peer cannot overwrite the main topic or borrow another peer's context.
The suite also checks partial success. A failed peer is shown as failed while successful peers remain useful. Duplicate peers, empty plans, invalid envelopes, and budget sharing have explicit outcomes.
References: tests/competitor_cli_audit_helpers.py• L12-68 tests/test_company_resolve.py• L346-381 tests/test_external_plan_source_contract.py• L123-185
Research execution
Pipeline tests check that a plan becomes a report with evidence, clusters, source status, provider data, and source artifacts. They cover ordinary topics, comparisons, company and person research, hiring flows, zero-key paths, source exclusion, recovery, paid limits, cancellation, and bounded work.
Time and repeated work are product contracts. Shared deadlines reach source lanes and child tools. Queued work stops after cancellation or budget expiry. A useful partial result remains useful when a later lane fails. A clean no-results state stays distinct from a transport or access failure.
Finalization has the same deadline contract as retrieval. Transcript backfill and model reranking receive only the time that remains. When that time is gone, the report keeps its evidence and uses a clear fallback.
References: tests/test_pipeline_v3.py• L421-584 tests/test_deepline.py• L295-418 tests/test_person_posts_lane.py• L175-270 tests/test_finalization_deadline.py• L14-136
Follow-up research
Drill-mode tests protect the flow that starts from a saved report and researches one selected cluster more deeply. The follow-up plan keeps the contributing sources and identity context, merges new evidence without duplicating old items, and keeps local-only evidence private when output is published.
The cached report is part of the contract. Expired, replaced, or differently scoped cache data must stop before retrieval. A concurrent research or freshness write must not be lost when the follow-up saves its result.
References: tests/test_drill_mode.py• L152-331 tests/test_drill_mode.py• L498-545 tests/test_drill_mode.py• L628-785 tests/test_drill_mode.py• L929-941
Company and hiring research
Company-shaped topics receive extra checks because the product may add official sites, tickers, hiring evidence, patents, and company digests. Recorded lookup responses cover company, person-to-company, site, exchange, ticker, private-company, delisted, and ambiguous-listing outcomes. A failed lookup remains a failed lookup.
Jobs tests protect board ownership, platform detection, sitemap fallback, person-to-company routing, one shared lane budget, partial reads, role selection, digest size, story-term search, named teams, team leads, and the difference between board counts and owned-company evidence. The platform fixture set keeps several careers response shapes available for replay. Patent tests check that only eligible company runs start the patent lane and pass the report date window and budget.
References: tests/test_company_resolve.py• L1-23 tests/test_jobs_story_terms.py• L1-136 tests/test_jobs_story_search.py• L1-260 tests/test_jobs_teams.py• L1-98 tests/test_jobs_platform_boards.py• L69-109 tests/…/jobs_platforms.json• L1-233 tests/test_patents_activation.py• L150-324
Market research
Polymarket tests check that a topic keeps markets that name the subject or a valid alias in the market's own fields. Related names, such as a player's name for a team, can contribute only through explicit host hints. Unrelated events and negative controls must produce no markets.
The selection tests protect the product meaning of a market result. They check the matched outcome, live price, contender rank, event cap, close date, race floors, large multi-market events, and selection order. Pipeline tests then check that the selected market survives normalization, relevance filters, compact output, and footer output. Plan tests ensure hints validate, serialize, and reach the correct main or peer run.
References: tests/test_polymarket_entity_match.py• L1-165 tests/test_polymarket_entity_selection.py• L64-277 tests/test_polymarket_pipeline_survival.py• L82-277 tests/test_polymarket_hints_plan.py• L1-142 tests/test_polymarket_regression_set.py• L1-201
Discovery runs
Discovery tests cover the separate flow that finds rising topics from listing feeds before a user asks for a full brief. They check source plans, availability, cross-source confirmation, velocity, non-Latin topics, shallow mode, queue annotations, and honest empty states.
A blocked or changed listing is not treated as a clean empty feed. A topic must clear its confidence floor before it becomes a trend card, and a later queue record must not change the meaning of the current run.
References: tests/test_discover_mode.py• L62-115 tests/test_discover_mode.py• L365-524 tests/test_discover_render.py• L45-85
Source integration coverage
Source normalization
Source tests protect the boundary between changing external data and the common research model. They check identity, dates, URLs, text, engagement, comments, transcripts, reviews, jobs, provider fields, and source-specific artifacts before those fields reach ranking or output.
The suite includes keyless and keyed paths, paid and free lanes, command-line providers, structured APIs, social recovery, company lookup, market data, hiring boards, and person profile services. Recorded responses let the tests check provider shape changes without live services. Deepline tests also protect request shape, response parsing, retries, idempotency, budgets, and named failure states.
References: tests/test_normalize_v3.py• L88-207 tests/test_deepline.py• L181-292 tests/test_linkedin.py• L18-128 tests/test_person_linkedin.py• L123-367 tests/test_jobs_platform_boards.py• L111-195 tests/test_polymarket_entity_selection.py• L23-55
Relevance and identity floors
Ranking tests prevent popular but unrelated results from becoming evidence. GitHub must relate to the topic. Person and company paths use identity context so namesakes, unrelated market items, and related handles do not fill the report or source totals.
Capability searches have a narrow exception. They may find tools that provide the requested capability even when the tool does not name the host. Tests reserve limited space for these results and keep ordinary host evidence ahead of them.
References: tests/test_person_identity.py• L25-161 tests/…/test_person_employee_fixture.py• L49-88 tests/test_github_topical_floor.py• L65-228 tests/test_enablement_capability_results.py• L108-193 tests/test_fallback_ranking.py• L63-245
Recovery and source outcomes
Fallback tests define when recovery is success and when it is still a failure. A clean no-results response is different from an auth error, rate limit, timeout, unrelated 404, or network error. Useful items remain available when a later lane fails.
The same rule applies across social sources, jobs, discovery, person services, and other integrations. Each source exposes a typed outcome so an operator does not mistake an outage for a quiet topic. Person and Deepline tests make failed, timed-out, over-budget, and out-of-credits states visible without stopping unrelated work.
References: tests/test_source_outcomes.py• L56-126 tests/test_deepline.py• L295-418 tests/test_person_posts_lane.py• L175-270 tests/test_instagram_discovery_outcomes.py• L65-124 tests/test_reddit_degraded_run.py• L351-385 tests/test_jobs_platform_boards.py• L381-455
Evidence enrichment
Enrichment tests protect the extra context used by a report: comments, excerpts, transcripts, reviews, ratings, creator flags, and source-specific counts. Enrichment respects the run budget and does not repeat work for an item already enriched.
Quote tests separate spoken content from written text and preserve author attribution. Source-specific enrichment can add context without replacing fresher normalized fields. A failed enrichment pull does not remove the useful evidence already present.
References: tests/test_comment_excerpt.py• L25-129 tests/test_comment_own_author_flag.py• L27-151 tests/test_render_content_quotes.py• L125-273 tests/…/test_eval_harness.py• L143-200
Story confirmation
Digg confirmation tests protect the link between a high-level story and matching evidence on other sources. A confirmed item can carry story provenance and a bounded ranking boost. Similar authors, domains, subjects, and unconfirmed twins must not receive that label.
The prominence bar separates prominent, ordinary, and dropped stories. It keeps on-topic titles and rejects stories that only mention the topic in a summary. Rejected stories cannot confirm other evidence later in the run.
References: tests/…/test_eval_harness.py• L246-325 tests/test_digg.py• L1153-1384 tests/test_digg_bar.py• L53-245
Configuration and security coverage
Configuration contracts
Configuration tests protect the agreement between setup, environment loading, command options, source activation, and runtime behavior. They check registered keys, exported .env syntax, empty values, placeholder precedence, source inclusion and exclusion, backend selection, endpoint policy, supported Python versions, and preflight behavior.
The docs and loader form one contract. Documented user keys must be accepted by configuration loading unless they are intentionally process-only. Doctor and prescription checks keep source status tied to an actionable repair path without exposing a credential value. Hosted and proxy tests also ensure that partial API settings do not select a remote path by accident.
References: tests/test_env_doc_contract.py• L11-109 tests/test_hosted.py• L120-170 tests/test_proxy_routing.py• L89-130 tests/test_runtime_preflight_contract.py• L22-50 tests/test_doctor.py• L473-545 tests/test_prescriptions.py• L80-117
Credential storage
Credential tests keep environment files, platform key stores, setup helpers, and runtime readers aligned. They check source precedence, aliases, opt-out behavior, key-list parity, missing tools, and safe handling of subprocess errors. A key can be available to a run and to diagnostics without being printed or copied into a public artifact.
References: tests/test_env_keychain.py• L38-84 tests/test_env_keychain.py• L87-265 tests/test_env_keychain.py• L271-404
Setup and consent
Setup tests treat first-run work as a user flow. They keep modal, non-modal, and connector paths aligned while preserving the first-run gate. Browser consent is explicit and durable. A refusal prevents later browser reads. Credential values use the allowed storage path and never appear in setup output.
The setup contract also covers optional provider installation, device authorization, partial saves, off-PATH tools, and clear differences between missing, broken, timed-out, and already-linked states. These checks prevent setup changes from silently changing which sources a later run can use.
References: tests/test_onboarding_contract.py• L45-157 tests/test_setup_browser_persistence.py• L69-195 tests/test_setup_wizard.py• L372-720
Trust boundaries
Security tests protect local reads, browser credentials, redirects, provider endpoints, child processes, host envelopes, returned evidence, and skill-package files. Retrieved text remains untrusted data. It cannot become a new instruction, close the evidence boundary, or change a private-only output into a public briefing.
Renderer tests keep unsafe URL schemes, markup, forged footer markers, response commands, and secret values inert. Session tests keep cookies inside the request boundary. Job sitemap tests reject private addresses, unsafe schemes, and off-site redirects before fetch. Proxy tests keep sentinel credentials and unlisted endpoints from leaving the machine. Public-name checks keep private hosted-service names out of the public skill. Hermes regression tests keep support-path references as files and prevent critical scanner findings from returning to the packaged skill.
References: tests/test_evidence_output_boundary.py• L12-53 tests/test_person_inputs.py• L159-260 tests/test_proxy_routing.py• L295-318 tests/test_public_name_hygiene.py• L27-48 tests/test_doc_security_contract.py• L16-110 tests/test_reddit_session.py• L232-401 tests/test_jobs_platform_boards.py• L698-897 tests/…/test_skill_md_support_paths.py• L22-55 tests/…/test_no_critical_scan_findings.py• L92-128
Hosted and proxy routing
Hosted-mode tests treat backend choice as a public command contract. Both the key and the configured base are required before remote submission. Unsupported flags, comparisons, unsafe endpoints, clarify responses, polling errors, output profiles, and save paths fail before they can produce misleading output.
Proxy-mode tests protect a different boundary. Only configured proxy lanes receive sentinel values. User keys stay direct, free sources stay direct, paid calls share one session and charge total, and a credit stop does not disable free work. The same backend order must appear in diagnostics and runtime behavior.
References: tests/test_hosted.py• L146-329 tests/test_hosted.py• L475-713 tests/test_proxy_routing.py• L138-234 tests/test_proxy_routing.py• L237-329 tests/test_proxy_routing.py• L496-555
Watchlist delivery
Watchlist tests check the boundary between a child research command, structured report validation, persistent findings, and delivery. Completed reports can create findings. Invalid or failed child output does not. Degraded reports preserve findings and status while delivery errors remain delivery errors.
Comparison output is stored under the original topic. A bad entity or malformed report fails only that run, so later topics can still complete. Unsupported hosted backends fail before submission because watchlists require structured local reports.
References: tests/test_watchlist_output.py• L15-159 tests/test_watchlist_strict_exit.py• L15-164
Report and output coverage
Report data contract
Schema tests protect the report as the shared handoff between retrieval, ranking, persistence, freshness checks, export, and rendering. Round trips preserve topic, provider, source, candidate, warning, freshness, outcome, and meaningful zero values.
Freshness tests extract only claims with source-backed values. Rechecks label claims current, stale, contradicted, or unsupported. A failed source does not become a stale claim, and unsupported claims keep provenance without invented evidence.
References: tests/test_freshness.py• L83-157 tests/test_freshness.py• L272-329 tests/test_freshness.py• L924-967
Rendered views
Rendering tests check compact, full, context, brief, comparison, HTML, discovery, and machine-readable views. They protect source coverage, empty states, freshness notes, audience registers, person profiles, hiring blocks, market odds, comment attribution, and rejected-evidence boundaries.
The footer and evidence envelope are contracts. Source totals count the right on-topic evidence. Person profiles show identity basis, role, bio, links, LinkedIn status, and older activity without moving older posts into ranked evidence. The report checklist repeats the quiet and identity rules without becoming a second report. HTML turns safe links into links and leaves unsafe schemes as text. Versioned agent output preserves stable source states while raw JSON remains a legacy option.
References: tests/test_render_person_profile.py• L121-300 tests/test_report_checklist.py• L8-26 tests/test_html_render.py• L280-360 tests/test_agent_export.py• L166-255 tests/test_render_jobs_aggregates.py• L63-145 tests/test_polymarket_pipeline_survival.py• L150-239 tests/test_discover_render.py• L149-279
Host and skill surfaces
Skill tests connect documented commands to engine behavior. They check instruction priority, progressive disclosure, route selection, literal argument handling, comparison cleanup, usable flags, save behavior, host-plan gates, packaged references, and localized README structure.
These checks catch drift that engine tests cannot see: a missing route, a documented flag that does not work, a shell recipe that expands topic text, a support path that is a directory, or a translated command that differs from the English contract.
References: tests/skill_contract.py• L7-85 tests/test_skill_instruction_priority.py• L14-68 tests/test_skill_progressive_disclosure.py• L23-169 tests/…/test_skill_md_support_paths.py• L43-55
Evaluation and comparison
Recorded research cases
A recorded research case fixes a topic, date window, plan, safe configuration, and source responses. The normal pipeline runs against those records with live network access blocked. Fixed time and fixed inputs make a score change point to a product change rather than a changing web response.
The case set keeps different research shapes visible, including products, people, employee-linked people, comparisons, breaking events, niche topics, non-English topics, and entity market queries. Cases can require clusters or allow sparse evidence, so an aggregate result cannot hide a local failure. The person fixture also checks that the case contains no private names when the local denylist is enabled.
References: tests/…/harness.py• L38-106 tests/…/manifest.json• L1-47 tests/…/http.json• L1-254 tests/…/test_eval_harness.py• L10-31 tests/…/test_person_employee_fixture.py• L49-109 tests/test_polymarket_regression_set.py• L107-201
Quality measures
The evaluation harness turns each report into a small set of quality measures. It checks whether links came from recorded evidence, dated evidence stayed in the requested window, clusters stayed coherent, planned sources returned evidence or an explicit outcome, and repeated runs serialized the same way.
Scores are reported per case and as an aggregate. Aggregate floors protect the overall suite. Per-case floors, negative controls, and expected-cluster settings protect each research shape. A deliberately broken case proves that the aggregate cannot hide a local regression.
| Metric | Quality question |
|---|---|
| Citation grounding | Do exported links come from recorded fixture evidence? |
| Recency compliance | Does dated evidence stay in the requested window? |
| Cluster coherence | Do grouped items share enough entity meaning? |
| Coverage | Does each planned source return evidence or an explicit outcome? |
| Determinism | Do repeated runs produce the same serialized report? |
References: tests/…/harness.py• L202-327 tests/…/test_eval_harness.py• L25-140
Revision comparison
End-to-end comparison runs fixed topics through a current engine and a reference engine. It reports source and item counts, candidates, clusters, intent, subqueries, elapsed time, source errors, and command errors. A separate evaluator compares reviewer topics for overlap, retention, precision, ranking quality, and source coverage.
Recorded fixtures answer whether known quality floors still hold. Revision comparison answers how a change shifted results. Neither is a complete truth benchmark for the open web.
References: tests/e2e_comparison.py• L47-237 tests/test_evaluator_v3.py• L15-68 tests/test_verify_v3.py• L16-60
Evaluation flow
This diagram shows how controlled research inputs move through the normal product path before quality and revision checks produce a verdict.
flowchart LR
A[Fixed topic case] --> B[Recorded source inputs]
B --> C[Normal research pipeline]
C --> D[Report and exports]
D --> E[Per-case quality measures]
D --> F[Current versus reference comparison]
E --> G[Case and aggregate verdict]
F --> G
References: tests/…/harness.py• L169-199 tests/…/harness.py• L271-357 tests/e2e_comparison.py• L47-237
Change checklist
When these files or sections are changed, remember to consider these:
- When a source adapter, query rule, or source key changes, check normalization, source activation, topical floors, fallback outcomes, footer labels, and recorded cases using the source.
- When a person input, identity anchor, profile resolver, employer bio, or first-party post rule changes, check namesake filtering, derived handles, source counts, older activity, cache inputs, and public rendering.
- When a plan or comparison contract changes, check unavailable-source handling, per-entity inputs, peer limits, cache scope, envelope validation, and shared budgets.
- When a deadline, retry, enrichment, or memo rule changes, check every affected lane, finalization, child cleanup, partial results, and the difference between no-results and failure.
- When Deepline request or response handling changes, check payload shape, idempotency, retries, tool caps, named failure states, key loading, and log redaction.
- When hosted or proxy routing changes, check mode selection, endpoint safety, sentinel handling, free-source routing, charge accounting, backend order, and diagnostics.
- When jobs story terms, board readers, named teams, or digest fields change, check term derivation, per-board search, role deduplication, ownership, count floors, team labels, and footer wording.
- When Polymarket matching, hints, event selection, or prices change, check aliases, related-name limits, negative controls, race floors, plan round trips, normalized evidence, render order, and the saved regression set.
- When a report or export field changes, check schema round trips, freshness claims, persistence, all render modes, source totals, watchlist validation, and evaluation metrics that read the field.
- When configuration, credentials, or endpoint handling changes, check documented-key registration, .env syntax, setup writers, browser consent, child environments, doctor output, hosted rejection, and secret redaction.
- When skill, package, or translated documentation changes, check safe links, support paths, scanner findings, instruction priority, documented flags, and translation structure.
- When a quality floor or comparison metric changes, record the old and new case results and explain why the product change makes the old floor invalid.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| A test reaches a non-owned network destination | The offline suite depends on live services | The network guard records attempted destinations and fails the test. |
| Ambient credentials, tools, home files, or caches affect a test | Local results differ from CI results | Autouse fixtures isolate environment, paths, tools, probes, and memos. |
| An unavailable planned source is silently replaced | The report answers a different research plan | Plan tests require early failure or explicit usable-source filtering. |
| A person run accepts name-only evidence for a namesake | The report attributes another person's activity to the subject | Identity anchors, first-party gates, and the employee fixture require a matching identity basis. |
| Older first-party posts enter ranked evidence | The report breaks its requested time window | Person-lane and fixture tests keep older activity in the profile artifact only. |
| A source failure becomes an empty result | An outage looks like a quiet topic | Typed outcome tests follow failures into reports, exports, footers, and diagnostics. |
| Finalization ignores the remaining deadline | Paid or slow work runs late and evidence is lost | Deadline tests require bounded transcript and rerank work with evidence-preserving fallback. |
| A job board returns another company's roles | The hiring digest contains false company evidence | Ownership and platform tests require a confirmed board and honest fallback state. |
| Story search consumes the whole jobs budget | The normal board read is skipped or becomes too small | Story-search tests reserve time for listing reads and mark partial search. |
| A related market name is treated as the main entity | An unrelated or weak market becomes evidence | Entity tests require explicit hints and keep related matches separate. |
| A hosted key appears without a safe base or a proxy sentinel is sent directly | A command phones an unintended service or leaks a placeholder credential | Hosted and proxy tests require complete mode configuration and reject unlisted routes. |
| One case regresses while the aggregate stays above its floor | A major research shape is missed | Per-case floors and expected-cluster checks fail locally. |
| Retrieved text or metadata closes an output boundary | Source data becomes a response command or unsafe HTML | Evidence-boundary and renderer tests keep instructions inert and links safe. |
| A skill support path names a directory or a critical scanner rule returns | A host rejects the packaged skill | Hermes contract tests require file paths and zero critical findings. |
| Invalid child output is persisted as a completed report | Watchlists save false findings or deliver bad data | Strict-exit and watchlist tests validate output before persistence. |
Key files
| File | Why to read it |
|---|---|
| pyproject.toml | Defines test discovery, coverage scope, and the 81 percent product coverage floor. |
| tests/conftest.py | Shows shared isolation for home state, credentials, tools, host markers, probes, and caches. |
| tests/…/harness.py | Defines deterministic recorded cases, person resolution during replay, scoring, aggregate floors, and per-case floors. |
| tests/…/test_eval_harness.py | Shows quality floors, negative controls, replay behavior, and source confirmation checks. |
| tests/…/manifest.json | Defines the employee-linked person case, its plan, sources, and safe configuration. |
| tests/…/http.json | Provides recorded web, profile, post, and employer-page responses for that case. |
| tests/…/test_person_employee_fixture.py | Shows end-to-end person quality checks for time windows, namesakes, profile output, and privacy hygiene. |
| tests/test_pipeline_v3.py | Shows the main research flow from plan to report and bounded execution. |
| tests/test_planner_v3.py | Shows intent routing, source breadth, explicit plan handling, and capability searches. |
| tests/test_person_inputs.py | Shows how person flags become identity context without exposing email values. |
| tests/test_person_identity.py | Shows identity gates across planning, fusion, ranking, and person runs. |
| tests/test_person_resolve_step.py | Shows profile, employer-site, and derived-handle resolution before retrieval. |
| tests/test_person_posts_lane.py | Shows first-party post windows, older activity, partial reads, and source outcomes. |
| tests/test_deepline.py | Shows external profile-service request contracts, retries, budgets, failures, and privacy rules. |
| tests/test_linkedin.py | Shows normalization and date handling for LinkedIn search and profile content. |
| tests/test_drill_mode.py | Shows cached cluster follow-up, merge behavior, private evidence handling, and save boundaries. |
| tests/test_external_plan_source_contract.py | Shows how plans respect available, requested, excluded, paid, and backfilled sources. |
| tests/test_env_doc_contract.py | Shows the contract between documented configuration keys and the environment loader. |
| tests/test_env_keychain.py | Shows credential-source precedence, aliases, setup parity, and safe key-store failures. |
| tests/test_hosted.py | Shows local-versus-hosted selection, remote polling, endpoint safety, output limits, and errors. |
| tests/test_proxy_routing.py | Shows proxy lane routing, sentinel protection, paid-session accounting, and backend parity. |
| tests/test_finalization_deadline.py | Shows deadline handling for transcript backfill and reranking after retrieval. |
| tests/test_render_person_profile.py | Shows identity, role, bio, links, older activity, status, and unsafe-content rendering rules. |
| tests/test_report_checklist.py | Shows how person-profile guidance enters the short report checklist. |
| tests/test_evidence_output_boundary.py | Shows that returned evidence stays data and provenance, not instructions. |
| tests/test_public_name_hygiene.py | Guards the public package against private hosted-service names. |
| tests/test_html_render.py | Shows HTML safety, output structure, and artifact behavior. |
| tests/test_watchlist_output.py | Shows structured child-output validation and persistence for comparison runs. |
| tests/test_watchlist_strict_exit.py | Shows completed, degraded, failed, and delivery-error watchlist states. |
| tests/…/test_skill_md_support_paths.py | Guards the host package contract for support-file references. |
| tests/…/test_no_critical_scan_findings.py | Guards the packaged skill against critical Hermes scanner findings. |
| tests/e2e_comparison.py | Defines current-versus-reference engine comparison. |
| tests/test_evaluator_v3.py | Defines reviewer-topic overlap, retention, precision, and ranking comparison. |