Health diagnostics and recovery
Last30Days separates dependency readiness from evidence about what happened during a research run. Its health and diagnostic flows inspect tools, providers, source access, permissions, and recent source outcomes. They turn that information into safe status labels and practical recovery steps. This page belongs with runtime configuration and safety and complements setup, credentials, and preflight and execution boundaries and HTTP access.
Operational model
Readiness and run evidence
Readiness means that a source appears configured and able to serve before retrieval starts. Run evidence means that the source actually returned data or failed during one research run. The system keeps these facts separate so a key or binary alone does not make a source appear healthy.
The health flow moves from a configuration check to retrieval evidence and then to a repair action:
flowchart LR
A[Runtime configuration] --> B[Readiness diagnosis]
B --> C[Research run]
C --> D[Per-source outcome]
D --> E[Last-run evidence]
E --> F[Doctor or post-mortem view]
F --> G[Actionable fix]
G --> B
The health vocabulary describes dependency state. The run vocabulary describes retrieval results. A no-results outcome means the source completed without matches. States such as partial, rate-limited, auth-failed, payment-required, unreachable, timeout, schema-drift, and error mean the run does not prove that the source was quiet.
References: skills/…/health.py• L26-53 skills/…/doctor.py• L1009-1089 skills/…/doctor.py• L1217-1305
Health states and audit states
Health states give a reason for a dependency problem. Doctor rolls that reason into a small tier and an audit view that is easier to act on.
| View | States | Meaning |
|---|---|---|
| Dependency health | ok, degraded, missing, broken, timeout, error | What a source or tool can do now |
| Doctor tier | ok, warn, off, error | Whether the source is ready, limited, inactive, or failing |
| Audit state | working, unverified, not-working, could-be-on | The combined view of configuration, recent runs, and permitted probes |
off does not mean failure. It covers an optional source that is not enabled or needs user setup. error means that something configured or installed did not serve. A fresh run can refine the audit state. States from an optional source such as Meta Ad Library stay inactive until its required credential and explicit source opt-in are present. The normal doctor overlay ignores old run evidence outside its freshness window.
Doctor also keeps a source's specific status beside its tier. This preserves the difference between a missing dependency, a broken dependency, and a timeout while still giving operators a short group for action.
References: skills/…/health.py• L26-43 skills/…/doctor.py• L60-153 skills/…/doctor.py• L223-233
Diagnostic surfaces
Doctor report
The doctor command is the main operational view. It combines source readiness, provider and backend selection, external command checks, setup status, permissions, recent run evidence, and repair text in one report. Run python3 skills/last30days/scripts/last30days.py doctor; add --json for machine-readable output.
The report groups sources by audit state and shows the backend the system predicts it will use. That prediction is not proof that the backend served a request. Doctor never shows secret values. It reports credential presence as a boolean and keeps normal configuration inspection free of browser-cookie reads.
The command provides these views:
doctorruns a live report and refreshes the doctor cache. When no fresh research run exists, it also runs bounded checks for the sources that can be tested safely.doctor --cacheduses a recent matching report when possible. A stale, corrupt, or changed-configuration cache falls back to a live report.doctor --probeperforms bounded live checks for free HTTP sources and keyless command-line tools. Credit-gated and session-gated sources remain unverified.doctor --postmortemreports what the last research run actually did, even when that run is old.
Reporting a problem is a successful doctor run. The exit code stays zero so an operator can read the diagnosis and apply its printed fix.
References: skills/…/last30days.py• L2544-2553 skills/…/last30days.py• L2970-2982 skills/…/doctor.py• L1391-1503 skills/…/doctor.py• L2074-2166
Provider and source diagnosis
The diagnostic flow answers which reasoning providers and source paths are available before a run. It also records the selected reasoning mode, native host search, external command presence, browser-cookie mode, configuration source, local write targets, and permission status.
The older --diagnose flag remains a compact compatibility surface for provider and source availability. The --preflight flag uses the same safe diagnostic base to present permission findings before work starts. Doctor adds a broader source-by-source view without changing those existing JSON contracts. That view includes Meta Ad Library as an explicit opt-in path that needs the shared ScrapeCreators credential and meta_ads in INCLUDE_SOURCES.
References: skills/…/pipeline.py• L1786-1880 skills/…/last30days.py• L624-728 skills/…/last30days.py• L2952-2968 skills/…/doctor.py• L1144-1215 skills/…/doctor.py• L867-881
Dependency probes
Dependency probes test whether an external command is reachable and runnable in the process that will execute the research. This distinction catches stale command shims and other cases where a binary appears on PATH but cannot run.
Each result carries a repair prescription. A missing command points to installation or a PATH change. A present but unusable command points to reinstallation. A slow version check becomes a timeout instead of an indefinite doctor run. Probes also distinguish a binary installed in a known directory from one available to the agent process. Results are kept per process so several source checks can share one observation.
Doctor shows command health both on the related source line and in a dedicated CLI health block. Optional command-line tools remain an optional source when absent. An installed tool that is off PATH, broken, or too slow is reported as a repairable failure instead.
References: skills/…/health.py• L78-119 skills/…/health.py• L206-237 skills/…/health.py• L371-465 skills/…/doctor.py• L184-200 skills/…/doctor.py• L1391-1417
Live liveness probes
Live probes answer a narrower question than a full research run: can selected free endpoints respond, or can selected keyless tools run now? They use short per-source deadlines and isolate one slow or failed check from the rest of the report.
Doctor limits these checks to free HTTP sources and keyless command-line tools. It does not spend paid-source credits or trigger session-based authentication. A live probe can verify a public path while leaving a configured cookie or paid-key path as unverified. Transient Reddit rate limits receive one retry and remain unverified when the probe is refused twice, rather than being reported as a confirmed outage.
Probe results validate more than an HTTP status for paths with known challenge responses. Each probe uses the source's expected request identity and can check the response body. A per-source deadline prevents one slow endpoint from delaying the whole report.
References: skills/…/doctor.py• L1824-1904 skills/…/doctor.py• L1907-2029 skills/…/doctor.py• L2032-2071 skills/…/doctor.py• L2130-2150
Diagnostic cache
The doctor cache lets routine checks reuse a recent report without repeating every probe. The cache is valid only when its schema, time limit, configuration fingerprint, and report shape still match the current run.
Credential values are not part of the fingerprint or the stored report. A credential presence change still changes the fingerprint. Backend pins, source opt-ins, browser-access signals, and host selection also invalidate old conclusions when they change.
A malformed, incompatible, stale, or failed cache is treated as absent. An explicit live doctor run refreshes the cache. A cached report records that it came from cache so operators and machine readers can see its origin.
References: skills/…/doctor.py• L1627-1677 skills/…/doctor.py• L1680-1787 skills/…/doctor.py• L1790-1821
Research failure recovery
Per-source outcomes
Each research report records an observed outcome for each source that the run considered. The outcome includes the source name, state, item count, detail, time, and an optional fix hint. This record lets later diagnostics distinguish no matching items from partial evidence or a failed source path.
The outcome is preserved in the last-report cache. Doctor overlays fresh outcomes on its readiness report. The post-mortem view can inspect any saved outcome and label its age. For successful sources, it also shows item counts so an operator can distinguish a plain success from a success with a lane note.
Doctor also exposes parent-source sub-lanes for important recovery paths. These lines show whether a backup path or comment capability is armed without presenting it as a separate source.
References: skills/…/doctor.py• L1010-1089 skills/…/doctor.py• L1092-1168 skills/…/doctor.py• L1243-1276 skills/…/doctor.py• L1329-1388
Post-mortem diagnosis
Post-mortem diagnosis explains what the last research run actually did. It separates failed, partial, successful, and skipped sources, then prints any source-specific fix hint. Unlike the normal doctor overlay, it reads old evidence and labels its age instead of treating it as current readiness.
When a source reports payment-required, the post-mortem uses a credits-exhausted label. This directs the operator to restore credits rather than repeat authentication work.
If no saved run exists, post-mortem gives the next useful action: run research first or use a live doctor probe. A successful run can still show zero items, which is different from a source failure.
References: skills/…/doctor.py• L1506-1558 skills/…/doctor.py• L1561-1624 skills/…/health.py• L34-53
Remediation paths
Recovery follows the most specific known cause. Doctor keeps optional sources distinct from broken sources so an operator does not repair or enable a source that was never meant to run.
| Reported condition | Operational meaning | Usual next step |
|---|---|---|
missing | The dependency or credential is not available to this process | Install it, add it to PATH, or complete credential setup |
broken | The dependency resolves but cannot run | Reinstall or repair the current dependency |
timeout | The dependency or probe exceeded its time limit | Retry the check, then repair the slow dependency if the timeout persists |
degraded or partial | A fallback or limited path served less than expected | Inspect the printed fix and decide whether to enable the backup path |
auth-failed or unreachable | The run reached a source path but could not use it | Repair authentication or network access, then rerun the research |
payment-required | A paid path reached its credit limit | Restore credits, then rerun the research |
opt-in or unconfigured | The source is inactive or lacks required setup | Enable the source only when the workflow needs it |
The printed prescription links diagnosis to recovery. After applying it, rerun doctor to confirm readiness. Then rerun the research command to create fresh evidence.
References: skills/…/doctor.py• L229-233 skills/…/doctor.py• L348-384 skills/…/doctor.py• L692-766 skills/…/doctor.py• L1522-1558
Recovery boundaries
A diagnostic result does not promise that every source was live-tested. It reports what the system can know without spending credits, reading protected cookie values, or making an unsafe change. A source can be configured and still need a real research run before it becomes working in the audit view.
When a source fails, the rest of the report and the rest of the research result can survive. The system isolates source failures and keeps returned evidence separate from missing evidence. This prevents a failed lane from looking like a quiet topic or hiding useful results from other lanes.
References: skills/…/doctor.py• L27-39 skills/…/doctor.py• L1217-1231 skills/…/doctor.py• L2040-2059
Claim verification
Freshness verification
Freshness verification re-checks a small set of source-grounded claims after research. A claim is a structured fact such as a market probability, a GitHub star count, a StockTwits sentiment value, or a tightly shaped status statement. The verifier compares the current source response with the value used in the report.
The result is one of current, stale, contradicted, or unsupported. unsupported means that the system could not obtain a safe positive re-check. It does not mean that the original claim is false. Network access is required for live re-checks. Cached or mock runs can record that network verification was disabled.
References: skills/…/freshness.py• L15-63 skills/…/freshness.py• L96-194 skills/…/freshness.py• L419-531 skills/…/schema.py• L216-234
Verification entry points
Use --verify-freshness with a topic to verify the report after the research pass. Use it without a topic to verify a fresh cached report. The command updates the stored report with its verdicts and refuses to work when no fresh report exists. --no-verify-freshness disables a configured freshness check for a run.
The repository also provides home/…/verify_v3.py as a broader engine check. Its verification bundle covers unit and syntax checks, the diagnose path, smoke runs across reasoning providers, latency samples, and an optional quality evaluation. This validates system behavior at a wider level than claim freshness verification.
References: skills/…/last30days.py• L646-657 skills/…/last30days.py• L1316-1390 skills/…/verify_v3.py• L42-130 skills/…/verify_v3.py• L160-191
Change checklist
When these files or sections are changed, remember to consider these:
- Keep health states aligned with source outcome states and with the doctor tier and audit mappings. A renamed state can change operator text and machine-readable JSON.
- Keep doctor cache validation aligned with the report fields that the text and JSON renderers read. A changed report shape must invalidate old cached reports.
- Keep dependency probe prescriptions aligned with installers and with the process
PATH. An off-PATHbinary needs a path fix, not a second install. - Keep the last-report reader aligned with the per-source outcome contract. A mismatch can erase the evidence needed to explain a failed run.
- Keep the credits-exhausted label aligned with the
payment-requiredoutcome so operators do not repeat login work when credits are the cause. - Keep safe diagnostic paths free of cookie reads and secret values. Live probes must remain limited to their stated free or keyless scope.
- Keep optional source readiness aligned with the source's actual activation contract. For Meta Ad Library, a ScrapeCreators key alone is not enough without the
meta_adsopt-in.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
A command resolves on PATH but its version check fails | Doctor reports a false healthy dependency or gives an install fix that does nothing | probe_dependency classifies the command as broken and prescribes a reinstall |
A binary exists in a known install directory but that directory is not on the agent subprocess PATH | The engine cannot use the binary even though it is installed | The probe reports missing with an explicit PATH fix and marks it off-path |
A paid source returns payment-required | The operator may waste time re-authenticating instead of restoring credits | Health labels the outcome as credits exhausted and post-mortem prints the source-specific label |
| A fresh research run has a source error, rate limit, or unreachable path | The result may look like no discussion exists for that source | Per-source outcomes preserve the failure state and doctor/post-mortem display failed coverage |
| An old run remains in the report cache | Doctor could claim that a source works based on stale evidence | The normal overlay requires fresh evidence and post-mortem labels old evidence |
| A doctor cache is stale, corrupt, or tied to changed credentials or pins | The operator could act on an old readiness result | Schema, fingerprint, shape, and time checks treat the cache as absent and trigger a live report |
| A freshness re-check cannot obtain current evidence | A current claim could be presented as freshly confirmed | The verifier records unsupported with a reason instead of claiming confirmation |
| A source probe raises an exception | One diagnostic failure could hide all other source states | Doctor isolates the exception to that source and keeps the report available |
| Meta Ad Library has a key but lacks explicit opt-in | Doctor can report the source as available even though the research flow will not run it | The source check requires both the ScrapeCreators credential and meta_ads in INCLUDE_SOURCES |
Key files
| File | Why to read it |
|---|---|
| skills/…/doctor.py | Defines the unified report, source tiers, audit states, run evidence, cache, post-mortem view, live probes, and remediation guidance. |
| skills/…/health.py | Defines the shared dependency and source health vocabulary and command probe rules. |
| skills/…/pipeline.py | Defines the provider and source diagnosis summary used by legacy diagnostics and doctor permissions. |
| skills/…/schema.py | Defines the per-source run outcome and freshness verdict contracts carried by reports. |
| skills/…/freshness.py | Defines which grounded claims can be re-checked and how current evidence becomes a verdict. |
| skills/…/last30days.py | Defines command entry points and dispatch between doctor, diagnose, preflight, and freshness verification. |
| skills/…/verify_v3.py | Defines the wider verification bundle for unit health, provider smoke runs, latency, and quality checks. |