Watchlists, library, and stored research
The persistent research system turns one-off research into a repeatable workflow. Engineers can monitor named topics, keep prior runs and findings in shared local storage, publish saved briefs as a browsable library or feed, and retrieve older material without starting new network research. See research skill execution for the wider run model, research planning and querying for fresh-query behavior, source retrieval and connectors for source collection, research quality and ranking for evidence treatment, briefings and rendered output for report presentation, and competitor and hiring research modes for related research modes.
Persistent research model
The system separates three kinds of saved material. A watchlist topic describes what to check again. A research run records one attempt and the findings it observed. A library artifact is a saved report or briefing that a person can read, publish, or search later.
Watchlist topics
Watchlists provide scheduled or on-demand monitoring for named research topics. Each topic can carry search queries and a schedule. Its enabled state controls whether an all-topics run includes it. The watchlist command surface supports adding, removing, listing, running one topic, running all enabled topics, and comparing the latest two completed runs (skills/…/watchlist.py• L90-177).
The watchlist runner uses the same research engine as a normal topic run. It records the run before retrieval and stores the resulting findings after a valid JSON response. Hard retrieval, parsing, and time-limit errors become failed runs, while a strict run with usable degraded output remains a degraded completed result. The unattended path uses a 90-day lookback and does not read browser cookies. This keeps scheduled work independent of an interactive browser session (skills/…/watchlist.py• L182-322).
References: skills/…/watchlist.py• L90-177 skills/…/watchlist.py• L182-322
Run history and findings
The shared research store links a topic to its runs and findings. A finding is identified by its source URL. A later run updates that record instead of creating a second copy. The store also keeps a sighting for each run and topic. This preserves the history needed to tell a new finding from one that continued across runs (skills/…/store.py• L89-159, skills/…/store.py• L557-737).
This relationship makes stored research useful for both monitoring and retrieval. Watchlist deltas compare URL sets from the latest two completed runs and classify findings as new, continued, or dropped. Findings can also be queried by topic, searched across the store, dismissed, or ranked by recent activity (skills/…/store.py• L740-834, skills/…/store.py• L837-917, skills/…/store.py• L1241-1259).
References: skills/…/store.py• L89-159 skills/…/store.py• L557-737 skills/…/store.py• L740-834 skills/…/store.py• L837-917 skills/…/store.py• L1241-1259
Storage scope
Unscoped work uses the shared local research database. A scoped workflow uses its own database path. This choice keeps one client or library from exposing another client's findings when later commands search or publish saved material (skills/…/store.py• L36-57).
The same scope applies to saved reports, library indexes, and library retrieval. A scoped library reads its own reports, briefings, index, and research database. It does not merge the global briefing archive or global store into that view.
References: skills/…/store.py• L36-57
Saved artifacts and library views
Saved reports form the durable document layer above the run store. Normal research can save Markdown, JSON, or HTML output. Watchlist persistence keeps findings in the store, while saved Markdown and briefing JSON become readable library entries when the library is refreshed.
Artifact scanning
The library scanner builds one common view from saved Markdown reports and briefing JSON files. It reads report titles, dates, headlines, and summaries so the feed, index, and search layer can treat both formats as research entries. It tolerates hand-edited Markdown with a normal title and skips files it cannot recognize instead of stopping the whole scan (skills/…/library.py• L99-137, skills/…/library.py• L146-235).
Each entry has a stable identity based on its topic and source file. The source update time controls refresh order. The report date controls the published date. If more than one source represents the same entry, the newer source wins (skills/…/library.py• L39-67, skills/…/library.py• L140-143). Private corpus blocks are removed from the public library content while local source content remains available to the entry (skills/…/library.py• L146-166, skills/…/library.py• L201-235).
References: skills/…/library.py• L39-67 skills/…/library.py• L99-143 skills/…/library.py• L146-235
Library indexing
The library index makes saved material searchable without network access. It incrementally tracks scanned entries and indexes their topic, headline, summary, and content. Changed entries are refreshed and missing entries are removed, so the search view follows the saved files rather than becoming a second archive.
Library search combines two local sources: saved report and briefing entries from the library index, plus completed or degraded per-run findings from the research store. It merges and de-duplicates the ranked results while retaining whether each match came from a brief or from a stored finding.
The index is optional for saving. A missing or unsupported full-text search capability does not prevent a report from being saved. An on-demand library search reports that capability as unavailable. Confirmed index corruption is rebuilt from scanned files, while a transient database error is not treated as corruption (skills/…/library_index.py• L94-126, skills/…/library_index.py• L197-212).
References: skills/…/store.py• L869-885 skills/…/store.py• L905-917 skills/…/library.py• L99-137 skills/…/library_index.py• L94-126 skills/…/library_index.py• L197-212
Library feed and index
The library feed turns scanned entries into a local reading surface. It writes an HTML index, one standalone HTML page per entry, and an Atom feed.xml. The index groups entries by topic and shows them newest first. Each page carries the saved entry's date, topic, headline, and summary.
The Atom feed gives each library and entry a stable identifier. It uses the saved report date as the publication date and the source update time as the update time. A stable library ID is stored in the output directory, so a refresh does not make subscribers see the same library as a new feed.
The local feed is a file, not a hosted subscription URL. Hosting the output directory on a static host makes feed.xml subscribable. The command can publish the HTML index and brief pages only after explicit opt-in. Hosted pages are public by default unless the operator supplies a shared password. The local Atom feed remains local and must still be hosted separately.
References: skills/…/library.py• L75-91
Retrieval and delivery
Stored material serves two different readers. An engineer can ask for a direct local search or feed refresh. A fresh research run can also use a small amount of prior material as history, while watchlists can deliver only meaningful new changes to an external channel.
Prior-material retrieval
The library search command is an offline lookup over the effective library and store. It refreshes the library index, searches saved briefs and stored findings, and prints dated matches as text. It does not publish output or call a source, model, or network service.
Fresh research can use the same index for passive historical context when LAST30DAYS_LIBRARY_CONTEXT is enabled. The lookup is limited to a few distinct prior runs that overlap the current topic or related entities. The material supports synthesis but is not evidence inside the current date range. An index problem does not make the fresh run fail.
References: skills/…/store.py• L869-885 skills/…/library.py• L99-137
Watchlist deltas
A watchlist delta is a run-to-run change view, not a general search result. It compares the stable URLs seen in the two newest completed runs for one topic. It needs two completed runs. URL-less findings have no stable identity across time and are excluded from the comparison (skills/…/store.py• L755-815).
The delta supports a compact monitoring loop: run a topic, persist its findings, inspect the new and continued groups, and treat dropped URLs as items that no longer appeared in the latest run. The counts stored on each run feed command output and later briefing summaries (skills/…/store.py• L665-674, skills/…/watchlist.py• L130-135).
References: skills/…/store.py• L755-815 skills/…/watchlist.py• L130-135
Briefing retrieval
Daily and weekly briefings turn stored findings into dated snapshots for reading. A daily briefing gathers non-dismissed findings since a chosen date and marks topics as stale when their latest run is more than 36 hours old. A weekly briefing compares this week's findings and engagement with the prior week.
Briefing data is saved in the local briefing archive and can be loaded by date. The library scanner later converts these JSON snapshots into normal library entries, which lets daily and weekly views appear beside saved topic reports. Public library metadata excludes corpus-only highlights, while local briefing content can retain a private highlight block.
References: skills/…/library.py• L201-265 skills/…/library.py• L275-299
Notification delivery
Watchlist delivery is a side effect of a completed run. It sends a notification only when a delivery channel is configured and the run has at least one new finding. Slack receives a text payload when the exact hostname is hooks.slack.com. All other HTTPS channels receive a generic JSON payload (skills/…/watchlist.py• L25-85).
Delivery does not control research success. The runner records the completed run before it attempts the webhook. A delivery exception is reported without changing the research result. Non-HTTPS channels are rejected both when configured and when a legacy value reaches delivery (skills/…/watchlist.py• L34-54, skills/…/watchlist.py• L278-294).
The all-topics command applies the daily cost budget before starting each topic. Topics that would exceed the budget or have unknown provider charges are reported as skipped. Earlier topics keep their completed or failed results (skills/…/watchlist.py• L146-177).
References: skills/…/watchlist.py• L25-85 skills/…/watchlist.py• L146-177 skills/…/watchlist.py• L278-294
Change checklist
When these files or sections are changed, remember to consider these:
- Changes to URL identity, finding updates, or run membership must preserve the sighting ledger. Watchlist deltas and store search depend on that ledger rather than only on the aggregate finding row.
- Changes to save-directory resolution must apply to report files, briefing files, the library index, and the research store together. A single global path in a scoped flow can leak another scope's material.
- Changes to entry identity or output names must keep Atom IDs, HTML links, generated-brief pruning, and duplicate handling aligned.
- Changes to library scanning must account for both Markdown reports and briefing JSON. A parser change can affect the feed, index, search index, and passive historical context at once.
- Changes to publication filtering must preserve the private-corpus boundary. Local evidence must not enter hosted HTML, published library pages, or default agent exports.
- Changes to delivery settings must keep the HTTPS write-time guard and the delivery-time routing guard consistent. Slack routing depends on the exact hostname rather than a substring.
- Changes to the full-text index must retain the clear unavailable path and the distinction between rebuildable corruption and transient database errors.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| A topic has fewer than two completed runs or its findings have no stable URLs | The delta cannot classify new, continued, or dropped material | Return insufficient_history for the first case and exclude URL-less sightings from cross-run comparison (skills/…/store.py• L755-815) |
| A scoped command reads the global store or briefing archive | One client's prior research can appear in another client's search, context, or published feed | Route scoped store access through the scoped database path and keep the library inputs in the same scope (skills/…/store.py• L40-57) |
| SQLite full-text search is unavailable | Library indexing and search cannot provide full-text lookup | Keep saving independent from indexing and expose search as unavailable rather than failing the save |
| A saved source is edited or deleted without an index refresh | Search results can be stale or can retain removed entries | Refresh the library before search and sync changes from the scanned files |
| A webhook uses a non-HTTPS URL or delivery raises an error | The notification is skipped, but the research run still completes | Reject non-HTTPS settings and isolate delivery exceptions from run status (skills/…/watchlist.py• L34-54, skills/…/watchlist.py• L278-294) |
| A hosted library publish fails after some brief pages succeed | The publication is partial and the local feed may not contain a complete hosted URL set | Stop on the first publish error and report the number of public pages already created |
Key files
| File | Why to read it |
|---|---|
| skills/…/watchlist.py | Main command surface for topic registration, scheduled-style runs, deltas, budgets, and webhook delivery. |
| skills/…/store.py | Shared model for topics, runs, findings, URL identity, sightings, search, settings, and activity stats. |
| skills/…/library.py | Defines the common view of saved Markdown reports and briefing JSON. |
| skills/…/library_index.py | Builds the offline search view and merges saved artifacts with stored run findings. |
| skills/…/feed.py | Defines the stable Atom feed and entry identity used by subscribers. |
| skills/…/html_render.py | Defines the local library index and standalone saved-brief pages. |
| skills/…/briefing.py | Produces dated daily and weekly snapshots from stored findings. |
| skills/…/pipeline.py | Shows how prior saved material can enter a fresh run as bounded historical context. |
| skills/…/html_publish.py | Defines the optional hosted publication boundary and partial-failure behavior. |