Introducing a new way to review your PRs.

Read the announcement

mvanhorn / last30days-skill

AI agent skill that researches any topic across Reddit, X, YouTube, HN, Polymarket, and the web - then synthesizes a grounded summary

View on GitHub

21 documents · Updated Nov 20, 2026

docs / watchlists-library-and-store

Watchlists, library, and stored research

The persistent research system turns one-off research into a repeatable workflow. Engineers can monitor named topics, keep prior runs and findings in shared local storage, publish saved briefs as a browsable library or feed, and retrieve older material without starting new network research. See research skill execution for the wider run model, research planning and querying for fresh-query behavior, source retrieval and connectors for source collection, research quality and ranking for evidence treatment, briefings and rendered output for report presentation, and competitor and hiring research modes for related research modes.

Persistent research model

The system separates three kinds of saved material. A watchlist topic describes what to check again. A research run records one attempt and the findings it observed. A library artifact is a saved report or briefing that a person can read, publish, or search later.

Watchlist topics

Watchlists provide scheduled or on-demand monitoring for named research topics. Each topic can carry search queries and a schedule. Its enabled state controls whether an all-topics run includes it. The watchlist command surface supports adding, removing, listing, running one topic, running all enabled topics, and comparing the latest two completed runs (skills/…/watchlist.py• L90-177).

The watchlist runner uses the same research engine as a normal topic run. It records the run before retrieval and stores the resulting findings after a valid JSON response. Hard retrieval, parsing, and time-limit errors become failed runs, while a strict run with usable degraded output remains a degraded completed result. The unattended path uses a 90-day lookback and does not read browser cookies. This keeps scheduled work independent of an interactive browser session (skills/…/watchlist.py• L182-322).

References: skills/…/watchlist.py• L90-177 skills/…/watchlist.py• L182-322

Run history and findings

The shared research store links a topic to its runs and findings. A finding is identified by its source URL. A later run updates that record instead of creating a second copy. The store also keeps a sighting for each run and topic. This preserves the history needed to tell a new finding from one that continued across runs (skills/…/store.py• L89-159, skills/…/store.py• L557-737).

This relationship makes stored research useful for both monitoring and retrieval. Watchlist deltas compare URL sets from the latest two completed runs and classify findings as new, continued, or dropped. Findings can also be queried by topic, searched across the store, dismissed, or ranked by recent activity (skills/…/store.py• L740-834, skills/…/store.py• L837-917, skills/…/store.py• L1241-1259).

References: skills/…/store.py• L89-159 skills/…/store.py• L557-737 skills/…/store.py• L740-834 skills/…/store.py• L837-917 skills/…/store.py• L1241-1259

Storage scope

Unscoped work uses the shared local research database. A scoped workflow uses its own database path. This choice keeps one client or library from exposing another client's findings when later commands search or publish saved material (skills/…/store.py• L36-57).

The same scope applies to saved reports, library indexes, and library retrieval. A scoped library reads its own reports, briefings, index, and research database. It does not merge the global briefing archive or global store into that view.

References: skills/…/store.py• L36-57

Saved artifacts and library views

Saved reports form the durable document layer above the run store. Normal research can save Markdown, JSON, or HTML output. Watchlist persistence keeps findings in the store, while saved Markdown and briefing JSON become readable library entries when the library is refreshed.

Artifact scanning

The library scanner builds one common view from saved Markdown reports and briefing JSON files. It reads report titles, dates, headlines, and summaries so the feed, index, and search layer can treat both formats as research entries. It tolerates hand-edited Markdown with a normal title and skips files it cannot recognize instead of stopping the whole scan (skills/…/library.py• L99-137, skills/…/library.py• L146-235).

Each entry has a stable identity based on its topic and source file. The source update time controls refresh order. The report date controls the published date. If more than one source represents the same entry, the newer source wins (skills/…/library.py• L39-67, skills/…/library.py• L140-143). Private corpus blocks are removed from the public library content while local source content remains available to the entry (skills/…/library.py• L146-166, skills/…/library.py• L201-235).

References: skills/…/library.py• L39-67 skills/…/library.py• L99-143 skills/…/library.py• L146-235

Library indexing

The library index makes saved material searchable without network access. It incrementally tracks scanned entries and indexes their topic, headline, summary, and content. Changed entries are refreshed and missing entries are removed, so the search view follows the saved files rather than becoming a second archive.

Library search combines two local sources: saved report and briefing entries from the library index, plus completed or degraded per-run findings from the research store. It merges and de-duplicates the ranked results while retaining whether each match came from a brief or from a stored finding.

The index is optional for saving. A missing or unsupported full-text search capability does not prevent a report from being saved. An on-demand library search reports that capability as unavailable. Confirmed index corruption is rebuilt from scanned files, while a transient database error is not treated as corruption (skills/…/library_index.py• L94-126, skills/…/library_index.py• L197-212).

References: skills/…/store.py• L869-885 skills/…/store.py• L905-917 skills/…/library.py• L99-137 skills/…/library_index.py• L94-126 skills/…/library_index.py• L197-212

Library feed and index

The library feed turns scanned entries into a local reading surface. It writes an HTML index, one standalone HTML page per entry, and an Atom feed.xml. The index groups entries by topic and shows them newest first. Each page carries the saved entry's date, topic, headline, and summary.

The Atom feed gives each library and entry a stable identifier. It uses the saved report date as the publication date and the source update time as the update time. A stable library ID is stored in the output directory, so a refresh does not make subscribers see the same library as a new feed.

The local feed is a file, not a hosted subscription URL. Hosting the output directory on a static host makes feed.xml subscribable. The command can publish the HTML index and brief pages only after explicit opt-in. Hosted pages are public by default unless the operator supplies a shared password. The local Atom feed remains local and must still be hosted separately.

References: skills/…/library.py• L75-91

Retrieval and delivery

Stored material serves two different readers. An engineer can ask for a direct local search or feed refresh. A fresh research run can also use a small amount of prior material as history, while watchlists can deliver only meaningful new changes to an external channel.

Prior-material retrieval

The library search command is an offline lookup over the effective library and store. It refreshes the library index, searches saved briefs and stored findings, and prints dated matches as text. It does not publish output or call a source, model, or network service.

Fresh research can use the same index for passive historical context when LAST30DAYS_LIBRARY_CONTEXT is enabled. The lookup is limited to a few distinct prior runs that overlap the current topic or related entities. The material supports synthesis but is not evidence inside the current date range. An index problem does not make the fresh run fail.

References: skills/…/store.py• L869-885 skills/…/library.py• L99-137

Watchlist deltas

A watchlist delta is a run-to-run change view, not a general search result. It compares the stable URLs seen in the two newest completed runs for one topic. It needs two completed runs. URL-less findings have no stable identity across time and are excluded from the comparison (skills/…/store.py• L755-815).

The delta supports a compact monitoring loop: run a topic, persist its findings, inspect the new and continued groups, and treat dropped URLs as items that no longer appeared in the latest run. The counts stored on each run feed command output and later briefing summaries (skills/…/store.py• L665-674, skills/…/watchlist.py• L130-135).

References: skills/…/store.py• L755-815 skills/…/watchlist.py• L130-135

Briefing retrieval

Daily and weekly briefings turn stored findings into dated snapshots for reading. A daily briefing gathers non-dismissed findings since a chosen date and marks topics as stale when their latest run is more than 36 hours old. A weekly briefing compares this week's findings and engagement with the prior week.

Briefing data is saved in the local briefing archive and can be loaded by date. The library scanner later converts these JSON snapshots into normal library entries, which lets daily and weekly views appear beside saved topic reports. Public library metadata excludes corpus-only highlights, while local briefing content can retain a private highlight block.

References: skills/…/library.py• L201-265 skills/…/library.py• L275-299

Notification delivery

Watchlist delivery is a side effect of a completed run. It sends a notification only when a delivery channel is configured and the run has at least one new finding. Slack receives a text payload when the exact hostname is hooks.slack.com. All other HTTPS channels receive a generic JSON payload (skills/…/watchlist.py• L25-85).

Delivery does not control research success. The runner records the completed run before it attempts the webhook. A delivery exception is reported without changing the research result. Non-HTTPS channels are rejected both when configured and when a legacy value reaches delivery (skills/…/watchlist.py• L34-54, skills/…/watchlist.py• L278-294).

The all-topics command applies the daily cost budget before starting each topic. Topics that would exceed the budget or have unknown provider charges are reported as skipped. Earlier topics keep their completed or failed results (skills/…/watchlist.py• L146-177).

References: skills/…/watchlist.py• L25-85 skills/…/watchlist.py• L146-177 skills/…/watchlist.py• L278-294

Change checklist

When these files or sections are changed, remember to consider these:

  • Changes to URL identity, finding updates, or run membership must preserve the sighting ledger. Watchlist deltas and store search depend on that ledger rather than only on the aggregate finding row.
  • Changes to save-directory resolution must apply to report files, briefing files, the library index, and the research store together. A single global path in a scoped flow can leak another scope's material.
  • Changes to entry identity or output names must keep Atom IDs, HTML links, generated-brief pruning, and duplicate handling aligned.
  • Changes to library scanning must account for both Markdown reports and briefing JSON. A parser change can affect the feed, index, search index, and passive historical context at once.
  • Changes to publication filtering must preserve the private-corpus boundary. Local evidence must not enter hosted HTML, published library pages, or default agent exports.
  • Changes to delivery settings must keep the HTTPS write-time guard and the delivery-time routing guard consistent. Slack routing depends on the exact hostname rather than a substring.
  • Changes to the full-text index must retain the clear unavailable path and the distinction between rebuildable corruption and transient database errors.

Important failure modes

TriggerConsequenceGuard
A topic has fewer than two completed runs or its findings have no stable URLsThe delta cannot classify new, continued, or dropped materialReturn insufficient_history for the first case and exclude URL-less sightings from cross-run comparison (skills/…/store.py• L755-815)
A scoped command reads the global store or briefing archiveOne client's prior research can appear in another client's search, context, or published feedRoute scoped store access through the scoped database path and keep the library inputs in the same scope (skills/…/store.py• L40-57)
SQLite full-text search is unavailableLibrary indexing and search cannot provide full-text lookupKeep saving independent from indexing and expose search as unavailable rather than failing the save
A saved source is edited or deleted without an index refreshSearch results can be stale or can retain removed entriesRefresh the library before search and sync changes from the scanned files
A webhook uses a non-HTTPS URL or delivery raises an errorThe notification is skipped, but the research run still completesReject non-HTTPS settings and isolate delivery exceptions from run status (skills/…/watchlist.py• L34-54, skills/…/watchlist.py• L278-294)
A hosted library publish fails after some brief pages succeedThe publication is partial and the local feed may not contain a complete hosted URL setStop on the first publish error and report the number of public pages already created

Key files

FileWhy to read it
skills/…/watchlist.pyMain command surface for topic registration, scheduled-style runs, deltas, budgets, and webhook delivery.
skills/…/store.pyShared model for topics, runs, findings, URL identity, sightings, search, settings, and activity stats.
skills/…/library.pyDefines the common view of saved Markdown reports and briefing JSON.
skills/…/library_index.pyBuilds the offline search view and merges saved artifacts with stored run findings.
skills/…/feed.pyDefines the stable Atom feed and entry identity used by subscribers.
skills/…/html_render.pyDefines the local library index and standalone saved-brief pages.
skills/…/briefing.pyProduces dated daily and weekly snapshots from stored findings.
skills/…/pipeline.pyShows how prior saved material can enter a fresh run as bounded historical context.
skills/…/html_publish.pyDefines the optional hosted publication boundary and partial-failure behavior.