Introducing a new way to review your PRs.

Read the announcement

koala73 / worldmonitor

Real-time global intelligence dashboard. AI-powered news aggregation, geopolitical monitoring, and infrastructure tracking in a unified situational awareness interface

View on GitHub

29 documents · Updated Nov 20, 2026

docs / data-ingestion

Data ingestion and refresh pipelines

WorldMonitor turns external monitoring data into current product views, derived signals, and a bounded historical record. Seed runners collect and check snapshots. Workers process leased monitoring work. Relay services provide a safe server path for live or protected sources. Redis serves the live data path, while Convex stores selected durable event history. Consumer-price sources add dated market observations and country context.

Ingestion system

The ingestion system gives each source a clear path from an outside observation to a product-ready data set. A source can be a scheduled snapshot, a relay feed, a derived signal, a durable event stream, a committed dataset, or a coverage-indexed dataset.

Source shapeProduct purposeResult
Scheduled snapshotKeep monitoring views currentA validated value and freshness record
Consumer-price sourceTrack dated food prices and country contextNormalized observations, context, and source health
Relay-backed feedReach live or protected sourcesA cached or proxied response
Derived data setCombine current source viewsA product signal with its own freshness state
Durable event historyKeep events after live snapshots roll forwardSearchable, deduplicated records
Leased work streamProcess account-scoped monitoring workA durable result or replayable work item
Committed publicationServe dated crawlable viewsStatic pages and downloads
Coverage-indexed datasetSeparate unsupported scope from failed refreshA data set plus a scope manifest

Every contracted seed names its source, schema, record-count rule, time-to-live, and staleness limit. Large seed values can use a versioned compressed representation. Readers decode it before they inspect the seed envelope. See Seed pipeline orchestration for source schedules and bundle rules.

References: scripts/_seed-contract.mjs• L22-188 scripts/_seed-payload-codec.mjs• L1-22 scripts/_seed-utils.mjs• L2434-2529 scripts/_seed-envelope-source.mjs• L11-73

Refresh lifecycle

The refresh lifecycle changes an external response into a product-readable snapshot without replacing good data with a bad response. A run locks its resource, fetches within a time budget, validates the result, publishes the canonical value, and records freshness.

The normal sequence is:

  1. A scheduler starts a due seed or bundle member.
  2. The seed claims a resource lock.
  3. The fetch stage reads the source and may use a declared fallback or prior snapshot.
  4. Validation checks shape, coverage, content age, and source-specific floors.
  5. A valid result is normalized and published to Redis.
  6. Related projections, history records, and source-health state are updated.
  7. The run records its outcome and releases the lock.

This diagram shows how a source response becomes a new snapshot or a visible retained-data state.

flowchart LR
    S[Scheduler or relay loop] --> L[Resource lock]
    L --> F[Fetch external source]
    F --> V[Validate response]
    V -->|valid| N[Normalize and publish]
    N --> P[Projections and history]
    N --> H[Freshness and source health]
    V -->|partial or empty| R[Retain last good value]
    F -->|timeout or failure| R
    R --> H
    H --> U[Product and operator readers]

The shared runner bounds network and storage work. Retries consume the same phase budget as the next request. A failed run can keep the last good value for a limited period, but its freshness clock does not move. A large value can be compressed without changing the product-level snapshot contract.

References: scripts/_seed-utils.mjs• L73-117 scripts/_seed-utils.mjs• L567-775 scripts/_seed-utils.mjs• L2537-2644 scripts/_seed-payload-codec.mjs• L7-22 scripts/_bundle-runner.mjs• L406-521

Validation and data quality

Validation protects product views from responses that are technically successful but not useful. Transport success and data quality are separate states, so an HTTP success with no meaningful records does not automatically become a fresh snapshot.

The main quality checks are:

  • Record coverage: enough records exist for the product view.
  • Required coverage: required countries, regions, series, or categories are present.
  • Content freshness: the newest observation is not too old or in the future.
  • Source state: the source worked, failed, was blocked, or was not configured.
  • Contract identity: the schema and source versions match a known producer.

A source may retain a last-good value while reporting a current failure. This keeps safe data available during a short outage without hiding the outage from health tools. An empty result is accepted only when the source can prove that empty is the correct current state.

Operational datasets apply the same rule to multi-stage output. FAA NOTAM data keeps its cursor and daily bulk-load guard separate from its public airspace and closure views. Travel advisories require a broad country index rather than only a non-empty news list. Natural-event output keeps provider failures visible when retained records still serve the map. These rules stop partial coverage from looking like a current all-clear.

References: scripts/_seed-contract.mjs• L64-188 scripts/_seed-utils.mjs• L2241-2295 scripts/_seed-utils.mjs• L2884-2997 scripts/seed-natural-events.mjs• L1088-1133 scripts/seed-notam-nms.mjs• L182-244 scripts/seed-security-advisories.mjs• L401-419

Scheduled seed orchestration

Seed bundles group refresh jobs that share a scheduled service while keeping each member's due time and result separate. The bundle runner skips fresh members, runs due members in dependency order, streams output, and limits member and bundle time. Climate refreshes coordinate climate, orbital, and reservoir datasets with separate cadences, while macro refreshes give slow economic, food-price, market, and trade jobs bounded admission. This keeps one slow source from starving unrelated refreshes. See Seed pipeline orchestration for the detailed schedule view.

Derived-signal bundles run short-interval correlation, market-alert, cross-source, China-decision, and tension-evidence work before slower regional and cross-Strait work. Members with several required writes use a completion marker so the next tick retries a partial publication. Regional bundles run snapshots before dependent briefs. Standalone seeds have their own service and cadence. The deployment registry is the operational map for those entry points and schedules.

Bundle rules keep unrelated sources independent. Missing configuration fails the affected member. An impossible timeout or repeated deferral remains an operator-visible failure rather than an endless silent skip.

References: scripts/_bundle-runner.mjs• L129-197 scripts/_bundle-runner.mjs• L594-731 scripts/seed-bundle-climate.mjs• L4-14 scripts/seed-bundle-derived-signals.mjs• L1-27 scripts/seed-bundle-health.mjs• L4-16 scripts/seed-bundle-macro.mjs• L9-13 scripts/seed-bundle-macro.mjs• L121-159 scripts/railway-services.json• L1722-1769

Domain data pipelines

Domain seeds turn different external data shapes into stable product records. They use source-specific coverage rules because a complete market series, a sparse outage list, and an event feed do not have the same meaning when empty.

Monitoring event feeds

Monitoring event feeds combine source observations into bounded views for hazards, conflict, disease, displacement, aviation, and cyber activity. Each feed keeps source identity and observation time so the product can show what was observed and how current it is.

Conflict data uses a shared GDELT bulk materializer for rolling conflict, unrest, article, dyad, and per-country event views. It accepts only a recent, matching GKG and export cohort, checks each archive file, and repairs missing export days from the archive. The conflict seed reads the materialized window when it is current and uses a bounded country sweep only when the bulk path cannot provide a usable result. Per-country counts use each country's own recent norm, so media volume does not by itself look like a conflict surge. If a secondary output write fails, the materializer holds its cursor and replays the same cohort instead of advancing past an incomplete publication.

Other event feeds retain the same fail-closed approach. Climate, disease, and natural-event sources combine provider records only after shape and age checks. A source failure is kept distinct from a valid empty state. Natural-event refreshes keep EONET, GDACS, NHC, and regional warning coverage separate, retain usable provider snapshots for a limited time, and publish storm and warning projections as independent views. Wildfire, marine weather, cyber, and aviation views publish bounded records with explicit source or coverage state.

UKMTO incident data adds a maritime event view from a public republished table when the original endpoint blocks automated clients. The seed decodes the compact table, checks incident identity, time, position, and text, removes repeated reports for the same event, and activates the canonical snapshot only after a valid publication. An empty snapshot is not treated as a successful refresh.

References: scripts/seed-climate-disasters.mjs• L13-30 scripts/seed-natural-events.mjs• L961-1118 scripts/seed-conflict-intel.mjs• L467-680 scripts/seed-gdelt-bulk-materializer.mjs• L172-205 scripts/seed-gdelt-bulk-materializer.mjs• L279-326 scripts/seed-gdelt-bulk-materializer.mjs• L702-795 scripts/seed-gdelt-bulk-materializer.mjs• L828-975 scripts/seed-ukmto-incidents.mjs• L6-113

Atmospheric activity datasets

The atmospheric activity dataset turns Sentinel-5P nitrogen-dioxide observations into comparable activity signals for registered countries and industrial clusters. It stores a daily series for each region, refreshes the recent window, and fills older baseline years in bounded work so late source data can repair past days without replacing the full dataset at once.

Each region reports valid-day coverage, baseline coverage, refresh age, and a year-over-year direction. Cloud or source gaps remain explicit exclusions rather than becoming a flat activity reading. The China nowcast uses a composite of its eligible industrial clusters, and the dataset becomes active only after a successful atomic publication.

References: scripts/seed-no2-activity.mjs• L1-34 scripts/seed-no2-activity.mjs• L193-337 scripts/seed-no2-activity.mjs• L395-559 scripts/…/no2-activity.mjs• L11-18

Reservoir levels

The reservoir-level dataset adds strategic water-surface levels to climate monitoring. The daily seed starts with a committed historical baseline and merges durable per-pass state in Redis, then finds newly released NASA ICESat-2 granules for each watchlist reservoir and obtains bounded measurements through SlideRule Earth. Source observations arrive weeks after satellite acquisition and each reservoir has its own revisit rate, so observation age describes source coverage while health tracks the seed run itself.

The pipeline turns satellite segments into one level for each usable overpass. It removes weak or noisy passes, computes recent and seasonal change measures, and publishes a result only when most of the watchlist has a current level. Discovery and fetch failures remain visible, while incomplete work is retried and an unavailable upstream keeps the last good canonical value. The baseline supplies the initial historical series, so a new run can refresh the recent window without rebuilding the full record.

References: scripts/seed-reservoir-levels.mjs• L5-13 scripts/seed-reservoir-levels.mjs• L47-91 scripts/seed-reservoir-levels.mjs• L273-304 scripts/seed-reservoir-levels.mjs• L347-365 scripts/seed-reservoir-levels.mjs• L484-537 scripts/_reservoir-levels.mjs• L1-20 scripts/_reservoir-levels.mjs• L642-689 scripts/_reservoir-levels.mjs• L801-867 scripts/_reservoir-levels.mjs• L931-1041 scripts/…/reservoir-levels-baseline.json• L1

Orbital monitoring datasets

Orbital monitoring adds satellite-derived evidence to watched sites and strategic reservoirs. The catalog seed turns the active CelesTrak object list into a sensor-profiled catalog with a short daily ring of orbital elements. The site seed uses that catalog to evaluate watched locations and publish site records plus country rollups for product context.

The water-extent seed uses Sentinel-1 radar scenes to build an open-water series for reservoir sites. It keeps the series and its latest summary as separate product outputs. The watched-site pass evaluates a bounded set of sites and resumes from a cursor, then writes site records and country rollups before the index that exposes them. Archive refusal creates a backoff window and preserves the last usable publication. All three seeds share archive refusal handling and launch behind the Orbital Watch gate, so a rate-limited archive or a disabled feature does not appear as a fresh empty dataset.

This diagram shows how the orbital catalog supports site evidence and reservoir water observations.

flowchart LR
    C[CelesTrak orbital catalog] --> S[Watched-site evidence]
    A[Planetary Computer archive] --> S
    A --> W[Reservoir water-extent series]
    S --> R[Country rollups and alerts]
    W --> L[Latest water summary]
    S --> P[Product context]
    L --> P

References: scripts/seed-orbital-catalog.mjs• L39-91 scripts/seed-orbital-sites.mjs• L264-348 scripts/seed-orbital-sites.mjs• L351-391 scripts/seed-orbital-water-extent.mjs• L122-219

Displacement datasets

Displacement datasets separate cross-border movement from internal displacement because their counts have different meaning. The cross-border view combines stock totals, arrival flows, route deaths, and short trend histories without adding overlapping situations twice.

Internal displacement keeps the newest reporting round for each IOM DTM operation and keeps operation totals separate. IDMC Internal Displacement Updates add a separate event view from country CSV files. That view keeps displacement date separate from publication date, so delayed reporting can drive a reported-activity signal without pretending that the event is new. It requires a broad country set and activates only after a successful publication.

References: scripts/seed-cross-border-arrivals.mjs• L277-335 scripts/seed-dtm-displacement.mjs• L58-72 scripts/seed-idmc-idu-displacement.mjs• L7-41 scripts/seed-idmc-idu-displacement.mjs• L264-320 src/…/internal.ts• L1-110

Border movement datasets

Border movement datasets turn public arrival and encounter reports into dated route series. Each source keeps its own measure and reporting clock because daily arrivals, monthly encounters, and monthly route reports are not interchangeable.

The route seeds cover the UK English Channel, US southwest border, Frontex migratory routes, and Panama Darién reports. They accept bounded responses with valid dates and non-negative counts. Source-specific content-age limits prevent an old report from appearing current. A durable activation marker separates a dataset that has not yet published from one that has gone stale.

References: scripts/…/border-route-sources.mjs• L11-16 scripts/…/border-route-sources.mjs• L45-105 scripts/seed-border-cbp.mjs• L1-6 scripts/seed-border-frontex.mjs• L1-6 scripts/seed-border-panama.mjs• L1-6 scripts/seed-border-uk.mjs• L1-6

Grid disruption clock

The grid disruption clock turns regional outage reports and warning feeds into a current infrastructure view with a short observation history. It keeps source observation time separate from the time the seed runs, so retries do not make an old grid state look new.

Each region can be current, partial, or unavailable. A late utility report makes the region partial without making the whole dataset a failure, while missing, stale, or unreachable inputs mark the affected region as degraded. The dataset becomes active only after at least one usable live input is published.

References: scripts/seed-grid-disruption-clock.mjs• L7-15 scripts/seed-grid-disruption-clock.mjs• L17-47 scripts/seed-grid-disruption-clock.mjs• L50-77 scripts/seed-grid-disruption-clock.mjs• L80-101

Webcam metadata

The webcam catalog gives map views a current set of camera locations and display metadata. It removes duplicate camera identities and publishes a versioned geographic index beside the matching metadata set.

The active version changes only after both parts of the catalog are written. Short data-key expiry and a longer-lived active pointer limit exposure to an incomplete refresh. An empty collection leaves the active catalog unchanged.

References: scripts/seed-webcams.mjs• L28-44 scripts/seed-webcams.mjs• L61-126 scripts/seed-webcams.mjs• L152-220

Infrastructure coverage datasets

Infrastructure coverage datasets describe strategic assets that need a stable curated scope rather than an unbounded upstream catalog. The submarine-cable seed fetches route and landing-point data for selected in-service cables, normalizes countries and map geometry, and refuses to replace the product view when too few cables are usable.

The gap checker compares the curated list with the upstream catalog and writes a review list of missing in-service cables. The review output supports coverage maintenance. It is not a second live publication path.

References: scripts/seed-submarine-cables.mjs• L7-18 scripts/seed-submarine-cables.mjs• L285-355 scripts/check-submarine-cable-gaps.mjs• L1-24 scripts/check-submarine-cable-gaps.mjs• L41-67

Aviation disruption feeds

The aviation seed creates one product view from several disruption sources. It collects paid international delay aggregates, FAA delays, NOTAM closures and restrictions, and aviation news on one scheduled tick. The sources refresh independently, so a missing paid key or one provider outage does not stop the other views. Airport coverage is explicit, so an unsupported airport is not presented as normal.

The page-load aggregate combines the latest usable FAA, international, and NOTAM views. It includes normal, open, and unknown airport rows so readers can distinguish a covered airport with no disruption from an airport that the current source set did not cover. The aggregate gets its own freshness record after the aggregate is stored. Each side source keeps its own lock and freshness, so an international provider outage does not stop FAA, NOTAM, or aviation-news updates.

The paid international path has two controls. A recent successful publication can suppress another sweep, and a shared billing-cycle counter limits the total calls made by scheduled and request-time paths. A failed or rejected paid request does not become a fresh snapshot. The seed may retain the last good international value within its stale-data budget while still exposing the failure to health checks.

FAA NMS data is maintained by a separate stateful seeder. It uses frequent delta pulls and a guarded daily full load, then exposes airspace restrictions and aerodrome closures to the aviation seed. Once NMS is active, ICAO data alone is treated as partial. The seed keeps NOTAM last-good data only for its defined stale period.

The relay keeps only the request-time AviationStack proxy. Long-lived relay loops use a boot freshness guard: a restart delays a refresh when the existing attempt is still within its interval, including a recent failed attempt. This prevents frequent service restarts from turning into repeated paid or rate-limited fetches. AIS liveness is based on accepted position data rather than an open socket. Silence is reported before the connection is recycled, while repeated throttling increases the reconnect wait.

Severe disruptions and new closures become normalized notification events. Redis-held prior state and queue deduplication prevent repeated alerts. See Alerts and notification delivery for the event-to-user path.

References: scripts/seed-aviation.mjs• L61-125 scripts/seed-aviation.mjs• L1405-1550 scripts/seed-aviation.mjs• L1567-1589 scripts/seed-aviation.mjs• L1267-1355 scripts/seed-aviation.mjs• L1363-1515 scripts/seed-aviation.mjs• L1528-1789 scripts/ais-relay.cjs• L640-718 scripts/ais-relay.cjs• L1000-1111

Cyber threat feeds

The cyber seed merges several indicator feeds into one bounded threat view. It validates provider payloads, deduplicates by source and indicator identity, and keeps a stable first-seen time when an upstream feed does not provide one. Geo enrichment improves map coverage but is not required for a valid threat.

The seed publishes when at least one source is usable. An incomplete export is different from a valid empty result because it cannot prove its coverage. A rate-limited provider can use its own cache while still reporting that the provider was not called.

References: scripts/seed-cyber-threats.mjs• L8-45 scripts/seed-cyber-threats.mjs• L466-698 scripts/seed-cyber-threats.mjs• L704-840

Travel and health advisories

The advisory pipeline turns government travel registers and health feeds into a country-level advisory index plus a bounded news list. The country index uses the complete fetched registers, while the stored list is capped for product size. A report is not publishable when the register-derived country coverage falls below its floor.

This separates a failed or changed register from a valid feed that only contains informational health items. Source health is written with the report so a partial publication remains visible to operators.

References: scripts/seed-security-advisories.mjs• L27-52 scripts/seed-security-advisories.mjs• L357-399 scripts/seed-security-advisories.mjs• L401-443

Procurement opportunity datasets

The global tender pipeline turns open procurement notices from several public registers into one dated opportunity view. Each adapter normalizes its source into the same record shape, keeps only open opportunities, and reports source-level health beside the merged snapshot.

Sources run in parallel, but one failed register does not erase usable records from the other registers. The pipeline paces the low-quota SAM source by its last attempt rather than its last success. It rejects incomplete provider windows and retains the prior canonical view when the combined result is not available. Source status keys let readers distinguish a missing credential, a provider failure, a paced request, and a valid empty source.

References: scripts/seed-global-tenders.mjs• L147-208 scripts/seed-global-tenders.mjs• L211-283 scripts/seed-global-tenders.mjs• L364-397 scripts/seed-global-tenders.mjs• L422-506

Market and economic data

Market data supports current monitoring and derived pressure signals. Weekly sentiment reads the current AAII page, validates the source-dated week and its numeric balance, and merges it with retained valid weeks for a bounded history. An old or unusable reading fails the refresh, so retained history does not make the source look newly current. Market quotes use authorized providers behind one normalized quote shape, retain missing symbols only as last-good values, and seed country stock indexes independently. When all tracked exchanges are closed, the refresh keeps confirmed last-good data alive instead of treating a non-trading day as a source failure.

China macro data is a dated set of official observations with source decisions and provenance. Its status distinguishes ready, degraded, and unavailable data. Transport freshness is recorded before publication, while completed-run freshness is recorded only after the canonical snapshot and its required metadata are valid. Yield-curve refreshes group daily market curves with the slower monthly OECD long-term-rate fallback. FRED publishes a whole-series cohort and its stress index together, while PortWatch keeps daily chokepoint history separate from a year-earlier baseline.

Prediction-market refreshes combine Polymarket and Kalshi into ranked geopolitical, technology, and finance pools. The same source observations also build a country market index. That index is published only when its source coverage is complete, while the main bootstrap and the country projection remain separate product outputs.

The market-alert ledger reads fresh market quotes, prediction markets, and digest data. It rejects stale inputs as new events. Sentiment fails rather than extending freshness when no usable reading exists. National debt excludes aggregates and territories and requires a broad country set before publication.

References: scripts/seed-aaii-sentiment.mjs• L386-468 scripts/seed-aaii-sentiment.mjs• L471-502 scripts/seed-market-quotes.mjs• L22-103 scripts/seed-market-quotes.mjs• L110-159 scripts/seed-market-quotes.mjs• L161-220 scripts/seed-china-macro.mjs• L427-577 scripts/seed-fred-rates.mjs• L58-175 scripts/seed-portwatch.mjs• L229-381 scripts/seed-prediction-markets.mjs• L200-397 scripts/seed-national-debt.mjs• L7-188

Consumer-price datasets

Consumer-price datasets turn national market reports and price indexes into dated food observations plus country context. Each source has its own cadence, fetch budget, and fetch-age limits. Context-only sources add background for a country without acting as the main observation series. See Consumer-price validation for the source-specific quality rules.

The refresh keeps prior observations when a source returns no usable rows or older dates. It merges valid new observations into history, marks failed context without discarding a usable reference date, and activates the source only after a successful publication. This keeps an outage from looking like zero prices or a backward move in the series.

References: scripts/…/food-prices-seed.mjs• L12-31 scripts/…/food-prices-seed.mjs• L34-63 scripts/…/food-price-sources.json• L1-72 scripts/…/food-price-sources.json• L87-155

Trade and supply-chain datasets

Trade and supply-chain refreshes combine freight indexes, trade policy, customs revenue, trade flows, tariff trends, and US import-duty coverage into product-ready data. Separate keys and coverage manifests stop a failed scope from erasing the wider dataset.

Flow and tariff windows are seeded at the widest supported range and narrowed at read time. A flow is published only when its export and import sides are complete. Oil measures keep production, exports, imports, and AIS loadings separate. PortWatch adds dated traffic history for named chokepoints and only enables year-over-year comparison when its prior-year window has enough published days. US duty data publishes shards before its release catalog, so readers see one complete release or the prior release. UN Comtrade bilateral data publishes country product views with matching partner evidence, uses a country coverage floor, and retains a limited number of last-good country pairs when a monthly pass is partial. The extended HS4 pass has its own scope and cadence so wider product coverage does not consume the core run's full request budget.

The LNG origin dataset uses UN Comtrade HS 271111 to estimate each importer's share supplied from Qatar and the UAE. It compares a pre-crisis baseline with the latest complete monthly flows and chooses weight or value as one consistent measure for each window. If too much trade is unattributed or the baseline country count collapses, the affected share is withheld or the last good dataset is kept. Malaysia's DOSM SITC dataset adds monthly section-level exports, imports, and trade indexes to the same product data path, with content age based on the latest reported month rather than the fetch time.

References: scripts/seed-supply-chain-trade.mjs• L16-51 scripts/seed-comtrade-bilateral-hs4.mjs• L28-71 scripts/seed-comtrade-bilateral-hs4.mjs• L797-859 scripts/seed-comtrade-bilateral-hs4-ext.mjs• L5-17 scripts/seed-comtrade-bilateral-hs4-ext.mjs• L46-72 scripts/seed-comtrade-bilateral-hs4-ext.mjs• L246-290 scripts/seed-trade-sitc-my.mjs• L3-13 scripts/seed-trade-sitc-my.mjs• L27-85 scripts/seed-portwatch.mjs• L21-40 scripts/seed-oil-trade-series.mjs• L1-63 scripts/…/us-hts-chapter99.mjs• L423-568 scripts/…/comtrade-lng-origin.mjs• L1-20

Sanctions and source lists

The sanctions seed turns official lists into separate pressure, screening, change-history, and lookup projections. OFAC and SEMA supply the country pressure view. Export-control and restricted-party lists support screening and changes without being mixed into that pressure count. One failed list can be retained without falsely marking it as current.

Large XML responses are processed as a stream. Each source has a bounded retention window and reports current, retained, or unavailable state. Content age remains separate from fetch time. Attribution checks expose missing country or registry links. Change history records gaps when a baseline is unavailable rather than inventing additions or removals.

References: scripts/seed-sanctions-pressure.mjs• L23-50 scripts/seed-sanctions-pressure.mjs• L75-164 scripts/seed-sanctions-pressure.mjs• L344-365 scripts/seed-sanctions-pressure.mjs• L869-1062 scripts/_csl-sanctions.mjs• L1-35 scripts/_sema-sanctions.mjs• L12-30 scripts/_sanctions-changes.mjs• L13-79 scripts/seed-sanctions-pressure.mjs• L629-866

Derived and operational datasets

Some seeds do not collect a primary feed. They build operational data from other snapshots so product readers can use a stable result without repeating source work.

Forecast refreshes publish full and dashboard projections. Probability trails and forecast history support later resolution. Pending judgments remain pending when evidence or independent judges are unavailable.

Insights refreshes digest data into clustered and ranked stories. A shared story identity keeps edited copies of one story together while preserving distinct events. Corroboration counts distinct publisher families rather than feed labels, and unknown labels remain separate so a new source cannot claim independence by accident. Independent publisher families, grounding checks, stable story identity, and shared source policy protect the brief. The seed preserves a good public payload when synthesis degrades and records the failure state separately. The digest runner then uses the same story pool for brief composition and channel messages. It records one run result for operations, sends only when a rule is due, and keeps a failed user's last-sent marker unchanged so the rule can retry on the next tick. See Alerts and notification delivery for the event-to-user path.

Resilience refreshes warm country scores, rebuild the ranking, and publish score intervals from the same score cohort. Country history then combines current risk, resilience, and news into dated country records only when all required inputs are fresh. Demographics and education data run as independent stages, retaining only the failed stage from the previous snapshot and marking the result degraded until all stages complete.

Military CII data is an operational projection rather than a primary feed. It combines classified flight data with relay AIS candidates and disruption counts, then separates own presence from foreign presence before the risk scorer reads it. Cross-Strait activity keeps a durable archive and a bounded bootstrap projection. Tension evidence reads several of these operational inputs and stores one reading per pair, channel, and day for a bounded history. Digest notifications turn accumulated stories into scheduled user deliveries. The digest path and notification relay share stable event identity and delivery rules so a story is not sent twice across channels.

References: scripts/seed-forecasts.mjs• L57-60 scripts/seed-country-history.mjs• L25-120 scripts/seed-demographics-capability.mjs• L31-137 scripts/seed-military-cii.mjs• L355-512 scripts/seed-cross-strait-activity.mjs• L97-304 scripts/seed-tension-evidence.mjs• L1-91 scripts/seed-digest-notifications.mjs• L484-521 scripts/seed-digest-notifications.mjs• L2990-3179 scripts/…/story-identity.js• L5-43 scripts/…/story-identity.js• L461-518 scripts/…/publisher-families.js• L331-440

Relay components

Relay components provide a server-side path for live feeds, protected upstreams, and long-running refresh loops. They apply authentication, request limits, cache behavior, and controlled failure states so browser requests do not carry source credentials or depend on source network rules.

The AIS relay maintains source connections, serves bounded snapshots, and runs related polling work. Its liveness decision uses recent accepted position data rather than an open socket, so a silent upstream is visible and can be recycled. The relay reports silence before recycling the connection. Repeated rate-limit failures move to a slower reconnect ceiling, while an authentication failure uses a slow probe instead of repeated immediate retries. Its boot guard reads the freshness marker written by a seed and delays a refresh when the last attempt is still recent. OpenSky selects one effective route per request and reports route rejection separately from account throttling so operators can choose the right recovery action.

RSS relay requests use an allowlisted source domain set and guarded address lookup before connecting. This keeps a source proxy from being used to reach private network targets through a changed or rebound feed host. The notification publisher uses the same relay path to place normalized events on the shared queue, with coalescing based on a stable event family when one exists.

The notification relay consumes normalized events from Redis, reads enabled rules through protected Convex actions, and sends only to configured channels. It suppresses blocked links before delivery and uses a short fail-closed entitlement check for paid alerts. Quiet-hour events remain bounded until wake-up and are checked again against current rules before release. Per-user deduplication keeps one event from reaching a channel twice. Shipping webhooks use one event registry and one signed delivery format. Threshold events create one delivery per upward crossing. Occurrence events create one stable delivery per event identity and retry it under its limit. The API CORS preflight worker owns the browser cross-origin boundary for api.worldmonitor.app.

This diagram shows the relationship between a product request, an edge route, and a long-lived relay.

sequenceDiagram
    participant Product as Product request
    participant Edge as Edge route
    participant Relay as Long-lived relay
    participant Source as External source
    participant Cache as Redis cache

    Product->>Edge: Request data
    Edge->>Relay: Authenticated bounded request
    Relay->>Source: Fetch or read feed
    Source-->>Relay: Data or source outcome
    Relay->>Cache: Store usable data and health
    Relay-->>Edge: Payload or declared fallback
    Edge-->>Product: Response

Relay authentication normally fails closed. An unauthenticated mode must be explicit for controlled development or self-hosted use.

References: api/_relay.js• L7-123 server/…/relay.ts• L3-25 scripts/ais-relay.cjs• L741-819 scripts/ais-relay.cjs• L937-981 scripts/ais-relay.cjs• L1004-1111 scripts/…/rss-allowed-domains.json• L1-10 scripts/notification-relay.cjs• L97-110 scripts/notification-relay.cjs• L149-224 scripts/notification-relay.cjs• L378-438 scripts/notification-relay.cjs• L602-820 scripts/…/event-triggers.json• L1-27 workers/…/index.js• L1-49

Worker-controlled ingestion

The company monitoring worker handles provider work that is not a simple periodic snapshot. Convex owns the queue, tenant scope, selection, leases, checkpoints, replay, and final receipts. The worker claims one item, calls the configured provider, and finalizes the result with its lease token.

The lease is the ownership boundary. If the worker stops after provider work but before finalization, the lease expires and Convex can replay the item. A stale worker cannot finalize a replacement result because lease and revision checks fence it out. Health metadata describes these outcomes but does not control queue state.

This diagram shows the lease boundary that protects work ownership and replay.

sequenceDiagram
    participant Worker
    participant Convex
    participant Provider
    participant Health as Redis health metadata

    Worker->>Convex: Claim work
    Convex-->>Worker: Work and lease token
    Worker->>Provider: Fetch or process evidence
    Provider-->>Worker: Result or error
    Worker->>Convex: Finalize with lease
    Convex-->>Worker: Completed, replayed, fenced, or error
    Worker->>Health: Publish bounded outcome

References: scripts/company-monitoring-worker.mjs• L1-15 scripts/company-monitoring-worker.mjs• L122-196 scripts/company-monitoring-worker.mjs• L353-803

Product data availability

Product readers use canonical data keys and freshness records instead of calling every external source directly. Health checks combine data presence, freshness, coverage, source state, and selected probes so operators can tell a dead scheduler from a slow source or a partial response.

The same data can have different availability states. A current snapshot can have a failed history append. A retained snapshot can remain useful while its source is failing. These states stay separate so product safety and operational truth do not compete.

Live product views

Live views read bounded API results after page load and apply timestamp and partial-result rules. The public bootstrap and RPC policy exposes a curated set of seeded results rather than every internal cache or source feed. The widget data policy is the catalog for bootstrap keys and generated GET routes. A byte budget keeps the bootstrap usable, while dashboard bundle budgets limit the size of the product code loaded with those views. The coverage catalog gives the product and external agents a consistent view of named panels, map layers, and data tools.

Source metadata keeps source identity, publisher family, outlet profile, source tier, language, and local geography consistent across ingestion and product views. The attribution inventory discovers used hosts and checks them against reviewed provider records. Source origin identifies where a publisher or issuing body is based, while coverage geography identifies what the source reports on. Allowlisted domains protect RSS collection, while browser and server views derive their feed placement and language rules from one catalog. These metadata describe trust and coverage; they do not create another source path.

Public filter contracts keep allowed economic series, forecast domains, infrastructure baselines, chokepoints, and fuel modes aligned between product readers and external API clients. These contracts describe the supported query space; they do not create another ingestion path.

Selected collectors create normalized notification events from newly observed domain data. The relay deduplicates events before the shared queue. The notification system applies user rules and delivery policy after that point. See Alerts and notification delivery for the event-to-user path.

News source definitions use one catalog for browser and server views. The catalog keeps source identity, placement, order, language, and surface-specific URL overrides together, then produces the feed views used by each product surface. Gate exemptions keep selected feeds available in editions where the normal source gate would omit them. Strategic local sources can remain reachable across digest languages. This prevents a source or category from silently diverging between the browser and server data paths.

References: src/…/country-brief-source.ts• L11-85 api/health.js• L1889-1975 api/seed-health.js• L552-731 scripts/_widget-data-policy.cjs• L1-4 scripts/…/bundle-budgets.json• L1-11 scripts/…/bundle-budgets-pro.json• L1-11 scripts/…/openapi-filter-param-contracts.json• L1-64 scripts/build-coverage-catalog.mjs• L1-25 scripts/source-attribution.mjs• L1-56 scripts/source-origin.mjs• L1-10 scripts/source-origin.mjs• L948-1018 shared/news-catalog.ts• L1-180 scripts/…/news-catalog.json• L1-219 scripts/…/outlet-profile.json• L1-8 scripts/…/publisher-families.js• L232-347 scripts/…/source-geography.json• L1-29 scripts/…/source-tiers.json• L1-12 scripts/…/x-account-source-tiers.json• L1-32 scripts/…/rss-allowed-domains.json• L1-10

Committed dataset publication

Committed dataset publication serves dated, crawlable views from known snapshots. The corpus builder does not fetch a source at page-build time. It checks snapshot identity and age, renders static pages and downloads, and uses the source observation date for page freshness. Editorial crisis records add reviewed background, source links, timelines, actors, and watch indicators to those dated views; they are curated context rather than a second live event feed.

Crawlable crisis trackers use explicit country boundaries and coverage notes, so a dated tracker is not mistaken for a complete regional event feed. The source catalog and attribution inventory connect published data to named providers, source origin, coverage geography, and licensing review. The origin catalog uses explicit overrides or host rules and rejects an unresolved provider instead of guessing. Generated coverage, product facts, sitemap, and agent-readable documents are downstream views of these reviewed inputs. They do not create a second live ingestion path.

References: scripts/build-crawlable-corpus.mjs• L1-5 scripts/build-crawlable-corpus.mjs• L380-487 scripts/…/crawlable-crises.json• L1-19 scripts/…/crisis-editorial.json• L1-35 scripts/build-coverage-catalog.mjs• L9-25 scripts/source-attribution.mjs• L1-56 scripts/source-origin.mjs• L1-16 scripts/source-origin.mjs• L994-1018 scripts/…/product-catalog.generated.json• L1-11 scripts/…/product-facts.generated.json• L1-11

Historical event dataset

The historical event dataset preserves selected events after live Redis snapshots are overwritten. Collectors project stable event fields into a common record, publish the live snapshot first, and append history through a protected Convex relay.

History answers time-range and similarity questions that a current snapshot cannot answer. Stable producer identities support deduplication and retraction. Event time remains separate from ingest time. Retention removes records outside the history window, while tombstones stop a retracted identity from returning.

History append is fail-open for the live collector. A relay or embedding failure does not undo a successful live publish. The append path also records durable per-source ingest health, separate from canonical seed freshness, so a healthy snapshot cannot hide a history path that receives nothing.

References: scripts/_seed-history.mjs• L1-32 scripts/_seed-history.mjs• L741-889 scripts/_seed-history.mjs• L993-1054 scripts/seed-cross-strait-activity.mjs• L346-445 convex/intelHistory.ts• L11-30 convex/intelHistory.ts• L198-324

Operational freshness acceptance

The freshness monitor turns source health into a fail-closed operational verdict. It allows only bounded states with a clear reason, owner, and expiry. A source that ran and failed is not treated as harmless merely because it is on-demand.

Committed snapshots need a separate pulse check because a live health endpoint cannot prove that a scheduled file refresh and corpus merge succeeded. Post-merge deploy checks cover the gap between a green repository and data services that are live.

The Convex probe filter keeps expected missing-public-function noise out of Sentry while leaving deploy skew visible. The generated list covers internal probe names only; public function names remain visible because they can signal a real deployment mismatch. This supports operational checks around worker and history paths without treating a missing public function as a successful data refresh.

References: scripts/check-seed-freshness.mjs• L38-103 scripts/check-seed-freshness.mjs• L144-280 scripts/check-pulse-freshness.mjs• L42-167 scripts/check-postmerge-deploys.mjs• L1-35 scripts/update-seed-health-statuses.mjs• L1-31 scripts/…/sentry-convex-internal-probe-filters.txt• L1-7 scripts/…/sentry-convex-internal-probe-filters.txt• L193-207 scripts/…/sentry-convex-internal-probe-filters.txt• L318-321

Change checklist

When these files or sections are changed, remember to consider these:

  • A new canonical key or projection needs matching freshness metadata and health registration.
  • A cadence change must keep the data TTL, staleness budget, bundle interval, and boot-refresh guard aligned.
  • A changed source shape must update both normalization and the product reader that consumes it.
  • A bundle dependency must preserve order and fit inside the admission budget.
  • A source fallback must record which channel produced the result and must not mix incomparable vintages in one snapshot.
  • A GDELT materializer change must keep the current GKG/export cohort symmetric, verified, and replayable until every required output is written.
  • An AAII change must keep the source-dated week, percentage checks, retained-week merge, and freshness failure behavior aligned.
  • A sanctions source change must keep pressure sources separate from screening sources and preserve per-source retention, attribution health, and change baselines.
  • A notification relay change must keep protected rule reads, entitlement fail-closed behavior, blocked-link checks, quiet-held entries, and per-user deduplication aligned.
  • A coverage-indexed dataset must publish its scope manifest only when the advertised member data is trustworthy.
  • A contract-mode seed must keep declareRecords, source and schema versions, content-age metadata, and reader decoding behavior aligned.
  • A compressed seed payload must keep its storage prefix, decoded-size limit, and every canonical reader on the same codec.
  • An aviation change must keep paid-source throttling, the shared billing counter, FAA/NOTAM coverage, the merged bootstrap, its separate meta key, and closure notification state aligned.
  • An NMS change must preserve the delta cursor, daily full-load guard, state retention, activation marker, and the production-only closure publication boundary.
  • A relay boot-refresh change must keep recent-attempt suppression, in-process cadence, source health, and paid-provider limits aligned.
  • An AIS watchdog change must keep accepted-data freshness, silence reporting, reconnect recycling, throttle backoff, and authentication probing aligned.
  • An advisory change must keep the complete country index separate from the capped news list and preserve the minimum coverage floor.
  • A tender adapter change must preserve open-opportunity filtering, source status, attempt pacing, incomplete-window rejection, and prior-snapshot retention.
  • A UKMTO change must keep the compact table decoder, incident identity rules, event deduplication, activation marker, and source contract aligned.
  • An LNG origin change must keep the HS code, Hormuz partner scope, complete-month selection, attribution threshold, and country-count floor aligned.
  • China macro changes must keep transport freshness, content age, provenance, source decisions, and completion metadata distinct.
  • A consumer-price source change must keep source cadence, context-only status, observation dates, reference dates, content-age limits, and last-good merge behavior aligned.
  • A reservoir-level change must keep the committed baseline, Redis pass state, granule discovery window, complete-day publication, product-specific fetch budgets, watchlist coverage floor, and source-age semantics aligned.
  • A new event producer must keep its stable identity and deduplication material aligned with history and notification consumers. See Alerts and notification delivery.
  • A history append change must keep ingest-health outcomes separate from canonical seed freshness and preserve the live publish fail-open rule.
  • An IDMC or border route change must keep source-specific content age, reporting date, coverage floor, activation marker, and reader status aligned.
  • A submarine-cable scope change must update the curated source list, gap report, map configuration, minimum coverage floor, and source attribution together.
  • Forecast input changes must preserve the full-versus-dashboard projection contract, probability trail, shadow-bet intake, and resolution ledger history reach.
  • LLM-generated product text must keep its grounding, citation, fallback, and degraded-state contract across the seed and reader.
  • A retained source must keep source health and content age separate from fetch time.
  • A source host or provider change must update attribution, origin, coverage geography, outlet profile, publisher family, and public source identity.
  • A source attribution change must keep lexical host discovery, provider grouping, reviewed manifest status, and origin resolution in agreement.
  • A source catalog change must keep feed placement, language scope, gate exemptions, source tiers, local geography, and browser/server projections aligned.
  • A new or changed RSS host must update the guarded domain allowlist and the source's public attribution together.
  • A new product-readable data set must be added to the correct bootstrap or RPC allowlist, coverage catalog, and machine-readable API contract when applicable.
  • A webcam catalog change must keep the geographic index, metadata set, active pointer, and expiry window on the same version.
  • Orbital changes must keep the CelesTrak catalog, watched-site records, water-extent series, shared archive backoff, activation markers, and launch gate aligned.
  • Watched-site changes must keep the site cursor, country rollups, change-scoring state, alert state, and publish order aligned.
  • Water-extent changes must keep the reservoir watchlist coverage, accumulated series, latest summary, thresholds, and source-age semantics aligned.
  • A CORS edge-worker change must preserve the API origin policy and endpoint-specific preflight exceptions.
  • A webhook event change must keep the shared event registry, producer identity, subscriber scope, signature, retry, and delivery worker contracts aligned.
  • Notification dedup changes must keep coalescing keys, severity policy, queue publication, rollback, and bounded fail-open fallback behavior aligned across every publisher.
  • A prediction-market change must keep source completeness, category partitioning, country-index publication, and the canonical bootstrap transform aligned.
  • A news source change must keep the shared catalog, browser projection, server placement, category order, language scope, gate exemptions, and surface-specific URL overrides aligned.

Important failure modes

TriggerConsequenceGuard
A seed treats an empty or thin response as freshA product view shows a false empty or incomplete stateRecord floors, coverage checks, and last-good preservation
A retry waits beyond the seed or bundle budgetDue members age or the bundle hides their resultBackoff and next-request time count against the phase budget
A data TTL expires before the health budgetA late run looks like missing data instead of stale dataTTLs outlive staleness gates with deliberate headroom
A source fallback changes data vintage without recording itA trend can look like a real changeSingle-channel selection and provenance
A GDELT export cohort is missing or partly rebuiltConflict or country baselines look falsely calm or incompleteCursor-based repair and complete-day checks
A reservoir's late satellite granule falls outside the discovery windowA revised or delayed level never reaches the product viewLook back beyond the cursor and retain processed names longer than the lookback
An incomplete reservoir granule group advances a day as completeA pass or change metric uses partial satellite coverageGroup-level completion and retry state
A reservoir upstream fails after some old data is availableA level view is erased or is reported as freshly updatedPreservable fetch failure, last-good retention, and separate run health
An aviation source fails while the merged bootstrap is stamped freshAirport coverage looks normal for an unsupported airportPer-source coverage, aggregate meta, and explicit unknown rows
The paid aviation path ignores the shared billing cycleSeed and request-time traffic can exceed the provider allowanceOne billing-cycle counter and reservation before the sweep
NMS is not current but ICAO data is published aloneExisting closures can be cleared or re-notified from a partial viewNMS activation state and last-good retention
A relay restarts and treats a recent failed attempt as duePaid or rate-limited sources are fetched too oftenBoot guard keyed to recent attempt time
An AIS socket stays open but stops sending positionsOperators and readers treat a silent feed as liveMonotonic data-based freshness verdict and bounded recycle policy
AIS throttling is retried at the ordinary reconnect ceilingThe provider keeps rejecting the stream and the block lasts longerFailure classification and an escalated throttle ceiling
Relay authentication is absent without explicit opt-inProtected upstream routes become publicShared-secret check that fails closed
An RSS host is added without guarded allowlist coverageThe relay rejects a valid feed or can expose a private target through DNS rebindingDomain allowlist and guarded lookup
Browser and server news catalog projections driftA feed appears on one product surface but not the otherOne JSON catalog with derived surface views
A language or gate key changes in only one catalog viewA digest or panel silently loses its intended sourceShared category keys, placement rules, and gate exemptions
Publisher labels are counted as independent publishersOne newsroom's reprints pass a corroboration gateCurated publisher families and fail-closed singleton labels
A worker dies after claiming workWork appears lost or may be finalized twiceLease expiry, lease tokens, replay, and fencing
History append fails after live publicationCurrent data stays usable but historical search stops growingFail-open live publish and separate history health
A food source returns no rows or older datesThe price view is erased or moves backwardValidated merge, content-age checks, and activation state
Orbital Watch is disabled or its activation marker is not writtenSatellite-derived views remain unavailable or appear only partially activeExplicit launch gate and activation after a valid publication
The shared satellite archive refuses a requestWatched sites or reservoir water data can be refreshed too often or diverge in recoveryShared refusal backoff and retained canonical data
The orbital catalog has no sensor-profiled satellitesSite evidence cannot use current orbital elementsRequire a non-empty catalog before site publication
A bootstrap grows beyond its volume budgetInitial product loading becomes slow or incompleteByte ceilings and growth checks
Redis SET NX fails during a high-priority eventA critical event may duplicate or a hot loop may suppress itOne bounded in-process fallback reservation and telemetry
A webhook occurrence loses its stable idSubscribers receive duplicatesShared event identity, pending state, tombstones, and retry ceiling
Story identity splits edited copies of one reportCorroboration and digest ranking become inconsistentOne shared identity and similarity contract for clustering and delivery

Key files

FileWhy to read it
scripts/_seed-utils.mjsShared seed lifecycle for locks, fetch budgets, validation, publication, preservation, payload decoding, and freshness
scripts/_seed-contract.mjsContract for source identity, record counts, versions, content age, and staleness limits
scripts/_seed-payload-codec.mjsShared representation for large stored seed payloads
scripts/_bundle-runner.mjsBundle scheduling, dependencies, time budgets, and failure reporting
scripts/railway-services.jsonDeployed seed and relay entry points with operational cadence
scripts/seed-reservoir-levels.mjsReservoir watchlist refresh, baseline and Redis state merge, coverage acceptance, and canonical publication
scripts/_reservoir-levels.mjsShared reservoir observation reduction, quality checks, change metrics, and source access
scripts/…/reservoir-levels-baseline.jsonCommitted historical reservoir passes used to start and recover the rolling dataset
scripts/seed-orbital-catalog.mjsSensor-profiled satellite catalog, orbital-element history ring, source validation, and activation
scripts/seed-orbital-sites.mjsWatched-site evidence, country rollups, change scoring, archive backoff, and alert publication
scripts/seed-orbital-water-extent.mjsReservoir radar water series, latest summary, watchlist coverage, and shared archive recovery
scripts/seed-aviation.mjsIndependent aviation source paths, paid refresh controls, coverage states, merged bootstrap, and disruption notifications
scripts/seed-notam-nms.mjsFAA NOTAM state, delta and full refresh policy, airspace data, and closure publication boundary
scripts/ais-relay.cjsLong-lived relay, source polling, boot freshness guard, AIS liveness, route selection, cache writes, transport health, and event publication
scripts/…/food-prices-seed.mjsConsumer-price retention, validation, history merge, source health, and activation
scripts/…/food-price-sources.jsonConsumer-price source identity, cadence, context scope, age limits, and fetch budgets
scripts/…/news-catalog.jsonShared source definitions, surface placement, language rules, gate exemptions, and URL overrides for browser and server news views
shared/news-catalog.tsDerived browser and server catalog views and surface-specific feed rules
scripts/…/outlet-profile.jsonReviewed source geography, language, audience, and ownership metadata
scripts/…/publisher-families.jsFail-closed publisher grouping for corroboration and article-domain identity
scripts/…/source-geography.jsonCurated local-country coverage used to measure source geography
scripts/…/source-tiers.json, scripts/…/x-account-source-tiers.jsonSource priority tiers for feed and X-account selection
scripts/…/rss-allowed-domains.jsonRSS host allowlist used by the guarded relay fetch path
scripts/…/notification-dedup.cjsShared event coalescing, severity-aware failure policy, fallback deduplication, and telemetry
scripts/…/openapi-filter-param-contracts.jsonAllowed public filter values that keep API queries aligned with product data
scripts/…/bundle-budgets.json, scripts/…/bundle-budgets-pro.jsonDashboard and Pro bundle size limits that protect product data loading
scripts/seed-gdelt-bulk-materializer.mjsGDELT rolling windows, archive repair, cohort checks, cursor replay, and operational output keys
scripts/seed-aaii-sentiment.mjsSource-dated sentiment validation, retained weekly history, and freshness rules
scripts/seed-sanctions-pressure.mjsStreaming list ingestion, per-source retention, pressure and screening projections, and attribution health
scripts/seed-no2-activity.mjsSentinel-5P activity refresh, stored regional series, baseline backfill, and publication gates
scripts/seed-natural-events.mjsNatural-event provider merge, storm projections, retention, and source health
scripts/seed-security-advisories.mjsAdvisory normalization, complete country index, and source-health state
scripts/seed-global-tenders.mjsMulti-register procurement collection, source pacing, normalization, and source health
scripts/seed-supply-chain-trade.mjsShipping, trade coverage manifests, tariff trends, customs data, and US duties
scripts/seed-prediction-markets.mjsPrediction-market collection, category pools, country index, and completeness gates
scripts/seed-cross-strait-activity.mjsDurable regional activity archive, bounded bootstrap, source health, and history projection
scripts/seed-tension-evidence.mjsCross-source tension readings, daily retention, and last-good behavior
scripts/company-monitoring-worker.mjsLeased provider work and replay-safe finalization
scripts/_seed-history.mjsLive-event projection, durable history delivery, and ingest-health reporting
convex/intelHistory.tsDurable history deduplication, search, retention, and retraction
scripts/…/story-identity.jsShared story matching used to keep edited reports together across clustering and digest delivery
scripts/…/crawlable-crises.jsonExplicit scope and coverage for dated crisis trackers
scripts/…/product-catalog.generated.json, scripts/…/product-facts.generated.jsonGenerated public product facts that downstream publication exposes without creating a live source path
scripts/notification-relay.cjsProtected rule loading, entitlement checks, quiet-hour handling, notification deduplication, suppression, and channel delivery
scripts/source-attribution.mjsRuntime host discovery, provider identity grouping, and reviewed attribution checks
scripts/source-origin.mjsClosed-world publisher-origin resolution kept separate from coverage geography