Data ingestion and refresh pipelines
WorldMonitor turns external monitoring data into current product views, derived signals, and a bounded historical record. Seed runners collect and check snapshots. Workers process leased monitoring work. Relay services provide a safe server path for live or protected sources. Redis serves the live data path, while Convex stores selected durable event history. Consumer-price sources add dated market observations and country context.
Ingestion system
The ingestion system gives each source a clear path from an outside observation to a product-ready data set. A source can be a scheduled snapshot, a relay feed, a derived signal, a durable event stream, a committed dataset, or a coverage-indexed dataset.
| Source shape | Product purpose | Result |
|---|---|---|
| Scheduled snapshot | Keep monitoring views current | A validated value and freshness record |
| Consumer-price source | Track dated food prices and country context | Normalized observations, context, and source health |
| Relay-backed feed | Reach live or protected sources | A cached or proxied response |
| Derived data set | Combine current source views | A product signal with its own freshness state |
| Durable event history | Keep events after live snapshots roll forward | Searchable, deduplicated records |
| Leased work stream | Process account-scoped monitoring work | A durable result or replayable work item |
| Committed publication | Serve dated crawlable views | Static pages and downloads |
| Coverage-indexed dataset | Separate unsupported scope from failed refresh | A data set plus a scope manifest |
Every contracted seed names its source, schema, record-count rule, time-to-live, and staleness limit. Large seed values can use a versioned compressed representation. Readers decode it before they inspect the seed envelope. See Seed pipeline orchestration for source schedules and bundle rules.
References: scripts/_seed-contract.mjs• L22-188 scripts/_seed-payload-codec.mjs• L1-22 scripts/_seed-utils.mjs• L2434-2529 scripts/_seed-envelope-source.mjs• L11-73
Refresh lifecycle
The refresh lifecycle changes an external response into a product-readable snapshot without replacing good data with a bad response. A run locks its resource, fetches within a time budget, validates the result, publishes the canonical value, and records freshness.
The normal sequence is:
- A scheduler starts a due seed or bundle member.
- The seed claims a resource lock.
- The fetch stage reads the source and may use a declared fallback or prior snapshot.
- Validation checks shape, coverage, content age, and source-specific floors.
- A valid result is normalized and published to Redis.
- Related projections, history records, and source-health state are updated.
- The run records its outcome and releases the lock.
This diagram shows how a source response becomes a new snapshot or a visible retained-data state.
flowchart LR
S[Scheduler or relay loop] --> L[Resource lock]
L --> F[Fetch external source]
F --> V[Validate response]
V -->|valid| N[Normalize and publish]
N --> P[Projections and history]
N --> H[Freshness and source health]
V -->|partial or empty| R[Retain last good value]
F -->|timeout or failure| R
R --> H
H --> U[Product and operator readers]
The shared runner bounds network and storage work. Retries consume the same phase budget as the next request. A failed run can keep the last good value for a limited period, but its freshness clock does not move. A large value can be compressed without changing the product-level snapshot contract.
References: scripts/_seed-utils.mjs• L73-117 scripts/_seed-utils.mjs• L567-775 scripts/_seed-utils.mjs• L2537-2644 scripts/_seed-payload-codec.mjs• L7-22 scripts/_bundle-runner.mjs• L406-521
Validation and data quality
Validation protects product views from responses that are technically successful but not useful. Transport success and data quality are separate states, so an HTTP success with no meaningful records does not automatically become a fresh snapshot.
The main quality checks are:
- Record coverage: enough records exist for the product view.
- Required coverage: required countries, regions, series, or categories are present.
- Content freshness: the newest observation is not too old or in the future.
- Source state: the source worked, failed, was blocked, or was not configured.
- Contract identity: the schema and source versions match a known producer.
A source may retain a last-good value while reporting a current failure. This keeps safe data available during a short outage without hiding the outage from health tools. An empty result is accepted only when the source can prove that empty is the correct current state.
Operational datasets apply the same rule to multi-stage output. FAA NOTAM data keeps its cursor and daily bulk-load guard separate from its public airspace and closure views. Travel advisories require a broad country index rather than only a non-empty news list. Natural-event output keeps provider failures visible when retained records still serve the map. These rules stop partial coverage from looking like a current all-clear.
References: scripts/_seed-contract.mjs• L64-188 scripts/_seed-utils.mjs• L2241-2295 scripts/_seed-utils.mjs• L2884-2997 scripts/seed-natural-events.mjs• L1088-1133 scripts/seed-notam-nms.mjs• L182-244 scripts/seed-security-advisories.mjs• L401-419
Scheduled seed orchestration
Seed bundles group refresh jobs that share a scheduled service while keeping each member's due time and result separate. The bundle runner skips fresh members, runs due members in dependency order, streams output, and limits member and bundle time. Climate refreshes coordinate climate, orbital, and reservoir datasets with separate cadences, while macro refreshes give slow economic, food-price, market, and trade jobs bounded admission. This keeps one slow source from starving unrelated refreshes. See Seed pipeline orchestration for the detailed schedule view.
Derived-signal bundles run short-interval correlation, market-alert, cross-source, China-decision, and tension-evidence work before slower regional and cross-Strait work. Members with several required writes use a completion marker so the next tick retries a partial publication. Regional bundles run snapshots before dependent briefs. Standalone seeds have their own service and cadence. The deployment registry is the operational map for those entry points and schedules.
Bundle rules keep unrelated sources independent. Missing configuration fails the affected member. An impossible timeout or repeated deferral remains an operator-visible failure rather than an endless silent skip.
References: scripts/_bundle-runner.mjs• L129-197 scripts/_bundle-runner.mjs• L594-731 scripts/seed-bundle-climate.mjs• L4-14 scripts/seed-bundle-derived-signals.mjs• L1-27 scripts/seed-bundle-health.mjs• L4-16 scripts/seed-bundle-macro.mjs• L9-13 scripts/seed-bundle-macro.mjs• L121-159 scripts/railway-services.json• L1722-1769
Domain data pipelines
Domain seeds turn different external data shapes into stable product records. They use source-specific coverage rules because a complete market series, a sparse outage list, and an event feed do not have the same meaning when empty.
Monitoring event feeds
Monitoring event feeds combine source observations into bounded views for hazards, conflict, disease, displacement, aviation, and cyber activity. Each feed keeps source identity and observation time so the product can show what was observed and how current it is.
Conflict data uses a shared GDELT bulk materializer for rolling conflict, unrest, article, dyad, and per-country event views. It accepts only a recent, matching GKG and export cohort, checks each archive file, and repairs missing export days from the archive. The conflict seed reads the materialized window when it is current and uses a bounded country sweep only when the bulk path cannot provide a usable result. Per-country counts use each country's own recent norm, so media volume does not by itself look like a conflict surge. If a secondary output write fails, the materializer holds its cursor and replays the same cohort instead of advancing past an incomplete publication.
Other event feeds retain the same fail-closed approach. Climate, disease, and natural-event sources combine provider records only after shape and age checks. A source failure is kept distinct from a valid empty state. Natural-event refreshes keep EONET, GDACS, NHC, and regional warning coverage separate, retain usable provider snapshots for a limited time, and publish storm and warning projections as independent views. Wildfire, marine weather, cyber, and aviation views publish bounded records with explicit source or coverage state.
UKMTO incident data adds a maritime event view from a public republished table when the original endpoint blocks automated clients. The seed decodes the compact table, checks incident identity, time, position, and text, removes repeated reports for the same event, and activates the canonical snapshot only after a valid publication. An empty snapshot is not treated as a successful refresh.
References: scripts/seed-climate-disasters.mjs• L13-30 scripts/seed-natural-events.mjs• L961-1118 scripts/seed-conflict-intel.mjs• L467-680 scripts/seed-gdelt-bulk-materializer.mjs• L172-205 scripts/seed-gdelt-bulk-materializer.mjs• L279-326 scripts/seed-gdelt-bulk-materializer.mjs• L702-795 scripts/seed-gdelt-bulk-materializer.mjs• L828-975 scripts/seed-ukmto-incidents.mjs• L6-113
Atmospheric activity datasets
The atmospheric activity dataset turns Sentinel-5P nitrogen-dioxide observations into comparable activity signals for registered countries and industrial clusters. It stores a daily series for each region, refreshes the recent window, and fills older baseline years in bounded work so late source data can repair past days without replacing the full dataset at once.
Each region reports valid-day coverage, baseline coverage, refresh age, and a year-over-year direction. Cloud or source gaps remain explicit exclusions rather than becoming a flat activity reading. The China nowcast uses a composite of its eligible industrial clusters, and the dataset becomes active only after a successful atomic publication.
References: scripts/seed-no2-activity.mjs• L1-34 scripts/seed-no2-activity.mjs• L193-337 scripts/seed-no2-activity.mjs• L395-559 scripts/…/no2-activity.mjs• L11-18
Reservoir levels
The reservoir-level dataset adds strategic water-surface levels to climate monitoring. The daily seed starts with a committed historical baseline and merges durable per-pass state in Redis, then finds newly released NASA ICESat-2 granules for each watchlist reservoir and obtains bounded measurements through SlideRule Earth. Source observations arrive weeks after satellite acquisition and each reservoir has its own revisit rate, so observation age describes source coverage while health tracks the seed run itself.
The pipeline turns satellite segments into one level for each usable overpass. It removes weak or noisy passes, computes recent and seasonal change measures, and publishes a result only when most of the watchlist has a current level. Discovery and fetch failures remain visible, while incomplete work is retried and an unavailable upstream keeps the last good canonical value. The baseline supplies the initial historical series, so a new run can refresh the recent window without rebuilding the full record.
References: scripts/seed-reservoir-levels.mjs• L5-13 scripts/seed-reservoir-levels.mjs• L47-91 scripts/seed-reservoir-levels.mjs• L273-304 scripts/seed-reservoir-levels.mjs• L347-365 scripts/seed-reservoir-levels.mjs• L484-537 scripts/_reservoir-levels.mjs• L1-20 scripts/_reservoir-levels.mjs• L642-689 scripts/_reservoir-levels.mjs• L801-867 scripts/_reservoir-levels.mjs• L931-1041 scripts/…/reservoir-levels-baseline.json• L1
Orbital monitoring datasets
Orbital monitoring adds satellite-derived evidence to watched sites and strategic reservoirs. The catalog seed turns the active CelesTrak object list into a sensor-profiled catalog with a short daily ring of orbital elements. The site seed uses that catalog to evaluate watched locations and publish site records plus country rollups for product context.
The water-extent seed uses Sentinel-1 radar scenes to build an open-water series for reservoir sites. It keeps the series and its latest summary as separate product outputs. The watched-site pass evaluates a bounded set of sites and resumes from a cursor, then writes site records and country rollups before the index that exposes them. Archive refusal creates a backoff window and preserves the last usable publication. All three seeds share archive refusal handling and launch behind the Orbital Watch gate, so a rate-limited archive or a disabled feature does not appear as a fresh empty dataset.
This diagram shows how the orbital catalog supports site evidence and reservoir water observations.
flowchart LR
C[CelesTrak orbital catalog] --> S[Watched-site evidence]
A[Planetary Computer archive] --> S
A --> W[Reservoir water-extent series]
S --> R[Country rollups and alerts]
W --> L[Latest water summary]
S --> P[Product context]
L --> P
References: scripts/seed-orbital-catalog.mjs• L39-91 scripts/seed-orbital-sites.mjs• L264-348 scripts/seed-orbital-sites.mjs• L351-391 scripts/seed-orbital-water-extent.mjs• L122-219
Displacement datasets
Displacement datasets separate cross-border movement from internal displacement because their counts have different meaning. The cross-border view combines stock totals, arrival flows, route deaths, and short trend histories without adding overlapping situations twice.
Internal displacement keeps the newest reporting round for each IOM DTM operation and keeps operation totals separate. IDMC Internal Displacement Updates add a separate event view from country CSV files. That view keeps displacement date separate from publication date, so delayed reporting can drive a reported-activity signal without pretending that the event is new. It requires a broad country set and activates only after a successful publication.
References: scripts/seed-cross-border-arrivals.mjs• L277-335 scripts/seed-dtm-displacement.mjs• L58-72 scripts/seed-idmc-idu-displacement.mjs• L7-41 scripts/seed-idmc-idu-displacement.mjs• L264-320 src/…/internal.ts• L1-110
Border movement datasets
Border movement datasets turn public arrival and encounter reports into dated route series. Each source keeps its own measure and reporting clock because daily arrivals, monthly encounters, and monthly route reports are not interchangeable.
The route seeds cover the UK English Channel, US southwest border, Frontex migratory routes, and Panama Darién reports. They accept bounded responses with valid dates and non-negative counts. Source-specific content-age limits prevent an old report from appearing current. A durable activation marker separates a dataset that has not yet published from one that has gone stale.
References: scripts/…/border-route-sources.mjs• L11-16 scripts/…/border-route-sources.mjs• L45-105 scripts/seed-border-cbp.mjs• L1-6 scripts/seed-border-frontex.mjs• L1-6 scripts/seed-border-panama.mjs• L1-6 scripts/seed-border-uk.mjs• L1-6
Grid disruption clock
The grid disruption clock turns regional outage reports and warning feeds into a current infrastructure view with a short observation history. It keeps source observation time separate from the time the seed runs, so retries do not make an old grid state look new.
Each region can be current, partial, or unavailable. A late utility report makes the region partial without making the whole dataset a failure, while missing, stale, or unreachable inputs mark the affected region as degraded. The dataset becomes active only after at least one usable live input is published.
References: scripts/seed-grid-disruption-clock.mjs• L7-15 scripts/seed-grid-disruption-clock.mjs• L17-47 scripts/seed-grid-disruption-clock.mjs• L50-77 scripts/seed-grid-disruption-clock.mjs• L80-101
Webcam metadata
The webcam catalog gives map views a current set of camera locations and display metadata. It removes duplicate camera identities and publishes a versioned geographic index beside the matching metadata set.
The active version changes only after both parts of the catalog are written. Short data-key expiry and a longer-lived active pointer limit exposure to an incomplete refresh. An empty collection leaves the active catalog unchanged.
References: scripts/seed-webcams.mjs• L28-44 scripts/seed-webcams.mjs• L61-126 scripts/seed-webcams.mjs• L152-220
Infrastructure coverage datasets
Infrastructure coverage datasets describe strategic assets that need a stable curated scope rather than an unbounded upstream catalog. The submarine-cable seed fetches route and landing-point data for selected in-service cables, normalizes countries and map geometry, and refuses to replace the product view when too few cables are usable.
The gap checker compares the curated list with the upstream catalog and writes a review list of missing in-service cables. The review output supports coverage maintenance. It is not a second live publication path.
References: scripts/seed-submarine-cables.mjs• L7-18 scripts/seed-submarine-cables.mjs• L285-355 scripts/check-submarine-cable-gaps.mjs• L1-24 scripts/check-submarine-cable-gaps.mjs• L41-67
Aviation disruption feeds
The aviation seed creates one product view from several disruption sources. It collects paid international delay aggregates, FAA delays, NOTAM closures and restrictions, and aviation news on one scheduled tick. The sources refresh independently, so a missing paid key or one provider outage does not stop the other views. Airport coverage is explicit, so an unsupported airport is not presented as normal.
The page-load aggregate combines the latest usable FAA, international, and NOTAM views. It includes normal, open, and unknown airport rows so readers can distinguish a covered airport with no disruption from an airport that the current source set did not cover. The aggregate gets its own freshness record after the aggregate is stored. Each side source keeps its own lock and freshness, so an international provider outage does not stop FAA, NOTAM, or aviation-news updates.
The paid international path has two controls. A recent successful publication can suppress another sweep, and a shared billing-cycle counter limits the total calls made by scheduled and request-time paths. A failed or rejected paid request does not become a fresh snapshot. The seed may retain the last good international value within its stale-data budget while still exposing the failure to health checks.
FAA NMS data is maintained by a separate stateful seeder. It uses frequent delta pulls and a guarded daily full load, then exposes airspace restrictions and aerodrome closures to the aviation seed. Once NMS is active, ICAO data alone is treated as partial. The seed keeps NOTAM last-good data only for its defined stale period.
The relay keeps only the request-time AviationStack proxy. Long-lived relay loops use a boot freshness guard: a restart delays a refresh when the existing attempt is still within its interval, including a recent failed attempt. This prevents frequent service restarts from turning into repeated paid or rate-limited fetches. AIS liveness is based on accepted position data rather than an open socket. Silence is reported before the connection is recycled, while repeated throttling increases the reconnect wait.
Severe disruptions and new closures become normalized notification events. Redis-held prior state and queue deduplication prevent repeated alerts. See Alerts and notification delivery for the event-to-user path.
References: scripts/seed-aviation.mjs• L61-125 scripts/seed-aviation.mjs• L1405-1550 scripts/seed-aviation.mjs• L1567-1589 scripts/seed-aviation.mjs• L1267-1355 scripts/seed-aviation.mjs• L1363-1515 scripts/seed-aviation.mjs• L1528-1789 scripts/ais-relay.cjs• L640-718 scripts/ais-relay.cjs• L1000-1111
Cyber threat feeds
The cyber seed merges several indicator feeds into one bounded threat view. It validates provider payloads, deduplicates by source and indicator identity, and keeps a stable first-seen time when an upstream feed does not provide one. Geo enrichment improves map coverage but is not required for a valid threat.
The seed publishes when at least one source is usable. An incomplete export is different from a valid empty result because it cannot prove its coverage. A rate-limited provider can use its own cache while still reporting that the provider was not called.
References: scripts/seed-cyber-threats.mjs• L8-45 scripts/seed-cyber-threats.mjs• L466-698 scripts/seed-cyber-threats.mjs• L704-840
Travel and health advisories
The advisory pipeline turns government travel registers and health feeds into a country-level advisory index plus a bounded news list. The country index uses the complete fetched registers, while the stored list is capped for product size. A report is not publishable when the register-derived country coverage falls below its floor.
This separates a failed or changed register from a valid feed that only contains informational health items. Source health is written with the report so a partial publication remains visible to operators.
References: scripts/seed-security-advisories.mjs• L27-52 scripts/seed-security-advisories.mjs• L357-399 scripts/seed-security-advisories.mjs• L401-443
Procurement opportunity datasets
The global tender pipeline turns open procurement notices from several public registers into one dated opportunity view. Each adapter normalizes its source into the same record shape, keeps only open opportunities, and reports source-level health beside the merged snapshot.
Sources run in parallel, but one failed register does not erase usable records from the other registers. The pipeline paces the low-quota SAM source by its last attempt rather than its last success. It rejects incomplete provider windows and retains the prior canonical view when the combined result is not available. Source status keys let readers distinguish a missing credential, a provider failure, a paced request, and a valid empty source.
References: scripts/seed-global-tenders.mjs• L147-208 scripts/seed-global-tenders.mjs• L211-283 scripts/seed-global-tenders.mjs• L364-397 scripts/seed-global-tenders.mjs• L422-506
Market and economic data
Market data supports current monitoring and derived pressure signals. Weekly sentiment reads the current AAII page, validates the source-dated week and its numeric balance, and merges it with retained valid weeks for a bounded history. An old or unusable reading fails the refresh, so retained history does not make the source look newly current. Market quotes use authorized providers behind one normalized quote shape, retain missing symbols only as last-good values, and seed country stock indexes independently. When all tracked exchanges are closed, the refresh keeps confirmed last-good data alive instead of treating a non-trading day as a source failure.
China macro data is a dated set of official observations with source decisions and provenance. Its status distinguishes ready, degraded, and unavailable data. Transport freshness is recorded before publication, while completed-run freshness is recorded only after the canonical snapshot and its required metadata are valid. Yield-curve refreshes group daily market curves with the slower monthly OECD long-term-rate fallback. FRED publishes a whole-series cohort and its stress index together, while PortWatch keeps daily chokepoint history separate from a year-earlier baseline.
Prediction-market refreshes combine Polymarket and Kalshi into ranked geopolitical, technology, and finance pools. The same source observations also build a country market index. That index is published only when its source coverage is complete, while the main bootstrap and the country projection remain separate product outputs.
The market-alert ledger reads fresh market quotes, prediction markets, and digest data. It rejects stale inputs as new events. Sentiment fails rather than extending freshness when no usable reading exists. National debt excludes aggregates and territories and requires a broad country set before publication.
References: scripts/seed-aaii-sentiment.mjs• L386-468 scripts/seed-aaii-sentiment.mjs• L471-502 scripts/seed-market-quotes.mjs• L22-103 scripts/seed-market-quotes.mjs• L110-159 scripts/seed-market-quotes.mjs• L161-220 scripts/seed-china-macro.mjs• L427-577 scripts/seed-fred-rates.mjs• L58-175 scripts/seed-portwatch.mjs• L229-381 scripts/seed-prediction-markets.mjs• L200-397 scripts/seed-national-debt.mjs• L7-188
Consumer-price datasets
Consumer-price datasets turn national market reports and price indexes into dated food observations plus country context. Each source has its own cadence, fetch budget, and fetch-age limits. Context-only sources add background for a country without acting as the main observation series. See Consumer-price validation for the source-specific quality rules.
The refresh keeps prior observations when a source returns no usable rows or older dates. It merges valid new observations into history, marks failed context without discarding a usable reference date, and activates the source only after a successful publication. This keeps an outage from looking like zero prices or a backward move in the series.
References: scripts/…/food-prices-seed.mjs• L12-31 scripts/…/food-prices-seed.mjs• L34-63 scripts/…/food-price-sources.json• L1-72 scripts/…/food-price-sources.json• L87-155
Trade and supply-chain datasets
Trade and supply-chain refreshes combine freight indexes, trade policy, customs revenue, trade flows, tariff trends, and US import-duty coverage into product-ready data. Separate keys and coverage manifests stop a failed scope from erasing the wider dataset.
Flow and tariff windows are seeded at the widest supported range and narrowed at read time. A flow is published only when its export and import sides are complete. Oil measures keep production, exports, imports, and AIS loadings separate. PortWatch adds dated traffic history for named chokepoints and only enables year-over-year comparison when its prior-year window has enough published days. US duty data publishes shards before its release catalog, so readers see one complete release or the prior release. UN Comtrade bilateral data publishes country product views with matching partner evidence, uses a country coverage floor, and retains a limited number of last-good country pairs when a monthly pass is partial. The extended HS4 pass has its own scope and cadence so wider product coverage does not consume the core run's full request budget.
The LNG origin dataset uses UN Comtrade HS 271111 to estimate each importer's share supplied from Qatar and the UAE. It compares a pre-crisis baseline with the latest complete monthly flows and chooses weight or value as one consistent measure for each window. If too much trade is unattributed or the baseline country count collapses, the affected share is withheld or the last good dataset is kept. Malaysia's DOSM SITC dataset adds monthly section-level exports, imports, and trade indexes to the same product data path, with content age based on the latest reported month rather than the fetch time.
References: scripts/seed-supply-chain-trade.mjs• L16-51 scripts/seed-comtrade-bilateral-hs4.mjs• L28-71 scripts/seed-comtrade-bilateral-hs4.mjs• L797-859 scripts/seed-comtrade-bilateral-hs4-ext.mjs• L5-17 scripts/seed-comtrade-bilateral-hs4-ext.mjs• L46-72 scripts/seed-comtrade-bilateral-hs4-ext.mjs• L246-290 scripts/seed-trade-sitc-my.mjs• L3-13 scripts/seed-trade-sitc-my.mjs• L27-85 scripts/seed-portwatch.mjs• L21-40 scripts/seed-oil-trade-series.mjs• L1-63 scripts/…/us-hts-chapter99.mjs• L423-568 scripts/…/comtrade-lng-origin.mjs• L1-20
Sanctions and source lists
The sanctions seed turns official lists into separate pressure, screening, change-history, and lookup projections. OFAC and SEMA supply the country pressure view. Export-control and restricted-party lists support screening and changes without being mixed into that pressure count. One failed list can be retained without falsely marking it as current.
Large XML responses are processed as a stream. Each source has a bounded retention window and reports current, retained, or unavailable state. Content age remains separate from fetch time. Attribution checks expose missing country or registry links. Change history records gaps when a baseline is unavailable rather than inventing additions or removals.
References: scripts/seed-sanctions-pressure.mjs• L23-50 scripts/seed-sanctions-pressure.mjs• L75-164 scripts/seed-sanctions-pressure.mjs• L344-365 scripts/seed-sanctions-pressure.mjs• L869-1062 scripts/_csl-sanctions.mjs• L1-35 scripts/_sema-sanctions.mjs• L12-30 scripts/_sanctions-changes.mjs• L13-79 scripts/seed-sanctions-pressure.mjs• L629-866
Derived and operational datasets
Some seeds do not collect a primary feed. They build operational data from other snapshots so product readers can use a stable result without repeating source work.
Forecast refreshes publish full and dashboard projections. Probability trails and forecast history support later resolution. Pending judgments remain pending when evidence or independent judges are unavailable.
Insights refreshes digest data into clustered and ranked stories. A shared story identity keeps edited copies of one story together while preserving distinct events. Corroboration counts distinct publisher families rather than feed labels, and unknown labels remain separate so a new source cannot claim independence by accident. Independent publisher families, grounding checks, stable story identity, and shared source policy protect the brief. The seed preserves a good public payload when synthesis degrades and records the failure state separately. The digest runner then uses the same story pool for brief composition and channel messages. It records one run result for operations, sends only when a rule is due, and keeps a failed user's last-sent marker unchanged so the rule can retry on the next tick. See Alerts and notification delivery for the event-to-user path.
Resilience refreshes warm country scores, rebuild the ranking, and publish score intervals from the same score cohort. Country history then combines current risk, resilience, and news into dated country records only when all required inputs are fresh. Demographics and education data run as independent stages, retaining only the failed stage from the previous snapshot and marking the result degraded until all stages complete.
Military CII data is an operational projection rather than a primary feed. It combines classified flight data with relay AIS candidates and disruption counts, then separates own presence from foreign presence before the risk scorer reads it. Cross-Strait activity keeps a durable archive and a bounded bootstrap projection. Tension evidence reads several of these operational inputs and stores one reading per pair, channel, and day for a bounded history. Digest notifications turn accumulated stories into scheduled user deliveries. The digest path and notification relay share stable event identity and delivery rules so a story is not sent twice across channels.
References: scripts/seed-forecasts.mjs• L57-60 scripts/seed-country-history.mjs• L25-120 scripts/seed-demographics-capability.mjs• L31-137 scripts/seed-military-cii.mjs• L355-512 scripts/seed-cross-strait-activity.mjs• L97-304 scripts/seed-tension-evidence.mjs• L1-91 scripts/seed-digest-notifications.mjs• L484-521 scripts/seed-digest-notifications.mjs• L2990-3179 scripts/…/story-identity.js• L5-43 scripts/…/story-identity.js• L461-518 scripts/…/publisher-families.js• L331-440
Relay components
Relay components provide a server-side path for live feeds, protected upstreams, and long-running refresh loops. They apply authentication, request limits, cache behavior, and controlled failure states so browser requests do not carry source credentials or depend on source network rules.
The AIS relay maintains source connections, serves bounded snapshots, and runs related polling work. Its liveness decision uses recent accepted position data rather than an open socket, so a silent upstream is visible and can be recycled. The relay reports silence before recycling the connection. Repeated rate-limit failures move to a slower reconnect ceiling, while an authentication failure uses a slow probe instead of repeated immediate retries. Its boot guard reads the freshness marker written by a seed and delays a refresh when the last attempt is still recent. OpenSky selects one effective route per request and reports route rejection separately from account throttling so operators can choose the right recovery action.
RSS relay requests use an allowlisted source domain set and guarded address lookup before connecting. This keeps a source proxy from being used to reach private network targets through a changed or rebound feed host. The notification publisher uses the same relay path to place normalized events on the shared queue, with coalescing based on a stable event family when one exists.
The notification relay consumes normalized events from Redis, reads enabled rules through protected Convex actions, and sends only to configured channels. It suppresses blocked links before delivery and uses a short fail-closed entitlement check for paid alerts. Quiet-hour events remain bounded until wake-up and are checked again against current rules before release. Per-user deduplication keeps one event from reaching a channel twice. Shipping webhooks use one event registry and one signed delivery format. Threshold events create one delivery per upward crossing. Occurrence events create one stable delivery per event identity and retry it under its limit. The API CORS preflight worker owns the browser cross-origin boundary for api.worldmonitor.app.
This diagram shows the relationship between a product request, an edge route, and a long-lived relay.
sequenceDiagram
participant Product as Product request
participant Edge as Edge route
participant Relay as Long-lived relay
participant Source as External source
participant Cache as Redis cache
Product->>Edge: Request data
Edge->>Relay: Authenticated bounded request
Relay->>Source: Fetch or read feed
Source-->>Relay: Data or source outcome
Relay->>Cache: Store usable data and health
Relay-->>Edge: Payload or declared fallback
Edge-->>Product: Response
Relay authentication normally fails closed. An unauthenticated mode must be explicit for controlled development or self-hosted use.
References: api/_relay.js• L7-123 server/…/relay.ts• L3-25 scripts/ais-relay.cjs• L741-819 scripts/ais-relay.cjs• L937-981 scripts/ais-relay.cjs• L1004-1111 scripts/…/rss-allowed-domains.json• L1-10 scripts/notification-relay.cjs• L97-110 scripts/notification-relay.cjs• L149-224 scripts/notification-relay.cjs• L378-438 scripts/notification-relay.cjs• L602-820 scripts/…/event-triggers.json• L1-27 workers/…/index.js• L1-49
Worker-controlled ingestion
The company monitoring worker handles provider work that is not a simple periodic snapshot. Convex owns the queue, tenant scope, selection, leases, checkpoints, replay, and final receipts. The worker claims one item, calls the configured provider, and finalizes the result with its lease token.
The lease is the ownership boundary. If the worker stops after provider work but before finalization, the lease expires and Convex can replay the item. A stale worker cannot finalize a replacement result because lease and revision checks fence it out. Health metadata describes these outcomes but does not control queue state.
This diagram shows the lease boundary that protects work ownership and replay.
sequenceDiagram
participant Worker
participant Convex
participant Provider
participant Health as Redis health metadata
Worker->>Convex: Claim work
Convex-->>Worker: Work and lease token
Worker->>Provider: Fetch or process evidence
Provider-->>Worker: Result or error
Worker->>Convex: Finalize with lease
Convex-->>Worker: Completed, replayed, fenced, or error
Worker->>Health: Publish bounded outcome
References: scripts/company-monitoring-worker.mjs• L1-15 scripts/company-monitoring-worker.mjs• L122-196 scripts/company-monitoring-worker.mjs• L353-803
Product data availability
Product readers use canonical data keys and freshness records instead of calling every external source directly. Health checks combine data presence, freshness, coverage, source state, and selected probes so operators can tell a dead scheduler from a slow source or a partial response.
The same data can have different availability states. A current snapshot can have a failed history append. A retained snapshot can remain useful while its source is failing. These states stay separate so product safety and operational truth do not compete.
Live product views
Live views read bounded API results after page load and apply timestamp and partial-result rules. The public bootstrap and RPC policy exposes a curated set of seeded results rather than every internal cache or source feed. The widget data policy is the catalog for bootstrap keys and generated GET routes. A byte budget keeps the bootstrap usable, while dashboard bundle budgets limit the size of the product code loaded with those views. The coverage catalog gives the product and external agents a consistent view of named panels, map layers, and data tools.
Source metadata keeps source identity, publisher family, outlet profile, source tier, language, and local geography consistent across ingestion and product views. The attribution inventory discovers used hosts and checks them against reviewed provider records. Source origin identifies where a publisher or issuing body is based, while coverage geography identifies what the source reports on. Allowlisted domains protect RSS collection, while browser and server views derive their feed placement and language rules from one catalog. These metadata describe trust and coverage; they do not create another source path.
Public filter contracts keep allowed economic series, forecast domains, infrastructure baselines, chokepoints, and fuel modes aligned between product readers and external API clients. These contracts describe the supported query space; they do not create another ingestion path.
Selected collectors create normalized notification events from newly observed domain data. The relay deduplicates events before the shared queue. The notification system applies user rules and delivery policy after that point. See Alerts and notification delivery for the event-to-user path.
News source definitions use one catalog for browser and server views. The catalog keeps source identity, placement, order, language, and surface-specific URL overrides together, then produces the feed views used by each product surface. Gate exemptions keep selected feeds available in editions where the normal source gate would omit them. Strategic local sources can remain reachable across digest languages. This prevents a source or category from silently diverging between the browser and server data paths.
References: src/…/country-brief-source.ts• L11-85 api/health.js• L1889-1975 api/seed-health.js• L552-731 scripts/_widget-data-policy.cjs• L1-4 scripts/…/bundle-budgets.json• L1-11 scripts/…/bundle-budgets-pro.json• L1-11 scripts/…/openapi-filter-param-contracts.json• L1-64 scripts/build-coverage-catalog.mjs• L1-25 scripts/source-attribution.mjs• L1-56 scripts/source-origin.mjs• L1-10 scripts/source-origin.mjs• L948-1018 shared/news-catalog.ts• L1-180 scripts/…/news-catalog.json• L1-219 scripts/…/outlet-profile.json• L1-8 scripts/…/publisher-families.js• L232-347 scripts/…/source-geography.json• L1-29 scripts/…/source-tiers.json• L1-12 scripts/…/x-account-source-tiers.json• L1-32 scripts/…/rss-allowed-domains.json• L1-10
Committed dataset publication
Committed dataset publication serves dated, crawlable views from known snapshots. The corpus builder does not fetch a source at page-build time. It checks snapshot identity and age, renders static pages and downloads, and uses the source observation date for page freshness. Editorial crisis records add reviewed background, source links, timelines, actors, and watch indicators to those dated views; they are curated context rather than a second live event feed.
Crawlable crisis trackers use explicit country boundaries and coverage notes, so a dated tracker is not mistaken for a complete regional event feed. The source catalog and attribution inventory connect published data to named providers, source origin, coverage geography, and licensing review. The origin catalog uses explicit overrides or host rules and rejects an unresolved provider instead of guessing. Generated coverage, product facts, sitemap, and agent-readable documents are downstream views of these reviewed inputs. They do not create a second live ingestion path.
References: scripts/build-crawlable-corpus.mjs• L1-5 scripts/build-crawlable-corpus.mjs• L380-487 scripts/…/crawlable-crises.json• L1-19 scripts/…/crisis-editorial.json• L1-35 scripts/build-coverage-catalog.mjs• L9-25 scripts/source-attribution.mjs• L1-56 scripts/source-origin.mjs• L1-16 scripts/source-origin.mjs• L994-1018 scripts/…/product-catalog.generated.json• L1-11 scripts/…/product-facts.generated.json• L1-11
Historical event dataset
The historical event dataset preserves selected events after live Redis snapshots are overwritten. Collectors project stable event fields into a common record, publish the live snapshot first, and append history through a protected Convex relay.
History answers time-range and similarity questions that a current snapshot cannot answer. Stable producer identities support deduplication and retraction. Event time remains separate from ingest time. Retention removes records outside the history window, while tombstones stop a retracted identity from returning.
History append is fail-open for the live collector. A relay or embedding failure does not undo a successful live publish. The append path also records durable per-source ingest health, separate from canonical seed freshness, so a healthy snapshot cannot hide a history path that receives nothing.
References: scripts/_seed-history.mjs• L1-32 scripts/_seed-history.mjs• L741-889 scripts/_seed-history.mjs• L993-1054 scripts/seed-cross-strait-activity.mjs• L346-445 convex/intelHistory.ts• L11-30 convex/intelHistory.ts• L198-324
Operational freshness acceptance
The freshness monitor turns source health into a fail-closed operational verdict. It allows only bounded states with a clear reason, owner, and expiry. A source that ran and failed is not treated as harmless merely because it is on-demand.
Committed snapshots need a separate pulse check because a live health endpoint cannot prove that a scheduled file refresh and corpus merge succeeded. Post-merge deploy checks cover the gap between a green repository and data services that are live.
The Convex probe filter keeps expected missing-public-function noise out of Sentry while leaving deploy skew visible. The generated list covers internal probe names only; public function names remain visible because they can signal a real deployment mismatch. This supports operational checks around worker and history paths without treating a missing public function as a successful data refresh.
References: scripts/check-seed-freshness.mjs• L38-103 scripts/check-seed-freshness.mjs• L144-280 scripts/check-pulse-freshness.mjs• L42-167 scripts/check-postmerge-deploys.mjs• L1-35 scripts/update-seed-health-statuses.mjs• L1-31 scripts/…/sentry-convex-internal-probe-filters.txt• L1-7 scripts/…/sentry-convex-internal-probe-filters.txt• L193-207 scripts/…/sentry-convex-internal-probe-filters.txt• L318-321
Change checklist
When these files or sections are changed, remember to consider these:
- A new canonical key or projection needs matching freshness metadata and health registration.
- A cadence change must keep the data TTL, staleness budget, bundle interval, and boot-refresh guard aligned.
- A changed source shape must update both normalization and the product reader that consumes it.
- A bundle dependency must preserve order and fit inside the admission budget.
- A source fallback must record which channel produced the result and must not mix incomparable vintages in one snapshot.
- A GDELT materializer change must keep the current GKG/export cohort symmetric, verified, and replayable until every required output is written.
- An AAII change must keep the source-dated week, percentage checks, retained-week merge, and freshness failure behavior aligned.
- A sanctions source change must keep pressure sources separate from screening sources and preserve per-source retention, attribution health, and change baselines.
- A notification relay change must keep protected rule reads, entitlement fail-closed behavior, blocked-link checks, quiet-held entries, and per-user deduplication aligned.
- A coverage-indexed dataset must publish its scope manifest only when the advertised member data is trustworthy.
- A contract-mode seed must keep
declareRecords, source and schema versions, content-age metadata, and reader decoding behavior aligned. - A compressed seed payload must keep its storage prefix, decoded-size limit, and every canonical reader on the same codec.
- An aviation change must keep paid-source throttling, the shared billing counter, FAA/NOTAM coverage, the merged bootstrap, its separate meta key, and closure notification state aligned.
- An NMS change must preserve the delta cursor, daily full-load guard, state retention, activation marker, and the production-only closure publication boundary.
- A relay boot-refresh change must keep recent-attempt suppression, in-process cadence, source health, and paid-provider limits aligned.
- An AIS watchdog change must keep accepted-data freshness, silence reporting, reconnect recycling, throttle backoff, and authentication probing aligned.
- An advisory change must keep the complete country index separate from the capped news list and preserve the minimum coverage floor.
- A tender adapter change must preserve open-opportunity filtering, source status, attempt pacing, incomplete-window rejection, and prior-snapshot retention.
- A UKMTO change must keep the compact table decoder, incident identity rules, event deduplication, activation marker, and source contract aligned.
- An LNG origin change must keep the HS code, Hormuz partner scope, complete-month selection, attribution threshold, and country-count floor aligned.
- China macro changes must keep transport freshness, content age, provenance, source decisions, and completion metadata distinct.
- A consumer-price source change must keep source cadence, context-only status, observation dates, reference dates, content-age limits, and last-good merge behavior aligned.
- A reservoir-level change must keep the committed baseline, Redis pass state, granule discovery window, complete-day publication, product-specific fetch budgets, watchlist coverage floor, and source-age semantics aligned.
- A new event producer must keep its stable identity and deduplication material aligned with history and notification consumers. See Alerts and notification delivery.
- A history append change must keep ingest-health outcomes separate from canonical seed freshness and preserve the live publish fail-open rule.
- An IDMC or border route change must keep source-specific content age, reporting date, coverage floor, activation marker, and reader status aligned.
- A submarine-cable scope change must update the curated source list, gap report, map configuration, minimum coverage floor, and source attribution together.
- Forecast input changes must preserve the full-versus-dashboard projection contract, probability trail, shadow-bet intake, and resolution ledger history reach.
- LLM-generated product text must keep its grounding, citation, fallback, and degraded-state contract across the seed and reader.
- A retained source must keep source health and content age separate from fetch time.
- A source host or provider change must update attribution, origin, coverage geography, outlet profile, publisher family, and public source identity.
- A source attribution change must keep lexical host discovery, provider grouping, reviewed manifest status, and origin resolution in agreement.
- A source catalog change must keep feed placement, language scope, gate exemptions, source tiers, local geography, and browser/server projections aligned.
- A new or changed RSS host must update the guarded domain allowlist and the source's public attribution together.
- A new product-readable data set must be added to the correct bootstrap or RPC allowlist, coverage catalog, and machine-readable API contract when applicable.
- A webcam catalog change must keep the geographic index, metadata set, active pointer, and expiry window on the same version.
- Orbital changes must keep the CelesTrak catalog, watched-site records, water-extent series, shared archive backoff, activation markers, and launch gate aligned.
- Watched-site changes must keep the site cursor, country rollups, change-scoring state, alert state, and publish order aligned.
- Water-extent changes must keep the reservoir watchlist coverage, accumulated series, latest summary, thresholds, and source-age semantics aligned.
- A CORS edge-worker change must preserve the API origin policy and endpoint-specific preflight exceptions.
- A webhook event change must keep the shared event registry, producer identity, subscriber scope, signature, retry, and delivery worker contracts aligned.
- Notification dedup changes must keep coalescing keys, severity policy, queue publication, rollback, and bounded fail-open fallback behavior aligned across every publisher.
- A prediction-market change must keep source completeness, category partitioning, country-index publication, and the canonical bootstrap transform aligned.
- A news source change must keep the shared catalog, browser projection, server placement, category order, language scope, gate exemptions, and surface-specific URL overrides aligned.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| A seed treats an empty or thin response as fresh | A product view shows a false empty or incomplete state | Record floors, coverage checks, and last-good preservation |
| A retry waits beyond the seed or bundle budget | Due members age or the bundle hides their result | Backoff and next-request time count against the phase budget |
| A data TTL expires before the health budget | A late run looks like missing data instead of stale data | TTLs outlive staleness gates with deliberate headroom |
| A source fallback changes data vintage without recording it | A trend can look like a real change | Single-channel selection and provenance |
| A GDELT export cohort is missing or partly rebuilt | Conflict or country baselines look falsely calm or incomplete | Cursor-based repair and complete-day checks |
| A reservoir's late satellite granule falls outside the discovery window | A revised or delayed level never reaches the product view | Look back beyond the cursor and retain processed names longer than the lookback |
| An incomplete reservoir granule group advances a day as complete | A pass or change metric uses partial satellite coverage | Group-level completion and retry state |
| A reservoir upstream fails after some old data is available | A level view is erased or is reported as freshly updated | Preservable fetch failure, last-good retention, and separate run health |
| An aviation source fails while the merged bootstrap is stamped fresh | Airport coverage looks normal for an unsupported airport | Per-source coverage, aggregate meta, and explicit unknown rows |
| The paid aviation path ignores the shared billing cycle | Seed and request-time traffic can exceed the provider allowance | One billing-cycle counter and reservation before the sweep |
| NMS is not current but ICAO data is published alone | Existing closures can be cleared or re-notified from a partial view | NMS activation state and last-good retention |
| A relay restarts and treats a recent failed attempt as due | Paid or rate-limited sources are fetched too often | Boot guard keyed to recent attempt time |
| An AIS socket stays open but stops sending positions | Operators and readers treat a silent feed as live | Monotonic data-based freshness verdict and bounded recycle policy |
| AIS throttling is retried at the ordinary reconnect ceiling | The provider keeps rejecting the stream and the block lasts longer | Failure classification and an escalated throttle ceiling |
| Relay authentication is absent without explicit opt-in | Protected upstream routes become public | Shared-secret check that fails closed |
| An RSS host is added without guarded allowlist coverage | The relay rejects a valid feed or can expose a private target through DNS rebinding | Domain allowlist and guarded lookup |
| Browser and server news catalog projections drift | A feed appears on one product surface but not the other | One JSON catalog with derived surface views |
| A language or gate key changes in only one catalog view | A digest or panel silently loses its intended source | Shared category keys, placement rules, and gate exemptions |
| Publisher labels are counted as independent publishers | One newsroom's reprints pass a corroboration gate | Curated publisher families and fail-closed singleton labels |
| A worker dies after claiming work | Work appears lost or may be finalized twice | Lease expiry, lease tokens, replay, and fencing |
| History append fails after live publication | Current data stays usable but historical search stops growing | Fail-open live publish and separate history health |
| A food source returns no rows or older dates | The price view is erased or moves backward | Validated merge, content-age checks, and activation state |
| Orbital Watch is disabled or its activation marker is not written | Satellite-derived views remain unavailable or appear only partially active | Explicit launch gate and activation after a valid publication |
| The shared satellite archive refuses a request | Watched sites or reservoir water data can be refreshed too often or diverge in recovery | Shared refusal backoff and retained canonical data |
| The orbital catalog has no sensor-profiled satellites | Site evidence cannot use current orbital elements | Require a non-empty catalog before site publication |
| A bootstrap grows beyond its volume budget | Initial product loading becomes slow or incomplete | Byte ceilings and growth checks |
| Redis SET NX fails during a high-priority event | A critical event may duplicate or a hot loop may suppress it | One bounded in-process fallback reservation and telemetry |
| A webhook occurrence loses its stable id | Subscribers receive duplicates | Shared event identity, pending state, tombstones, and retry ceiling |
| Story identity splits edited copies of one report | Corroboration and digest ranking become inconsistent | One shared identity and similarity contract for clustering and delivery |
Key files
| File | Why to read it |
|---|---|
| scripts/_seed-utils.mjs | Shared seed lifecycle for locks, fetch budgets, validation, publication, preservation, payload decoding, and freshness |
| scripts/_seed-contract.mjs | Contract for source identity, record counts, versions, content age, and staleness limits |
| scripts/_seed-payload-codec.mjs | Shared representation for large stored seed payloads |
| scripts/_bundle-runner.mjs | Bundle scheduling, dependencies, time budgets, and failure reporting |
| scripts/railway-services.json | Deployed seed and relay entry points with operational cadence |
| scripts/seed-reservoir-levels.mjs | Reservoir watchlist refresh, baseline and Redis state merge, coverage acceptance, and canonical publication |
| scripts/_reservoir-levels.mjs | Shared reservoir observation reduction, quality checks, change metrics, and source access |
| scripts/…/reservoir-levels-baseline.json | Committed historical reservoir passes used to start and recover the rolling dataset |
| scripts/seed-orbital-catalog.mjs | Sensor-profiled satellite catalog, orbital-element history ring, source validation, and activation |
| scripts/seed-orbital-sites.mjs | Watched-site evidence, country rollups, change scoring, archive backoff, and alert publication |
| scripts/seed-orbital-water-extent.mjs | Reservoir radar water series, latest summary, watchlist coverage, and shared archive recovery |
| scripts/seed-aviation.mjs | Independent aviation source paths, paid refresh controls, coverage states, merged bootstrap, and disruption notifications |
| scripts/seed-notam-nms.mjs | FAA NOTAM state, delta and full refresh policy, airspace data, and closure publication boundary |
| scripts/ais-relay.cjs | Long-lived relay, source polling, boot freshness guard, AIS liveness, route selection, cache writes, transport health, and event publication |
| scripts/…/food-prices-seed.mjs | Consumer-price retention, validation, history merge, source health, and activation |
| scripts/…/food-price-sources.json | Consumer-price source identity, cadence, context scope, age limits, and fetch budgets |
| scripts/…/news-catalog.json | Shared source definitions, surface placement, language rules, gate exemptions, and URL overrides for browser and server news views |
| shared/news-catalog.ts | Derived browser and server catalog views and surface-specific feed rules |
| scripts/…/outlet-profile.json | Reviewed source geography, language, audience, and ownership metadata |
| scripts/…/publisher-families.js | Fail-closed publisher grouping for corroboration and article-domain identity |
| scripts/…/source-geography.json | Curated local-country coverage used to measure source geography |
| scripts/…/source-tiers.json, scripts/…/x-account-source-tiers.json | Source priority tiers for feed and X-account selection |
| scripts/…/rss-allowed-domains.json | RSS host allowlist used by the guarded relay fetch path |
| scripts/…/notification-dedup.cjs | Shared event coalescing, severity-aware failure policy, fallback deduplication, and telemetry |
| scripts/…/openapi-filter-param-contracts.json | Allowed public filter values that keep API queries aligned with product data |
| scripts/…/bundle-budgets.json, scripts/…/bundle-budgets-pro.json | Dashboard and Pro bundle size limits that protect product data loading |
| scripts/seed-gdelt-bulk-materializer.mjs | GDELT rolling windows, archive repair, cohort checks, cursor replay, and operational output keys |
| scripts/seed-aaii-sentiment.mjs | Source-dated sentiment validation, retained weekly history, and freshness rules |
| scripts/seed-sanctions-pressure.mjs | Streaming list ingestion, per-source retention, pressure and screening projections, and attribution health |
| scripts/seed-no2-activity.mjs | Sentinel-5P activity refresh, stored regional series, baseline backfill, and publication gates |
| scripts/seed-natural-events.mjs | Natural-event provider merge, storm projections, retention, and source health |
| scripts/seed-security-advisories.mjs | Advisory normalization, complete country index, and source-health state |
| scripts/seed-global-tenders.mjs | Multi-register procurement collection, source pacing, normalization, and source health |
| scripts/seed-supply-chain-trade.mjs | Shipping, trade coverage manifests, tariff trends, customs data, and US duties |
| scripts/seed-prediction-markets.mjs | Prediction-market collection, category pools, country index, and completeness gates |
| scripts/seed-cross-strait-activity.mjs | Durable regional activity archive, bounded bootstrap, source health, and history projection |
| scripts/seed-tension-evidence.mjs | Cross-source tension readings, daily retention, and last-good behavior |
| scripts/company-monitoring-worker.mjs | Leased provider work and replay-safe finalization |
| scripts/_seed-history.mjs | Live-event projection, durable history delivery, and ingest-health reporting |
| convex/intelHistory.ts | Durable history deduplication, search, retention, and retraction |
| scripts/…/story-identity.js | Shared story matching used to keep edited reports together across clustering and digest delivery |
| scripts/…/crawlable-crises.json | Explicit scope and coverage for dated crisis trackers |
| scripts/…/product-catalog.generated.json, scripts/…/product-facts.generated.json | Generated public product facts that downstream publication exposes without creating a live source path |
| scripts/notification-relay.cjs | Protected rule loading, entitlement checks, quiet-hour handling, notification deduplication, suppression, and channel delivery |
| scripts/source-attribution.mjs | Runtime host discovery, provider identity grouping, and reviewed attribution checks |
| scripts/source-origin.mjs | Closed-world publisher-origin resolution kept separate from coverage geography |