Seed pipeline orchestration
Seed pipeline orchestration turns live feeds, official releases, reviewed registries, and data files into refreshable monitoring datasets. It gives each dataset a controlled refresh path, keeps accepted data available during source failure, and records freshness and coverage state for downstream readers.
This page is a child of Data ingestion and refresh pipelines. For the path from accepted signals to user messages, see Alerts and notification delivery. For source checks on consumer prices, see Consumer prices validation.
System purpose
Refresh pipeline
The refresh pipeline moves source data through a domain seeder into a stable dataset. Each seeder defines the source meaning, useful coverage, accepted age, and output identity for one resource. Readers use the accepted dataset instead of calling every upstream service directly.
The system separates data use from data collection. A source outage can leave the last accepted value available while freshness, content age, and coverage state show that the value is old or incomplete.
This diagram shows how source data becomes product-ready state.
flowchart LR
A[Live source or reviewed artifact] --> B[Domain seeder]
B --> C[Normalize and validate]
C -->|accepted| D[Canonical dataset]
C -->|failed or incomplete| E[Last accepted dataset]
D --> F[Freshness and coverage facts]
E --> F
D --> G[APIs, clients, and derived datasets]
F --> H[Health and operator views]
References: scripts/_seed-utils.mjs• L2691-2930 scripts/_seed-utils.mjs• L3090-3115 scripts/seed-portwatch.mjs• L597-615
Bundle scheduling
Bundles group related seeders under one scheduled job while keeping each member's cadence and output contract independent. A fast event feed can share a service with slow reference data without forcing all members to refresh together.
The macro bundle gives economic, food-price, education, physical-premium, and Malaysia trade sources their own members. It uses protected early slots for education and physical premiums, rotates the lead macro member on selected ticks, and places slow or large history loads late so an established member does not lose its admission slot. A section is admitted only when its timeout and cleanup reserve fit inside the bundle budget.
The climate bundle groups climate normals, anomalies, disasters, ocean and ice state, carbon monitoring, orbital catalog and site data, orbital water extent, and strategic reservoir levels. These members use very different refresh intervals, so the bundle is a scheduling boundary rather than a shared data contract. Reservoir levels run daily even though the satellite observations they use arrive much later than acquisition.
The regional-energy bundle uses a scheduler turn. A held lease or a temporary Redis failure defers the tick without advancing the turn. A completed tick advances it even when one member fails, so one bad source does not keep the same member first forever.
| Pattern | Purpose |
|---|---|
| Shared bundle | Reduce service count while preserving member cadence. |
| Ordered member | Let a derived member read a peer from the same run. |
| Freshness gate | Avoid work while accepted data is still within its interval. |
| Time budget | Stop work that cannot finish safely in the remaining window. |
| Member isolation | Disable or defer one source without stopping siblings. |
| Completion marker | Show that the member's related writes finished. |
| Rotating turn | Give delayed members a fair lead slot across recovery runs. |
References: scripts/seed-bundle-canada.mjs• L41-121 scripts/seed-bundle-climate.mjs• L4-14 scripts/seed-bundle-macro.mjs• L9-16 scripts/seed-bundle-macro.mjs• L121-159 scripts/seed-bundle-regional-energy.mjs• L6-55 scripts/_bundle-runner.mjs• L13-31 scripts/_macro-bundle-order.mjs• L4-27 scripts/_bundle-turn.mjs• L23-85 scripts/…/food-price-sources.json• L233-467 scripts/…/regional-energy-feeds.json• L222-257
Seeder lifecycle
The shared lifecycle gives a resource one controlled path through lock, fetch, validation, publication, metadata, verification, and unlock. The lock prevents overlapping runs from changing the same resource. A bounded fetch leaves time for publication and cleanup.
Publication replaces the canonical value only after domain checks pass. Companion keys and post-publish work can add read models, source snapshots, or durable records, but they do not make an unaccepted canonical result fresh. A completion marker is written after the related outputs are complete.
The lifecycle distinguishes an upstream failure from a bad local result. A failed fetch preserves last-good data when the domain allows it. A validation failure does not make old data look new. Empty results are rejected unless the domain declares zero valid. Fetch deadlines are clamped inside bundle section limits so the seeder can release its lock or preserve data before the bundle stops it. Lock renewal and fenced writes prevent a run that loses ownership from publishing over a newer run.
References: scripts/_seed-utils.mjs• L453-621 scripts/_seed-utils.mjs• L622-735 scripts/_seed-utils.mjs• L2641-2669 scripts/_seed-utils.mjs• L2691-2930 scripts/_seed-utils.mjs• L2940-3113
Dataset contract
Canonical dataset contract
The canonical dataset is the stable value that downstream readers use for a domain resource. Contract-aware seeders carry product data and run facts together, while public readers remove the run facts before returning product data.
The contract requires a resource identity, fetch and validation rules, a record-count declaration, a source version, a schema version, a time-to-live, and a maximum stale age. A schema change requires a schema version change. A source or meaning change requires a source version change.
The shared payload codec also permits large accepted values to use a marked compressed form. Readers must understand both the ordinary JSON form and the marked form, and must reject inflated data above the decode limit.
References: scripts/_seed-contract.mjs• L22-60 scripts/_seed-contract.mjs• L62-167 scripts/_seed-payload-codec.mjs• L1-22
Freshness and last-good state
Freshness is a product contract, not only a scheduler result. A successful publish advances the resource heartbeat. A failed fetch or rejected snapshot does not make old data appear new. The system may extend the lifetime of retained data without changing the data's original fetch time.
Run freshness and content age are separate. A recent request can contain an old statistical release, market period, or displacement report. Seeders that opt into content age publish the age of the observations so health can distinguish a live transport path from current content.
Derived readers apply their own age limits to preserved inputs. This prevents a retained envelope from being read as a new signal simply because its storage lifetime was extended.
References: scripts/_seed-utils.mjs• L630-681 scripts/_seed-utils.mjs• L785-881 scripts/_seed-utils.mjs• L884-955 scripts/seed-correlation.mjs• L18-36
Durable history
History is a secondary record of noteworthy seed output. It is sanitized, bounded, embedded when configured, and sent in independent chunks to a relay. A history failure does not fail the canonical publication.
History health has its own durable state. It records whether the relay is unconfigured, failing, healthy, or losing chunks without changing the canonical seed's freshness. This keeps a healthy monitoring dataset from being marked stale because its optional history copy failed.
References: scripts/_seed-history.mjs• L134-222 scripts/_seed-history.mjs• L328-390 scripts/_seed-history.mjs• L393-438 scripts/_seed-history.mjs• L562-679
Ingestion utilities
Shared source handling
Shared ingestion utilities provide common behavior for retries, deadlines, locks, Redis access, environment loading, proxy use, and metadata. Domain code can then focus on mapping an upstream response into a useful dataset.
The shared layer selects the right provider route, distinguishes temporary transport errors from permanent request or configuration errors, and honors retry hints within a bounded budget. It also unwraps contract envelopes for readers and keeps deployment paths safe when services run with the scripts directory as their root.
The runner preserves existing keys when a refresh cannot publish, while keeping the old freshness time visible to health checks. Empty extra datasets can be skipped without clobbering last-good data. Extra datasets can publish a data key and its freshness record as one atomic pair. A hard fetch deadline turns a non-settling source request into a normal failed refresh, with enough time left for publication or cleanup. Fenced writes stop a run that lost its lock from changing a newer result.
References: scripts/_seed-utils.mjs• L66-126 scripts/_seed-utils.mjs• L368-428 scripts/_seed-utils.mjs• L453-621 scripts/_seed-utils.mjs• L785-955 scripts/_seed-utils.mjs• L1044-1176 scripts/_seed-utils.mjs• L2532-2600 scripts/_seed-utils.mjs• L2603-2669 scripts/_seed-utils.mjs• L2691-2930
Geographic normalization
Geographic normalization gives country-level records a stable identity and gives coordinate records a common country assignment. Country identity remains separate from coordinates, so a country-level observation is not mistaken for an exact site.
The CII border service uses a shared generated border model for the seeder and the scoring consumer. It assigns Tier-1 countries on land and within territorial waters, while leaving other land and distant sea points unassigned. News extraction uses country, city, and region names to support the same kind of country-aware grouping for stories.
References: scripts/…/cii-country-borders.mjs• L1-18 scripts/…/cii-country-borders.mjs• L86-100 scripts/…/cii-country-borders.mjs• L146-187 scripts/…/geo-extract.mjs• L14-18 scripts/…/geo-extract.mjs• L165-189
GDELT materialization
GDELT materialization turns frequent bulk snapshots into several bounded products. It keeps topic intelligence, conflict and unrest events, article indexes, positive events, dyad tension, and per-country event counts from one shared source clock.
The materializer accepts only a current, contiguous pair of GKG and export cohorts. It repairs missing dyad and country-event history from the public archive within one shared budget. Missing or incomplete coverage remains unknown rather than calm. It advances the cursor only after all output keys are written; if a secondary output fails, the same source cohort is replayed on the next run.
References: scripts/seed-gdelt-bulk-materializer.mjs• L172-205 scripts/seed-gdelt-bulk-materializer.mjs• L279-327 scripts/seed-gdelt-bulk-materializer.mjs• L702-760 scripts/seed-gdelt-bulk-materializer.mjs• L828-975
Model-backed transformation
Model-backed stages turn accepted source records into short briefs or structured judgments. They run after source ingestion, use bounded provider work, reject unusable output, and keep provider diagnostics outside the product data path.
News insights normalize digest items, group related stories, rank a bounded top list, and publish a separate ranked story pool for broader readers. Brief synthesis needs corroborated evidence and passes content and source checks. When synthesis fails, the seeder uses a simpler brief or preserves the last known-good public payload. Audit coverage for China feeds and the story pool remains separate from the public brief.
References: scripts/seed-insights.mjs• L922-979 scripts/seed-insights.mjs• L982-1046 scripts/seed-insights.mjs• L1113-1202 scripts/seed-insights.mjs• L1244-1381
Derived dataset stages
Some seeders consume accepted datasets and publish a higher-level view. The stage must carry enough source state to show which inputs are present, partial, stale, or unavailable. Shared run time and source-age gates prevent a derived result from using an old peer while claiming a current refresh.
The correlation seeder reads accepted military, unrest, infrastructure, earthquake, market, and news inputs in one cycle. It discards missing, malformed, wrongly shaped, or stale inputs before it groups signals by place, country, or economic entity. It then publishes domain cards only when at least one usable card remains, with separate outputs for military, escalation, economic, and disaster domains.
The current derived lanes include:
- Cross-source signals: combine accepted risk, event, market, weather, infrastructure, displacement, and media inputs into bounded signals by theater and severity.
- Country history: combine current risk, resilience, and news evidence into dated country snapshots only when the required source set is current and complete enough.
- Tension evidence: compare accepted strike, airport, military, cross-Strait, and sanctions inputs with their usual levels for monitored pairs.
- Supply-chain exposure: combine bilateral trade with country-port routes to estimate commodity exposure at major chokepoints. Use route-based fallback when trade coverage is absent.
- Forecast views: generate bounded predictions, retain their input context, and resolve them later through a durable ledger and evidence archive.
- History and alert views: retain source records and state needed for later search, notification, or resolution.
References: scripts/seed-correlation.mjs• L26-53 scripts/seed-correlation.mjs• L67-152 scripts/seed-correlation.mjs• L830-965 scripts/seed-cross-source-signals.mjs• L14-70 scripts/seed-country-history.mjs• L18-68 scripts/seed-tension-evidence.mjs• L33-53 scripts/seed-hs2-chokepoint-exposure.mjs• L97-191 scripts/seed-market-alert-ledger.mjs• L1-12 scripts/seed-forecast-resolutions.mjs• L1-9
Monitoring datasets
Curated registries and data artifacts
Curated registries provide stable context that live feeds cannot supply. They describe source coverage, geographic identity, event rules, product facts, or domain calendars. Seeders validate these artifacts before publication and use a slow cadence with enough lifetime to survive a missed run.
Some artifacts are shared contracts rather than seed payloads. Event trigger definitions are read by registration, evaluation, API description, and delivery surfaces. Generated product facts support public product and checkout surfaces. The generated catalog and facts must retain the same plan keys and limits across those surfaces.
The news catalog is the single source for feed identity, category placement, edition order, language scope, and browser-specific overrides. Crisis editorial data adds reviewed background, actors, timelines, and watch indicators for named situations. The generated Sentry filter list suppresses known missing-function probe noise while leaving deploy-skew errors visible; its source of truth is the internal Convex export set, not the generated file.
References: scripts/…/event-triggers.json• L1-27 scripts/…/product-catalog.generated.json• L1-13 scripts/…/product-facts.generated.json• L1-13 scripts/…/weo-fiscal-periods.mjs• L1-14 scripts/…/news-catalog.json• L1-2 scripts/…/news-catalog.json• L4-90 scripts/…/crisis-editorial.json• L1-35 scripts/…/sentry-convex-internal-probe-filters.txt• L1-7
Procurement opportunity datasets
The global tenders dataset turns open procurement notices from official national and multilateral sources into one normalized opportunity view. It keeps source identity, official links, buyer and country context, deadlines, categories, and a small automation-fit classification so readers can compare unlike portals without treating them as one source.
Each source reports its own state. Successful sources contribute current records, while eligible sources retain still-open prior records during a short outage and mark them stale. The SAM source paces requests from its last attempt because its daily quota counts failed attempts. The merge removes duplicates and rejects records without a future deadline.
References: scripts/seed-global-tenders.mjs• L426-487 scripts/seed-global-tenders.mjs• L491-529 scripts/_global-tenders.mjs• L78-135 scripts/_global-tenders.mjs• L213-246
Economic and energy datasets
Economic and energy seeders adapt official statistical releases, market observations, food reports, fuel prices, and regional power records into country-indexed or history-based views. They preserve source age and provenance so a newly fetched old release does not look like a current observation.
The economy seeder publishes energy prices as its main view and refreshes capacity, macro signals, crude inventories, natural-gas storage, strategic petroleum reserves, and refinery inputs as separate companions. Weekly series need several usable periods before they replace their prior values. A failure in one companion does not erase healthy companions, while a total failure still blocks the main refresh.
The macro bundle runs many food-price sources as separate daily members. The food registry covers a wide mix of national retail, wholesale, farm-gate, market, consumer-price, and food-security reports across Africa, Asia, Europe, and the Americas. Each lane keeps its own cadence, age limits, attribution, and fetch budget. The shared food seeder merges dated observations with source history and retains prior country context when a report fails.
The Malaysia SITC trade seeder adds a small monthly merchandise-trade history from DOSM. It combines exports and imports by SITC section with trade unit-value and volume indexes. Its content-age rule allows for the normal release delay and one late release, while the daily macro-bundle member safely picks up revisions.
The grocery basket seeder builds comparable staple baskets from structured retail offers. It checks product identity, pack quantity, unit, currency, and plausible converted price before accepting an item. It uses learned retailer routes as a speed path and renews them when a route becomes bad. A country can remain in the result with unavailable items, but only complete baskets take part in rankings and week-over-week comparisons.
The long-term government-rate dataset combines ECB IRS data for EU markets with FRED OECD MEI series and uses a public CSV fallback when the keyed route is not available. It requires a broad country set before publication, keeps a full history beside a latest-reading view, and records the age of the newest market content separately from transport freshness.
Fuel prices combine required global coverage with optional national and regional feeds. Prices are converted to a common comparison unit and checked for plausible values. A missing optional source omits its countries, while a missing required source blocks publication so a partial snapshot cannot appear complete. The pipeline does not carry old country prices into a new weekly snapshot as if they were current; the last healthy canonical snapshot remains available through its cache lifetime.
World Bank ingestion turns annual indicator series into technology-readiness rankings, human-progress history, renewable-energy views, and RPC cache snapshots. The ranking view uses the latest valid country values within its selected year ranges. A large fall in ranking coverage preserves the prior bootstrap view while still allowing independent RPC data to refresh.
Regional energy feeds provide power demand, generation, outages, gas, and fuel-price observations for registered countries. Each provider has its own cadence and content-age limit. The feed registry keeps source identity, country mapping, licensing note, and timing rules in one reviewed artifact. A release that is posted ahead of its effective time is not treated as current until it takes effect.
Trade seeders use one shared annual-period policy so the strategic flow view, bilateral country data, and lazy fallback do not drift at the year boundary. The core bilateral HS4 run also publishes world-export state and LNG supplier shares for the Hormuz energy-shock view. Its extended companion run publishes a wider set of HS4 product groups under separate country and partner keys on a second monthly tick. Both runs pace requests, rotate countries that were missed by a short run, preserve country key pairs together, and mark coverage partial when the country floor is not met. Thin or mostly unattributed results preserve the prior supplier view.
References: scripts/seed-economy.mjs• L11-31 scripts/seed-economy.mjs• L622-688 scripts/seed-fuel-prices.mjs• L970-1011 scripts/seed-fuel-prices.mjs• L1053-1230 scripts/seed-grocery-basket.mjs• L76-95 scripts/seed-grocery-basket.mjs• L320-473 scripts/seed-oecd-lt-rates.mjs• L16-26 scripts/seed-oecd-lt-rates.mjs• L29-97 scripts/seed-oecd-lt-rates.mjs• L108-135 scripts/seed-bundle-macro.mjs• L81-124 scripts/…/food-prices-seed.mjs• L12-53 scripts/…/food-price-sources.json• L233-257 scripts/seed-trade-sitc-my.mjs• L3-23 scripts/seed-trade-sitc-my.mjs• L49-85 scripts/seed-wb-indicators.mjs• L26-40 scripts/seed-wb-indicators.mjs• L365-481 scripts/_regional-energy-source.mjs• L76-125 scripts/_regional-energy-source.mjs• L218-249 scripts/…/regional-energy-feeds.json• L222-257 scripts/seed-comtrade-bilateral-hs4.mjs• L596-683 scripts/seed-comtrade-bilateral-hs4.mjs• L685-845 scripts/seed-comtrade-bilateral-hs4-ext.mjs• L46-71 scripts/seed-comtrade-bilateral-hs4-ext.mjs• L154-290
Market and quote datasets
Market seeders turn provider quotes and historical bars into stable stock, commodity, currency, and derived market views. They preserve enough source context for country coverage, trend changes, and alert consumers to distinguish a missing quote from a true market move.
Equity quotes use the shared authorized-provider path first, with Alpha Vantage bulk data, Finnhub residual coverage, and Yahoo for configured residual symbols and regional listings. A closed multi-market day can retain the last published close without pretending that a new market observation exists. Country stock indexes are refreshed as independent companion datasets, so one country failure does not discard the global quote set and the recorded count reflects what actually landed. Commodity quotes use Alpha Vantage first and Yahoo for uncovered symbols. Their required aliases publish with the canonical result, while the optional gold context can age on its own. Commodity history uses the last completed Yahoo session when a chart has already rolled into an unopened session, so a closed market does not create a false zero change. The FX view adds year-over-year change and a two-year peak-to-trough measure for currencies outside narrow official coverage.
The FX path has a dated currency-api fallback for currencies whose Yahoo series is unusable. It samples recent and older windows, reverses the provider's currency-per-USD values into the view's USD-per-currency shape, and accepts the fallback only when both windows contain enough dated snapshots. This keeps a second source from creating a result from a single missing or late daily table.
The market-alert ledger reads fresh seeded quotes and digest evidence. It records alert emissions in persistent time windows, resolves closed windows against the evidence archive, and publishes a rolling scorecard. It therefore belongs after quote ingestion rather than inside a provider fetch.
References: scripts/seed-market-quotes.mjs• L22-39 scripts/seed-market-quotes.mjs• L52-103 scripts/seed-market-quotes.mjs• L110-159 scripts/seed-market-quotes.mjs• L176-224 scripts/seed-commodity-quotes.mjs• L12-48 scripts/seed-commodity-quotes.mjs• L160-229 scripts/seed-commodity-quotes.mjs• L244-329 scripts/…/yahoo-closed-session.cjs• L9-55 scripts/seed-fx-yoy.mjs• L17-50 scripts/_currency-api.mjs• L1-61 scripts/seed-market-alert-ledger.mjs• L1-12 scripts/seed-market-alert-ledger.mjs• L53-75
Sentiment and forecast datasets
Market sentiment seeders turn slow survey releases into compact histories for trend and extreme-reading views. The AAII seeder uses the current survey page as its source and keeps a bounded 52-week history with the latest reading, prior reading, rolling averages, a historical comparison, and recent extreme counts. It accepts a reading only when the source date and percentages form a valid current week, so a changed page shape or an old page does not replace the accepted dataset.
Forecast ingestion has separate generation, history, resolution, evidence, and scorecard stages. The resolver waits for a complete evidence window and independent judgments before creating a terminal result. Pending work keeps bounded attempt state and backoff. Terminal receipts and scorecards make the public result depend on a durable decision record rather than a transient worker run.
References: scripts/seed-aaii-sentiment.mjs• L7-17 scripts/seed-aaii-sentiment.mjs• L251-317 scripts/seed-aaii-sentiment.mjs• L386-468 scripts/seed-forecast-resolutions.mjs• L53-100 scripts/seed-forecast-resolutions.mjs• L647-790 scripts/_forecast-scorecard.mjs• L124-182
Conflict, military, and maritime datasets
Conflict and military seeders turn event records, media coverage, aircraft activity, vessel candidates, navigational warnings, and AIS disruption into country and theater intelligence. Military CII keeps own and foreign presence separate before scoring. Unknown ownership is not promoted to foreign presence.
UCDP ingestion combines the finalized annual conflict release with a newer GED Candidate release when one exists. It keeps the annual history as the base, adds candidate events within the recent window, and publishes a smaller dashboard projection beside the canonical map and RPC data. A failed annual fetch preserves the last accepted result, and a partial candidate release remains marked partial rather than looking complete.
Unrest ingestion combines high-confidence ACLED protest records with low-confidence GDELT media signals. ACLED has one bounded display view and a wider resolution cache. The materialized GDELT source must be recent and is labeled as an unverified media signal. The merged result deduplicates by source, place, and day while retaining source links.
Military flight ingestion classifies aircraft from global or regional tracking coverage and publishes both a live view and a bounded last-good view. It derives theater posture, surge history, forecast inputs, and classification audit data from the same accepted flight set. Regional coverage cannot replace a global fallback snapshot, and a failed provider does not turn an empty collection into a valid no-flight result.
Maritime warm-ping seeding warms the navigational-warning cache through the API that owns its source parsing. It does not claim to seed on-demand aircraft details, bounding-box flight queries, vessel snapshots, or per-article summaries. This keeps cache warming separate from data paths that already have another owner.
UKMTO incident data remains a compact maritime event view. It normalizes each report into an incident identity, time, place, description, and coordinate. Malformed rows, conflicting identifiers, impossible dates, unsafe coordinates, and repeated reports are rejected or collapsed before publication.
References: scripts/seed-ucdp-events.mjs• L24-34 scripts/seed-ucdp-events.mjs• L165-245 scripts/seed-ucdp-events.mjs• L296-381 scripts/seed-unrest-events.mjs• L462-508 scripts/seed-unrest-events.mjs• L519-538 scripts/seed-military-bases.mjs• L15-49 scripts/seed-military-flights.mjs• L1750-1799 scripts/seed-military-flights.mjs• L1861-1913 scripts/seed-military-flights.mjs• L1932-2123 scripts/seed-military-maritime-news.mjs• L3-26 scripts/seed-military-maritime-news.mjs• L50-98 scripts/seed-ukmto-incidents.mjs• L6-106
Displacement and border datasets
Displacement ingestion combines broad internal-displacement records with route-specific border and arrival feeds. The border seeders share one source-routing contract, but each source keeps its own coverage and freshness state.
The UNHCR cross-border view separates stock measures from monthly flow measures. It keeps a private report history for change and acceleration checks, while the public payload exposes situations and route results without that history. A failed situation does not become a zero result.
References: scripts/seed-cross-border-arrivals.mjs• L1-15 scripts/seed-cross-border-arrivals.mjs• L154-265 scripts/seed-cross-border-arrivals.mjs• L375-471 scripts/seed-idmc-idu-displacement.mjs• L186-320
Aviation and infrastructure datasets
Aviation ingestion combines international flight delays, US FAA notices, ICAO NOTAM closures, and aviation news into one scheduled refresh surface. A shared airport registry supplies identity, coordinates, regions, and source coverage. The seeder publishes source views and a page-load bootstrap together, so a missing paid flight feed becomes unknown coverage rather than a clear airport. Severe and major disruption changes also produce deduplicated notification events.
The international path treats provider omission and provider failure as different coverage states. FAA XML, NOTAM JSON, and RSS inputs have source-specific parsing and bounded retention. The bootstrap merges current source alerts with normal, open, or unknown rows from the union of the seeder and monitored-airport registries. This keeps the page-load view aligned with the API while preserving uncertainty when a source did not answer.
Infrastructure registries provide slower context such as submarine cable routes and landing points. They are validated against a minimum strategic set before publication, which prevents a partial response from replacing the whole map. Webcam ingestion adds a spatial camera catalog beside these registries; it writes location records and camera metadata as one versioned release and changes the active version only after both parts are ready.
References: scripts/seed-aviation.mjs• L59-89 scripts/seed-aviation.mjs• L127-148 scripts/seed-aviation.mjs• L677-745 scripts/seed-aviation.mjs• L861-950 scripts/seed-aviation.mjs• L1036-1123 scripts/seed-aviation.mjs• L1314-1373 scripts/seed-notam-nms.mjs• L170-395 scripts/seed-submarine-cables.mjs• L190-356 scripts/seed-webcams.mjs• L132-153 scripts/seed-webcams.mjs• L182-223
Cyber threat datasets
Cyber threat ingestion combines indicator feeds into a bounded list of malicious IPs, domains, and URLs. It validates provider coverage before combining results, deduplicates indicators by source identity, and adds location only when the evidence supports it.
The feed can retain a recent provider list during a rate window or temporary failure, but retained rows do not advance fresh observation time. A separate first-seen record gives indicators without upstream dates a stable discovery day. OTX pagination must complete before its contribution can qualify the snapshot, while a missing configured provider remains visible as unconfigured rather than as a valid empty feed.
The CVE reference dataset adds a focused vulnerability lookup for known-exploited CVEs. It joins the CISA KEV subset of NVD with severity, weakness, affected-product, and remediation fields. Incremental updates merge recent changes into the retained map, while periodic full refreshes remove entries that no longer meet the KEV condition and reject stores below a coverage floor. NVD request limits and retry hints are handled inside a bounded refresh, so a temporary upstream failure preserves the prior reference store.
References: scripts/seed-cyber-threats.mjs• L365-439 scripts/seed-cyber-threats.mjs• L602-641 scripts/seed-cyber-threats.mjs• L644-710 scripts/seed-cyber-threats.mjs• L717-780 scripts/seed-cve-reference.mjs• L4-28 scripts/seed-cve-reference.mjs• L195-264 scripts/seed-cve-reference.mjs• L274-373 scripts/seed-cve-reference.mjs• L385-403
Sanctions pressure datasets
Sanctions ingestion turns OFAC, SEMA, export-control, and US restricted-party lists into a pressure view and supporting lookup datasets. OFAC and SEMA drive country pressure, while the other lists add screening identities and change records. This separation keeps a display summary from replacing the complete data needed by screening and country risk.
Each source is validated independently. A valid source becomes a retained snapshot, while a failed source can remain available for a bounded period with its failure state visible. The OFAC path reads large XML releases as a stream, which keeps memory use tied to parsed records rather than the full file. Attribution checks distinguish missing country or registry evidence from a true change in sanctions pressure.
The canonical public transform removes entity indexes, source snapshots, internal state, attribution details, and screening-only fields. Companion keys retain the full entity index, country counts, source snapshots, source health, and change history. Change history advances only after current snapshots and the history record are encoded and written, so a failed secondary write does not silently move the comparison baseline. Degraded source or attribution state remains visible in freshness metadata even when the pressure view is still usable.
References: scripts/seed-sanctions-pressure.mjs• L23-50 scripts/seed-sanctions-pressure.mjs• L75-109 scripts/seed-sanctions-pressure.mjs• L134-164 scripts/seed-sanctions-pressure.mjs• L869-1058 scripts/seed-sanctions-pressure.mjs• L1084-1215
Supply-chain flow datasets
PortWatch turns daily chokepoint transit records into history, week-over-week change, and a delayed year-earlier baseline. The pipeline prefers a compact export and fills missing or older chokepoints from the query service. It can carry a recent last-good chokepoint entry without changing its original fetch time.
The chokepoint flow view combines PortWatch history with EIA baseline volumes. It needs enough history to compare a recent seven-day window with a prior ninety-day window, selects deadweight-ton baseline coverage only when that baseline is broad enough, and marks a disruption only when each of the last three days is below its threshold. Nearby severe hazards are annotations only; they do not change the flow estimate.
Port activity uses a separate country view with a target coverage set. It refreshes only a bounded number of cold countries per run and marks older carried data so a slow upstream cannot exhaust the bundle budget. An empty ArcGIS response is accepted as zero only when a separate observation proves that the country has no activity; otherwise the path retries through a proxy. Port disruptions remain a separate active-event view.
The Hormuz tracker adds daily shipment series for crude oil, LNG, fertilizer, and agriculture. It keeps a short chart window and uses the public WTO report as a source for a dedicated energy-shock view.
References: scripts/seed-chokepoint-flows.mjs• L8-26 scripts/seed-chokepoint-flows.mjs• L45-57 scripts/seed-chokepoint-flows.mjs• L85-142 scripts/seed-chokepoint-flows.mjs• L145-205 scripts/seed-portwatch.mjs• L7-38 scripts/seed-portwatch.mjs• L238-301 scripts/seed-portwatch.mjs• L304-399 scripts/seed-portwatch.mjs• L458-595 scripts/seed-portwatch-port-activity.mjs• L45-68 scripts/seed-portwatch-port-activity.mjs• L938-999 scripts/seed-portwatch-disruptions.mjs• L22-124 scripts/seed-hormuz.mjs• L1-32
News and notification datasets
News datasets need more than title parsing. Shared place data and story grouping give ingestion, insights, history, and alert paths one edit-tolerant view of a story and its countries.
Corroboration counts publisher families rather than feed labels. Unknown origins remain conservative. A syndicated copy can reduce the number of independent families, but it cannot create confidence that the source evidence does not support.
Digest notifications consume accepted stories and alert rules after ingestion. The digest job filters stale, opinion, lifestyle, and short-lived live-programming items before story grouping, then writes one bounded brief whose synthesis and ranked story pool are reused across email, chat, webhook, and brief surfaces. A separate watchlist scan can enqueue ticker events without blocking digest delivery. Delivery state, channel safety checks, entitlement handling, and unsubscribe handling remain separate from canonical monitoring datasets. For the complete condition-to-message flow, continue to Alerts and notification delivery.
References: scripts/…/geo-extract.mjs• L99-214 scripts/…/news-places.json• L1 scripts/seed-insights.mjs• L922-1046 scripts/seed-digest-notifications.mjs• L642-682 scripts/seed-digest-notifications.mjs• L754-1110 scripts/seed-digest-notifications.mjs• L1554-1587 scripts/…/news-catalog.json• L1-2 scripts/…/news-catalog.json• L187-219
Natural hazard datasets
Natural-event ingestion combines NASA EONET, GDACS, NHC storm data, and regional warnings into one event view. Source snapshots let one failed provider retain bounded evidence while source health remains visible.
GDACS type coverage stays separate from the general event cap, so a crowded earthquake list cannot hide a tropical cyclone. NHC supplies detailed storm tracks and cones when its required point coverage is complete. An event-free answer is not confused with a failed query. Western Pacific cyclone state remains separate from the general event list. The public transform removes source snapshots and failure internals, while freshness metadata keeps the degraded source state visible.
Wildfire ingestion combines FIRMS, CWFIS, and Canadian fire points. It publishes a bounded canonical list plus summary and recovery views. Missing FIRMS configuration is treated as a deployment failure rather than as a valid Canada-only result.
References: scripts/seed-natural-events.mjs• L144-171 scripts/seed-natural-events.mjs• L471-552 scripts/seed-natural-events.mjs• L1015-1164 scripts/seed-natural-events.mjs• L1167-1303 scripts/seed-fire-detections.mjs• L86-119 scripts/seed-fire-detections.mjs• L161-212
Orbital catalog datasets
The orbital catalog supplies a stable set of profiled satellites and recent orbit-element history for pass calculations. It reads the active CelesTrak catalog, keeps only satellites represented in the sensor inventory, and retains one current element set plus a daily ring for the prior seven days.
The catalog chooses the newest usable element set for each satellite identity. A refusal, unreadable response, or empty match blocks publication instead of replacing the prior catalog with an empty one. The ring can outlive a source outage, while the seed's fetch time and activation marker show whether the current deployment has received usable live data.
References: scripts/seed-orbital-catalog.mjs• L20-101 scripts/_orbital-catalog.mjs• L1-63 scripts/…/sensor-profiles.js• L1-20
Orbital site datasets
Orbital site data turns satellite archive scenes into evidence for a reviewed set of watched sites and country rollups. It resumes across sites with a cursor, keeps records within a bounded age window, and can add change scoring and site alerts after the observation pass.
The site pass depends on the orbital catalog and publishes site records, country rollups, flags, backoff state, and the index as one ordered release. Archive refusals trigger backoff rather than repeated requests. A partial run keeps evaluated sites and resumes later, while a missing catalog preserves the last accepted release. Change alerts remain in shadow mode until the review rule is ready, and alert failures do not fail the dataset publication.
References: scripts/seed-orbital-sites.mjs• L91-132 scripts/seed-orbital-sites.mjs• L215-269 scripts/seed-orbital-sites.mjs• L271-348 scripts/seed-orbital-sites.mjs• L351-391
Grid disruption clock datasets
The grid disruption clock turns outage counts and severe warning feeds into a time-aware view for a fixed set of monitored US regions. It combines MassOutage impact points with regional weather warnings and selected grid emergency notices, while keeping source health beside each region.
The dataset keeps current observations, a one-day history, and recent warnings so downstream views can distinguish disruption, recovery, watch, monitoring, stale data, and unavailable data. Source timestamps must be recent and unambiguous. Partial or stale regions publish with degraded freshness state, while activation waits until at least one live input is available; an unavailable source is never treated as zero impact.
This diagram shows how source reports become a regional disruption state.
flowchart LR
A[Outage and warning sources] --> B[Time and coverage checks]
B --> C[Regional current values and warnings]
C --> D[One-day history]
C --> E[Disruption clock state]
B --> F[Source health and degraded state]
References: scripts/seed-grid-disruption-clock.mjs• L7-101 scripts/…/grid-clock-sources.mjs• L4-170 scripts/…/grid-disruption-clock.js• L1-121
Reservoir and lake level datasets
Reservoir and lake level seeding turns delayed ICESat-2 satellite passes into a watchlist view of strategic water levels and recent change. It starts from a committed baseline, keeps per-pass state across runs, and discovers later-published granules so late source releases can update the correct observation days.
NASA CMR discovery and SlideRule segment fetches run within bounded work budgets. A valid run recomputes the watchlist and publishes only when most entries have a latest level. Discovery failures remain visible as partial coverage, while upstream outages preserve the last accepted dataset. The result is a water-surface height, not a storage-volume estimate; source delay and revisit pattern are data-age facts rather than evidence of a failed daily refresh.
References: scripts/seed-bundle-climate.mjs• L4-14 scripts/seed-reservoir-levels.mjs• L47-107 scripts/seed-reservoir-levels.mjs• L353-539
Resilience score datasets
Resilience score seeding keeps country score caches, a global ranking, and score intervals aligned with the active scoring model. It uses the reviewed country universe and the live runtime manifest before it reads or writes score data, so a stale formula or construct state cannot create a mixed generation.
The refresh warms missing country scores through the ranking service, retries lagging countries within a fixed budget, and then refreshes the ranking on every scheduled run. It reads the final score cohort again before building intervals. Interval publication requires broad country coverage and a probe country; otherwise the prior generation remains in place.
References: scripts/seed-resilience-scores.mjs• L81-105 scripts/seed-resilience-scores.mjs• L500-590 scripts/seed-resilience-scores.mjs• L614-646 scripts/seed-resilience-scores.mjs• L670-763
Security advisory datasets
Security advisory ingestion combines government travel registers with health and embassy notices. It builds a complete country-level advisory index from the full register response, then keeps a smaller recent list for general news views.
The country index must meet a coverage floor before publication. A failed or truncated travel register therefore preserves the prior global index instead of turning every country into an empty advisory state. Register-level records remain available for the dedicated advisory map, while the dashboard bootstrap omits that larger detail set.
References: scripts/seed-security-advisories.mjs• L8-25 scripts/seed-security-advisories.mjs• L510-593 scripts/seed-security-advisories.mjs• L595-668 scripts/seed-security-advisories.mjs• L674-696
Transit alert datasets
Transit alert seeding turns the official TTC GTFS-Realtime service feed into a stable alert snapshot. The combined feed is the normal path, while per-mode feeds provide a bounded fallback when one malformed alert breaks the combined response.
The fallback merges alert identities and records missing modes in freshness metadata. A complete empty alert list is valid when the feed itself is valid, but a failed combined feed with no usable mode feed is a refresh failure. The member runs inside the Canada bundle and keeps its five-minute cadence without becoming a separate service.
References: scripts/seed-ttc-alerts.mjs• L1-22 scripts/seed-ttc-alerts.mjs• L33-102 scripts/seed-ttc-alerts.mjs• L108-197
Notification projections
Notification projections consume accepted monitoring data after ingestion. The event-trigger registry defines the shared vocabulary for threshold and occurrence subscriptions, while delivery state remains separate from the canonical monitoring datasets.
The registry is a cross-surface contract. A change to an event name, source key, scope, delivery identity, or signature rule must stay aligned across registration, evaluation, API descriptions, and the sender. For the complete condition-to-message flow, continue to Alerts and notification delivery.
References: scripts/…/event-triggers.json• L1-27 scripts/…/event-triggers.json• L30-43 scripts/…/event-triggers.json• L242-320
Change checklist
When these files or sections are changed, remember to consider these:
- A canonical payload change must update its validator, record declaration, schema version, envelope readers, compression support, and every derived reader of the key.
- A bundle member change must fit its timeout and bundle budget while keeping its interval, required settings, dependency order, freshness key, and completion marker aligned.
- A source cadence change must keep transport freshness, content age, TTL, and health thresholds separate. A longer TTL must not make old observations appear current.
- A coverage rule must preserve the difference between a source failure, a valid partial result, an explicit no-content answer, and a true empty dataset.
- A new companion key, cursor, activation marker, or post-publish hook must advance only after the related canonical and secondary writes are complete.
- Shared runner changes must keep empty extra-key preservation, pre-publish field stripping, lock fencing, retry budgets, and bundle cleanup behavior aligned.
- The fetch deadline must leave enough bundle time for publication, lock release, and last-good preservation. A source timeout must not become a forced process kill before cleanup.
- Food-price changes must keep source-specific history, report dates, fetch budgets, content-age rules, attribution, and bundle section timeouts aligned.
- Fuel changes must keep required-source gates separate from tolerated-source omissions, must not reintroduce stale country carry-forward, and must rotate the prior snapshot only when week-over-week comparisons remain valid.
- Grocery changes must keep basket composition, basket version, learned-route migrations, unit normalization, outlier eviction, and complete-basket ranking rules aligned.
- Regional-energy changes must keep the feed registry, provider identifier, country mapping, effective-time rules, activation marker, source version, and dedicated bundle member aligned.
- Market quote changes must keep provider fallback, closed-session handling, catalog fresh-quote coverage, country-index coverage, and market-alert freshness gates aligned.
- FX fallback changes must keep dated snapshot windows, currency-per-USD inversion, minimum session coverage, and the primary Yahoo result shape aligned.
- Commodity changes must keep required alias writes coupled to canonical publication while allowing optional gold context to age independently.
- World Bank changes must keep environment prefixes, indicator date ranges, ranking coverage guards, bootstrap keys, RPC cache keys, and catalogue verification aligned.
- CVE reference changes must keep the KEV filter, full-refresh floor, incremental merge window, attribution, schema version, and downstream CVE lookup fields aligned.
- Military-flight changes must keep global versus regional coverage, live and stale fallback keys, derived theater outputs, and classification audit state aligned.
- UCDP changes must keep annual base coverage, candidate merge state, recent-window rules, annual-floor retention, and dashboard projection keys aligned.
- Unrest changes must keep ACLED display and resolution windows, GDELT freshness labels, source-version selection, deduplication, and source-link retention aligned.
- Webcam changes must keep the geo index, metadata index, active pointer, and pointer/data TTL relationship aligned.
- GDELT changes must keep output cohorts symmetric, keep cursor state behind successful outputs, and preserve archive repair as bounded and replay-safe.
- AAII changes must keep HTML date extraction, percentage validation, retained-week merging, and the three-week retention window aligned with the weekly source cadence.
- Correlation changes must keep input envelope unwrapping, source-age limits, accepted input shapes, domain thresholds, and companion card keys aligned.
- Changes to country or coordinate attribution must keep the CII seeder, scoring consumer, generated border data, news place data, and ISO mappings aligned.
- Changes to displacement sources must keep event dates separate from publication dates and must not turn missing country files into a valid zero result.
- Changes to news clustering or place extraction must keep story identity, publisher-family corroboration, country scope, and insight selection aligned.
- Changes to natural events or wildfires must keep source-specific retention, complete-coverage rules, recovery snapshots, summary keys, and public transforms aligned.
- Orbital catalog changes must keep sensor-profile matching, the inventory version, the daily element ring, source refusal handling, and activation state aligned.
- Orbital site changes must keep catalog dependency, site and country rollups, cursor progress, archive backoff, change-scoring shadow state, ordered publication, and alert isolation aligned.
- Grid clock changes must keep the monitored-region set, source-specific time rules, warning health, one-day history, degraded metadata, and live-input activation gate aligned.
- Reservoir-level changes must keep the watchlist, committed baseline, late-granule discovery window, per-product budgets, day-complete reductions, coverage guard, and activation marker aligned.
- Changes to PortWatch history, chokepoint flow windows, or baselines must keep daily history independent from the year-earlier comparison, preserve carried-entry age, and keep hazard annotations out of flow arithmetic.
- Changes to port activity recovery must keep the verified-zero rule separate from an unverified empty response and preserve the country coverage target.
- Sanctions changes must keep pressure sources separate from screening sources, source retention, entity indexes, country counts, change history, attribution health, and activation state aligned.
- Trade period changes, extended HS4 keys, or LNG origin shares must keep annual candidates, reserved world requests, request budgets, country rotation, pair-atomic writes, attribution floors, and Hormuz output state aligned.
- Malaysia SITC changes must keep the DOSM trade and index inputs, monthly section shape, release-delay content-age rule, activation marker, and trade-service reader aligned.
- OECD long-term-rate changes must keep the ECB and FRED source roles, country coverage floor, latest companion view, content-age metadata, and activation marker aligned.
- Security advisories must keep the full country index separate from the capped news list and preserve the travel-register coverage floor.
- TTC alerts must keep combined and per-mode feed shapes, missing-mode metadata, producer timestamps, and the Canada bundle member aligned.
- Resilience scores must keep the runtime formula and education state, ranking refresh, final score cohort, interval generation, and coverage floor aligned.
- Aviation changes must keep the airport registry, API bootstrap aggregate, source-specific coverage states, notification previous-state keys, severity bands, and country or coordinate attribution aligned.
- Crisis editorial changes must keep reviewed dates, source URLs, actor and timeline identity, watch indicators, and tracked dataset names aligned.
- Changes to the generated Sentry probe filter file must come from its Convex export generator and must retain the rule that deploy-skew errors remain visible.
- Changes to scripts/…/event-triggers.json must keep webhook registration, API descriptions, delivery evaluation, and integration consumers on the same event and signature contract.
- Changes to generated product facts or catalog data must preserve the shared plan keys and limits used by public product and checkout surfaces.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| A source returns too few records or cohorts | A thin snapshot replaces useful last-good data or creates a false signal. | Domain coverage floors, cohort checks, and guarded publication. |
| A recent fetch contains old observations | Health reports fresh transport while readers see stale content. | Separate content-age metadata and source-specific age limits. |
| A bundle lease or turn cannot be claimed | The run may overlap or the scheduler may repeat one member unfairly. | Lease-based claim, deferred tick, and turn advance only after terminal completion. |
| A macro section cannot fit its remaining bundle budget | The runner kills an in-flight member before it can publish or preserve data. | Admission checks, fetch-deadline clamping, and a reserved cleanup window. |
| A run loses its seed lock before publication | An older run can overwrite a newer dataset or freshness record. | Heartbeat renewal and Redis-fenced canonical and metadata writes. |
| A required companion write fails after a commodity publish | Alias readers see stale or missing data while the canonical seed looks fresh. | Await required companion writes in the publish lifecycle; keep optional gold context independent. |
| A food report regresses or fails during merge | A dated price history loses its newest valid observations. | Strict prior read, date regression check, and last-good retention. |
| AAII changes its page shape or reports an old week | The sentiment panel can replace a current history with an unusable or stale reading. | Strict source-date parsing, current-week validation, percentage checks, and last-good retention. |
| A grocery offer has the wrong product, unit, or scale | A search result or sub-unit price distorts a country basket and its ranking. | Structured offer evidence, unit checks, currency bounds, cross-country outlier checks, and route eviction. |
| A required fuel source fails | A partial country set is presented as a complete global comparison. | Required-country coverage gate and last healthy canonical snapshot. |
| AviationStack omits a hub or ICAO returns a quota or non-JSON response | The bootstrap can show a normal airport or empty closure list for missing coverage. | Source coverage states, unknown rows, and last-good retention for invalid NOTAM responses. |
| Comtrade quota or request budget ends a bilateral run early | The same country tail can remain old or the core and extended product views can drift. | Reserved requests, refusal breaker, rotated country order, paired-key preservation, and partial coverage metadata. |
| DOSM publishes late or one SITC CSV is malformed | The Malaysia trade view appears fresh without a usable month or index set. | Bounded CSV size, payload validation, release-delay content age, and last-good publication. |
| A market is closed and country index extension is incomplete | Health says the global quote set is fresh while an auxiliary country cache expires. | Multi-market close-day gate, confirmed canonical extension, and separate country-index freshness. |
| An orbital archive refuses searches | Repeated retries waste the archive budget and can replace the site index with an incomplete release. | Refusal backoff, cursor resume, ordered record-before-index publication, and last-good preservation. |
| A grid source is unavailable or its timestamp is old | Missing impact can be reported as zero or as a current regional state. | Explicit null observations, source health, strict time checks, and degraded metadata. |
| A digest story-track chunk fails or a live-programming item is retained | A digest either silently loses stories or sends stale, non-event content. | Bail-on-partial-read, read-time age filter, opinion and ephemeral-live filters, and retry by leaving the slot unsent. |
| A GDELT GKG or export cohort is missing, old, or not symmetric | Topic and event products can mix source clocks or report false calm. | Current-cohort checks, contiguous timestamps, coverage state, and cursor replay. |
| A sanctions source fails or loses country or registry attribution | Screening or country-risk views can look complete while their source coverage is incomplete. | Per-source retained snapshots, source health, attribution thresholds, and degraded freshness metadata. |
| The generated Convex probe filter hides a deploy-skew error | A broken internal export can look like expected probe noise. | Generated-source rule that omits public function names and preserves deploy-skew messages. |
Key files
| File | Why to read it |
|---|---|
| scripts/_seed-utils.mjs | Defines shared locks, fetch limits, publication, retry, TTL, freshness, and failure behavior. |
| scripts/_seed-contract.mjs | Defines the seeder descriptor and record-count contract. |
| scripts/_seed-payload-codec.mjs | Defines the large-payload storage format and decode boundary. |
| scripts/_seed-history.mjs | Shows best-effort history append and separate ingest health. |
| scripts/_bundle-runner.mjs | Shows freshness gates, section budgets, completion attestation, and member isolation. |
| scripts/seed-bundle-canada.mjs | Shows mixed local-source cadences, dependency order, and rollout-safe member handling. |
| scripts/seed-bundle-climate.mjs | Shows a bundle whose members use long and short climate and orbital refresh intervals. |
| scripts/seed-bundle-macro.mjs | Shows protected slots, rotating admission order, food members, and the Malaysia trade member. |
| scripts/_macro-bundle-order.mjs | Shows protected slots and rotating macro admission order. |
| scripts/_bundle-turn.mjs | Shows the leased scheduler turn used to prevent starvation. |
| scripts/seed-reservoir-levels.mjs | Shows delayed satellite observations, baseline and pass-state recovery, bounded granule work, and watchlist publication. |
| scripts/_currency-api.mjs | Shows the dated secondary FX source, window coverage guard, and quote-shape conversion. |
| scripts/…/food-prices-seed.mjs | Shows source-level food history merge, content-age rules, and refresh behavior. |
| scripts/…/food-price-sources.json | Defines food source identity, cadence, coverage role, age limits, attribution, and fetch budget. |
| scripts/seed-economy.mjs | Shows the main economic view and independent energy, macro, and weekly history companions. |
| scripts/seed-fuel-prices.mjs | Shows multi-source fuel coverage, currency normalization, source tolerance, and publication gates. |
| scripts/seed-grocery-basket.mjs | Shows structured offer checks, learned routes, basket history, and partial-country handling. |
| scripts/seed-wb-indicators.mjs | Shows annual World Bank rankings, progress and renewable views, and RPC cache publication. |
| scripts/seed-trade-sitc-my.mjs | Shows Malaysia DOSM monthly SITC trade and index ingestion with content-age validation. |
| scripts/seed-comtrade-bilateral-hs4.mjs | Shows bilateral trade, world exports, request budgets, country rotation, and LNG-origin state. |
| scripts/seed-comtrade-bilateral-hs4-ext.mjs | Shows the separate extended-HS4 product pass, paired country writes, and partial coverage handling. |
| scripts/seed-hs2-chokepoint-exposure.mjs | Shows trade-based and route-based chokepoint exposure. |
| scripts/seed-chokepoint-flows.mjs | Shows PortWatch and EIA flow windows, disruption thresholds, and hazard annotations. |
| scripts/seed-market-quotes.mjs | Shows authorized-provider fallback, market-day retention, companion country indexes, and quote coverage. |
| scripts/seed-market-alert-ledger.mjs | Shows persistent market alert windows, evidence reads, and scorecards. |
| scripts/…/yahoo-closed-session.cjs | Shows how closed-session quote changes use the last completed session. |
| scripts/seed-oecd-lt-rates.mjs | Shows long-term government-rate source fallback, history, latest view, and content age. |
| scripts/seed-correlation.mjs | Shows stale-input filtering and cross-domain signal cards. |
| scripts/seed-gdelt-bulk-materializer.mjs | Shows shared-cohort validation, bounded history repair, product publication, and cursor replay. |
| scripts/seed-aaii-sentiment.mjs | Shows weekly survey parsing, current-week validation, retained history, and sentiment projections. |
| scripts/seed-cyber-threats.mjs | Shows multi-provider threat coverage, first-seen state, geo hydration, and retention. |
| scripts/seed-sanctions-pressure.mjs | Shows sanctions source retention, pressure projections, screening indexes, change history, and attribution health. |
| scripts/seed-security-advisories.mjs | Shows full-register country coverage and the bounded advisory news view. |
| scripts/seed-ucdp-events.mjs | Shows annual and candidate conflict ingestion and the compact dashboard projection. |
| scripts/seed-unrest-events.mjs | Shows ACLED and GDELT event merge, source confidence, and resolution data. |
| scripts/seed-ttc-alerts.mjs | Shows combined and per-mode GTFS-Realtime alert ingestion. |
| scripts/seed-resilience-scores.mjs | Shows score warming, ranking refresh, runtime-state checks, and interval coverage. |
| scripts/seed-forecast-resolutions.mjs | Shows durable forecast evidence, resolution, receipts, and scorecards. |
| scripts/seed-insights.mjs | Shows story clustering, corroborated synthesis, story-pool output, and last-known-good preservation. |
| scripts/seed-digest-notifications.mjs | Shows post-ingestion story filters, shared brief composition, watchlist scans, and multi-channel delivery. |
| scripts/seed-aviation.mjs | Shows the unified airport registry, source coverage, bootstrap assembly, and aviation notifications. |
| scripts/seed-military-flights.mjs | Shows aircraft source coverage, classification, stale fallback, theater views, and audit outputs. |
| scripts/seed-webcams.mjs | Shows spatial camera catalog assembly, version activation, and old-version cleanup. |
| scripts/seed-orbital-catalog.mjs | Shows CelesTrak acquisition, sensor-profile filtering, daily orbit history, and activation. |
| scripts/seed-orbital-sites.mjs | Shows watched-site archive evaluation, cursor resume, backoff, change scoring, atomic publication, and alert isolation. |
| scripts/_orbital-catalog.mjs | Defines satellite identity selection, orbit parsing, inventory versioning, and the eight-day ring. |
| scripts/seed-grid-disruption-clock.mjs | Shows regional snapshot retention, publication validation, health state, and live-input activation. |
| scripts/…/grid-clock-sources.mjs | Shows outage and warning source collection, timestamp checks, and source health. |
| scripts/…/grid-disruption-clock.js | Defines monitored regions, snapshot validation, freshness, warning trust, and disruption phases. |
| scripts/…/crisis-editorial.json | Shows the reviewed editorial artifact for situation background, actors, timelines, and watch indicators. |
| scripts/…/sentry-convex-internal-probe-filters.txt | Shows the generated operational filter artifact for known internal probe noise and deploy-skew visibility. |
| scripts/…/news-catalog.json | Defines the reviewed feed registry and placement and language rules. |
| scripts/…/event-triggers.json | Defines the event and subscription contract shared by notification surfaces. |
| scripts/…/product-catalog.generated.json | Defines generated public product and plan facts used by product surfaces. |
| scripts/…/product-facts.generated.json | Provides the generated product facts consumed by public metadata and product surfaces. |