Introducing a new way to review your PRs.

Read the announcement

koala73 / worldmonitor

Real-time global intelligence dashboard. AI-powered news aggregation, geopolitical monitoring, and infrastructure tracking in a unified situational awareness interface

View on GitHub

29 documents · Updated Nov 20, 2026

docs / consumer-prices-validation

Consumer-price validation pipeline

The consumer-price pipeline turns public food-price reports and retailer observations into data that downstream indexes, alerts, and views can use. It normalizes product and quantity facts, rejects weak or stale observations, keeps source history safe during refreshes, and exposes quality signals with the accepted data. See the data-ingestion parent page for the wider ingestion model. The seed-pipeline-orchestration and alerts-and-notification-delivery pages describe nearby downstream flows.

Pipeline boundary

System purpose

The consumer-price pipeline moves data through four quality boundaries: source collection, observation admission, cross-retailer review, and publication. Each boundary can reduce coverage instead of passing uncertain data forward. This keeps a small data set visible as partial rather than turning a bad price or stale report into an index value.

The main jobs and source steps run in order:

  1. Source collectors read public reports and return dated price observations or dated context.
  2. scrape finds retailer pages and stores product matches and observations.
  3. validate checks accepted prices against fresh peer retailers.
  4. aggregate computes basket measures from usable matches.
  5. publish writes compact snapshots for WorldMonitor readers.

This diagram shows how source reports and retailer observations reach downstream consumers.

flowchart LR
    A[Public price report] --> B[Source adapter]
    B --> C[Normalize and date-check]
    C --> D[Retain last-good snapshot]
    E[Retailer page] --> F[Normalize product and size]
    F --> G[Admit observation]
    G --> H[Cross-retailer price check]
    D --> I[Consumer-price data]
    H --> I
    I --> J[Indexes, alerts, and panel views]

Quality boundary

The pipeline treats price quality as a publication decision, not only as a fetch result. A report can return rows and still fail because its schema, unit, date, currency, product identity, or context meaning changed. A retailer page can return a numeric price and still fail because the page names another product, uses another currency, lacks proof of the price, or falls outside the requested size range.

References: scripts/…/food-prices-fetch.mjs• L243-346 consumer-prices-core/…/scrape.ts• L201-255 consumer-prices-core/…/validate.ts• L130-182 consumer-prices-core/…/aggregate.ts• L52-65

Source collection

Each configured source has one adapter that converts its publisher format into a common report. The source registry supplies cadence, context-only status, freshness limits, fetch budget, and attribution. The adapter registry links source IDs to the correct collector, including country-specific FEWS feeds and separate Swiss FOAG cubes.

The common report shape separates market observations from context. Context-only sources can publish dated food CPI or benchmark information without pretending that an index or survey value is a market price. Observation sources retain market, market type, unit, currency, date, price, and origin so later consumers can compare like-for-like series. FEWS country feeds keep the local currency and market identity for the latest public monthly data, while MakStat supplies food CPI as dated context rather than market prices.

References: scripts/…/food-price-sources.json• L1-12 scripts/…/food-price-sources.json• L259-334 scripts/…/food-price-sources.json• L369-466 scripts/…/index.mjs• L15-29 scripts/…/fews.mjs• L8-24 scripts/…/makstat.mjs• L6-16 scripts/…/geostat.mjs• L7-23 scripts/…/foag.mjs• L18-41

Report normalization

Adapters reject structural changes instead of guessing how to read them. Shared helpers check required CSV columns, positive values, valid months, bounded cube dimensions, and non-future dates. A refreshed report replaces withdrawn quotes for its own dates while dates outside the requested report remain in history.

The source-specific rules preserve distinctions that affect meaning. Examples include pack units in national retail tables, market and source identity in FEWS data, production and origin qualifiers in FOAG series, modal wholesale quotes in Argentina, and premise medians in Malaysia. CPI and annual benchmark sources remain context because their values are indexes or survey measures rather than direct market prices. The MakStat adapter also checks that the selected cube labels still mean food CPI month-on-month and year-on-year changes before it publishes context.

References: scripts/…/monthly.mjs• L4-25 scripts/…/monthly.mjs• L28-49 scripts/…/fews.mjs• L8-24 scripts/…/makstat.mjs• L7-16 scripts/…/foag.mjs• L28-41 scripts/…/pricecatcher.mjs• L9-35 scripts/…/mcba.mjs• L11-33 scripts/…/food-prices.mjs• L27-78 scripts/…/food-prices.mjs• L114-195 scripts/…/food-prices.mjs• L221-255 scripts/…/food-prices.mjs• L272-309

Fetch boundary

Collection limits protect the job from publisher failures that could consume unbounded time or memory. Requests stay on the configured publisher hosts and protocols, follow only a small number of redirects, enforce response-size limits, and use the source fetch budget. PDF, workbook, archive, and Parquet readers apply their own page, sheet, expansion, row, and output limits.

Sources that cannot be reached reliably from the job environment can use the shared proxy route. The collector bounds proxy exits, preserves the same host and size checks, and tries a direct request only when the proxy did not reach the publisher. FEWS uses this route for its country feeds, and MakStat uses it for its statistical cube.

The collector also checks that observations and context dates are valid and no later than the collection date. A context-only source is accepted when it has valid context even when it has no observations. A normal source must return at least one dated observation.

References: scripts/…/food-prices-fetch.mjs• L22-72 scripts/…/food-prices-fetch.mjs• L108-127 scripts/…/food-prices-fetch.mjs• L129-174 scripts/…/food-prices-fetch.mjs• L194-259 scripts/…/food-prices-fetch.mjs• L262-365 scripts/…/food-prices.mjs• L1-22

Product normalization

Product identity

Normalization gives retailer data a stable form for matching. Titles are cleaned before token comparison, promotion words are removed, and brand aliases map common retailer spellings to one name. This reduces false differences between a canonical basket item and a retailer listing without changing the source price.

The canonical matcher combines identity and size evidence. Brand and category agreement raise confidence. Title overlap and size agreement provide further evidence. A strong result can become auto, a weaker result can enter review, and a clearly wrong result is not selected.

References: consumer-prices-core/…/title.ts• L9-36 consumer-prices-core/…/brand.ts• L30-46 consumer-prices-core/…/canonical.ts• L28-93

Quantity normalization

Size normalization converts pack sizes and common mass, volume, and count units into base quantities. It also handles local decimal and thousands separators. The result supports both unit-price calculation and a quantity window for a basket item. Unknown units return no size instead of inventing a quantity.

The validator treats content units differently from count units. It can accept a plausible mass-versus-volume match when the quantity could describe the same product, but it rejects a quantity that cannot fit the configured window.

References: consumer-prices-core/…/size.ts• L6-13 consumer-prices-core/…/size.ts• L129-192 consumer-prices-core/…/validator.ts• L210-250

Observation admission

Search-hit validation

Retailer search uses two bounded stages: it discovers candidate product URLs and then extracts a price and product facts from each allowed page. Stored pins and configured seed pages are tried first. If they fail, the flow can use normal discovery so one broken page does not leave the basket item uncovered. Host, path, and market-scope rules apply to pins, seeds, and discovered URLs.

An accepted candidate needs a positive finite price, a product name, the configured currency, and evidence that the price appears in the page content when rendered content is available. The validator checks identity tokens, negative product classes, non-food terms, and the quantity window. It returns a score from 0 to 1 as well as reasons and signals. The normal automatic admission threshold is 0.75.

Strict retailers reject failed validation. Other retailers can keep a failed result in shadow mode for diagnosis, while a weak but usable result becomes a candidate rather than entering an index. A direct pin receives the validator result again during scrape, so a page that has drifted to another product can be rejected or downgraded even when the pin itself was previously trusted.

Provider errors use bounded cooldowns and a single wider discovery retry. This limits paid calls during an outage while allowing a temporary empty or poorly ranked search result to recover. A failed provider path can still fall back to a valid seed candidate.

References: consumer-prices-core/…/validator.ts• L44-68 consumer-prices-core/…/validator.ts• L253-318 consumer-prices-core/…/search.ts• L197-203 consumer-prices-core/…/search.ts• L439-687 consumer-prices-core/…/search.ts• L729-999

Observation records

An admitted observation keeps the source payload with its price, currency, stock state, normalized size, validator evidence, and scrape run. Consecutive identical payloads are reused rather than added as new observations. This preserves a clean history while the retailer product's last-seen time still records that the page was checked.

The validator result is also available as stored scalar fields. Missing validator data keeps the legacy admission rule. Malformed validator data becomes unknown and fails the admission expression. This lets the database apply the same boundary during later validation without reading the full audit payload.

Match changes and observation writes share a transaction when a new match is admitted or a direct match is demoted. This prevents aggregation from seeing new evidence with an old match state, or an updated match state with an older observation.

References: consumer-prices-core/…/search.ts• L1002-1049 consumer-prices-core/…/observations.ts• L4-62 consumer-prices-core/…/012_observation_validator_columns.sql• L1-21 consumer-prices-core/…/scrape.ts• L214-359

Match states

Match state controls whether a product may affect a basket. auto and approved are usable states. candidate preserves a weak match for later work but does not publish it. review and rejected keep uncertain or blocked matches out of the aggregate.

Scrape updates do not overwrite a human approval, review hold, or rejection. A direct pin that later receives a weak validator result can move from auto to candidate; a human-approved match is not demoted by that machine path. This separation prevents a routine scrape from undoing curation.

References: consumer-prices-core/…/matches.ts• L9-23 consumer-prices-core/…/matches.ts• L25-83 consumer-prices-core/…/008_candidate_match_status.sql• L1-10

Scrape run accounting

The scrape job records page attempts, successes, errors, validator rejections, and classified failure reasons. A retailer run can finish as partial when its time budget stops the target loop. A match-admission persistence failure is stronger: it fails the run and prevents the health timestamp from claiming a successful publication boundary.

The job processes retailers in bounded parallel workers and keeps each retailer's target loop bounded. It writes health before the terminal run status, so resume logic cannot treat an unfinished health update as a completed cycle.

References: consumer-prices-core/…/scrape.ts• L163-194 consumer-prices-core/…/scrape.ts• L381-449 consumer-prices-core/…/scrape.ts• L454-479

Snapshot retention

Last-good merge

The seed job combines a new source report with the prior snapshot before publication. It refuses an empty incoming observation set when the prior snapshot has prices, refuses a date regression, and keeps older country context when a fetch fails or moves backward. This prevents a temporary publisher error from erasing usable downstream data.

Each source uses its configured fetch and content age limits. A successful publish records the source as activated. A failed first fetch records health without creating a false successful data receipt, while the seed lock and fetch margin allow the prior snapshot read, merge, cleanup, and freshness writes to finish after collection.

References: scripts/…/food-prices-seed.mjs• L9-31 scripts/…/food-prices-seed.mjs• L34-63 scripts/…/food-price-sources.json• L3-12 scripts/…/food-price-sources.json• L382-392

Downstream handoff

The publish job converts computed data into compact snapshots for overviews, movers, freshness, categories, retailer spreads, basket series, and coverage. It writes these snapshots to Redis with a TTL longer than the normal daily schedule. The payload also carries seed metadata for record count and scrape coverage.

Freshness controls whether a publication advances the seed clock. A stale scrape may replace a payload after a successful aggregate, but it retains the last fresh publication time. A failed snapshot build leaves the last good key in place and fails the job so the scheduler can report the fault.

The consumer-price panel reads overview, category, mover, retailer-spread, and freshness snapshots together. It shows a global market view only from market-independent data, and it loads the World inflation view from a separate IMF source. Upstream-unavailable responses produce a collection-in-progress state instead of repeated empty tab messages.

References: consumer-prices-core/…/publish.ts• L55-79 consumer-prices-core/…/publish.ts• L161-195 consumer-prices-core/…/publish.ts• L201-323 src/…/ConsumerPricesPanel.ts• L241-295 src/…/ConsumerPricesPanel.ts• L298-419

Manual seed fallback

home/…/seed-consumer-prices.mjs is a manual recovery path that reads the service API and writes short-lived placeholders or snapshots. It is not an authoritative scheduled writer. Running it as a cron would compete with the core publisher and replace its longer TTLs.

References: scripts/seed-consumer-prices.mjs• L1-17 scripts/seed-consumer-prices.mjs• L126-154 scripts/seed-consumer-prices.mjs• L266-282

Cross-retailer validation

Freshness boundary

Validation and aggregation use only products that the scrape job saw within the freshness window. The window is fourteen days. It uses the retailer product's last-seen time because an unchanged payload can reuse an older observation row while a new scrape still confirms the current price.

The latest observation must also be in stock before it can provide a priced peer. An out-of-stock latest observation can still prove that a retailer was recently seen, which matters when deciding whether a review hold is safe to release.

References: consumer-prices-core/…/observation-freshness.ts• L1-17 consumer-prices-core/…/validate.ts• L61-80 consumer-prices-core/…/aggregate.ts• L43-65

Price sanity checks

The validation job groups usable matches by basket item and gives each retailer one median price vote. It needs at least two retailers to compare prices. If two peers strongly disagree, it holds both sides instead of choosing one by majority. Otherwise it holds a price above 2.5x the peer median or below 0.35x the median.

The job writes evidence for each hold and preserves the owner of a human approval. It restores only holds created by the validation job. Restoration requires an admitted latest observation and either a fresh peer showing a normal range or no fresh peer left to support the old dispute.

References: consumer-prices-core/…/validate.ts• L24-33 consumer-prices-core/…/validate.ts• L130-212

Pin recovery

Pinned retailer pages improve repeat coverage, but repeated out-of-stock results or fetch errors can disable a pin. The scrape job probes disabled pins in bounded batches and restores a pin after three consecutive in-stock observations. An out-of-stock result resets the recovery streak, so intermittent recovery cannot silently re-enable a bad page.

The recovery migration clears old sticky disable state and its trigger counters once. Later scrapes can then disable genuinely bad pages again from current evidence. Recovery probes remain excluded from aggregate use until the disable marker is cleared.

References: consumer-prices-core/…/matches.ts• L137-167 consumer-prices-core/…/scrape.ts• L132-155 consumer-prices-core/…/scrape.ts• L361-378 consumer-prices-core/…/009_pin_auto_recovery.sql• L18-37

Basket aggregation

Index inputs

Aggregation reads only active basket items with auto or approved matches, fresh retailer products, and an in-stock latest observation. It measures each item against the first day on or after the basket base date when that item has data. If no item has a baseline, it does not publish a false index of 100.

The fixed basket index uses the average available price per item. The value basket index uses the cheapest available price. Both use basket weights. Coverage is written beside the index so downstream readers can tell a full result from a thin one.

References: consumer-prices-core/…/aggregate.ts• L81-163 consumer-prices-core/…/aggregate.ts• L218-249 consumer-prices-core/…/aggregate.test.ts• L36-75

Empty coverage

An empty fresh basket is a valid data-quality result, not a reason to keep showing an older price level as current. Aggregation clears that day's prior computed rows and writes coverage_pct = 0 in one transaction. This makes missing coverage visible to snapshot builders and health checks.

References: consumer-prices-core/…/aggregate.ts• L218-235 consumer-prices-core/…/validate.test.ts• L107-122

Schema evolution

Migration chain

The schema uses forward-only SQL migrations. The runner records each applied file in schema_migrations, runs files in sorted order, and wraps each migration in a transaction. A failed migration rolls back before the process stops.

The important quality changes are:

MigrationContract added
001_initial.sqlRetailers, products, observations, matches, baskets, indices, and run health.
007_pinning_columns.sqlSoft pin disable and failure counters without deleting match history.
008_candidate_match_status.sqlA non-publishing state for weak matches.
009_pin_auto_recovery.sqlRecovery streaks and the one-time reset of sticky pin state.
010_scrape_run_coverage.sqlA non-negative count of validator-rejected pages.
011_scrape_run_failure_reasons.sqlA bounded diagnostic object for terminal scrape failures.
012_observation_validator_columns.sqlStored validator results used by later admission checks.

References: consumer-prices-core/…/migrate.ts• L1-57 consumer-prices-core/…/001_initial.sql• L69-120 consumer-prices-core/…/010_scrape_run_coverage.sql• L1-12 consumer-prices-core/…/011_scrape_run_failure_reasons.sql• L1-25

Migration contracts

Migration 009 is more than a column change. Its one-time data reset must run only once because a repeat would erase current failure counters. Migration 012 must treat old, missing, and malformed validator payloads differently so a table rewrite does not fail and malformed data does not pass the admission gate.

The SQL contract test checks both backfill and later inserts and updates. This protects the link between raw audit payloads and the scalar fields read by the validation job.

References: consumer-prices-core/…/009_pin_auto_recovery.sql• L39-61 consumer-prices-core/…/009_pin_auto_recovery.sql• L63-98 consumer-prices-core/…/observation-validator-columns.sql• L1-51

Job test contracts

Validation contracts

The validation tests protect the conditions that can silently change what reaches an index. They require one peer vote per retailer, hold both sides of a two-retailer dispute, preserve human review ownership, and restore only validation-owned holds. They also assert that stale observations do not re-enter the peer set.

References: consumer-prices-core/…/validate.test.ts• L20-87 consumer-prices-core/…/validate.test.ts• L88-151

Scrape contracts

The scrape tests protect the symmetric pin lifecycle. Three consecutive out-of-stock observations disable a pin. Three consecutive in-stock observations clear the disable marker. Each side resets the other streak, which prevents intermittent results from creating a false recovery. The contract also caps the recovery counter and makes the clear action safe to repeat.

References: consumer-prices-core/…/scrape.test.ts• L31-86 consumer-prices-core/…/scrape.test.ts• L88-138

Aggregate contracts

The aggregate tests protect the meaning of the published index. They require first-observation baselines, reject a fabricated 100 when no baseline exists, and keep the methodology version on computed rows. The validation tests also cover zero-coverage replacement as a transaction.

References: consumer-prices-core/…/aggregate.test.ts• L36-75 consumer-prices-core/…/validate.test.ts• L107-122

Publication contracts

The publish tests protect the handoff to Redis. They check coverage diagnostics, per-range snapshot keys, retained freshness metadata, skipped all-gated movers, last-good behavior after a build failure, and failure signaling when Redis does not confirm a write.

References: consumer-prices-core/…/publish.test.ts• L67-160 consumer-prices-core/…/publish.test.ts• L162-179 consumer-prices-core/…/publish.test.ts• L256-258

Change checklist

When these files or sections are changed, remember to consider these:

  • Keep source IDs aligned across food-price-sources.json, the adapter registry, and the per-source seed entry point.
  • Keep contextOnly aligned with whether an adapter returns observations or only dated context. Context must not enter market-price aggregates.
  • Keep report replacement dates aligned with mergeHistory so withdrawn quotes are removed without deleting unrequested history.
  • Keep units, currencies, market types, and series qualifiers intact when changing an adapter. A collapsed dimension can create false comparisons.
  • Keep source fetch and content-age limits aligned with the source cadence and with the panel's freshness display.
  • Keep proxy-required sources aligned with proxy resolution, bounded exits, and the direct-fallback rule. A source that works locally can still fail in the job environment.
  • Keep last-good retention and future-date checks in place when changing report parsing or merge behavior.
  • Keep the validator score threshold aligned across validator.ts, match admission, generated observation columns, and the validation query.
  • Keep page host rules, optional path filters, and required path segments aligned for pins, seeds, and discovered URLs. A host can serve more than one market.
  • Keep price-evidence handling aligned with provider output. A missing rendered page must remain visible as unverified rather than silently becoming proven evidence.
  • Keep candidate excluded from every aggregate and snapshot path while retaining its evidence for later review.
  • Keep the fourteen-day last_seen_at rule consistent between validation, aggregation, and freshness reporting.
  • Keep scrape failure reasons, rejected counts, terminal status, and health updates aligned when changing the target loop. A run that cannot persist admission must not refresh health.
  • When changing the Redis envelope or seed metadata, update the publisher, the seed merge path, and the panel response contract together.
  • Treat pin-disable and pin-recovery thresholds as a pair. Changing one without the other can make coverage decay or re-enable bad pages.

Important failure modes

TriggerConsequenceGuard
A publisher changes a column, cube dimension, unit, or labelThe source can publish misread prices or silently lose coverageRequired-field checks, bounded cube decoding, and adapter-specific format errors
A proxy exit cannot reach a source from the job environmentA valid source appears unavailable or misses its freshness limitBounded proxy retries, per-exit time sharing, and direct fallback only after no proxy exit reaches the publisher
A context-only CPI or survey value is treated as a market priceDownstream comparisons mix unlike measurescontextOnly source metadata and separate contexts handling
A source returns an empty or older report after a successful runCurrent downstream data is erased or moves backwardLast-good merge rejects missing observations and date regression
A search result points to another host, market path, or productA foreign or wrong-product price can enter a retailer matchHost allowlist, path policy, page-title checks, and validator gates
A provider returns a made-up price or no rendered contentAn unproven value can look like a verified observationSame-call price evidence, explicit no-content logging, and bounded provider fallback
A weak or malformed validator result is admittedA wrong product can enter a match or indexStrict validation, the 0.75 admission threshold, and IS TRUE on stored validator fields
A pinned page drifts to another productA trusted URL can write a wrong price repeatedlyDirect-hit validator enforcement and automatic demotion only for machine-owned auto matches
A two-retailer price disagreement is resolved by choosing one sideThe index can use a bad price while hiding the conflictThe validation job holds both sides of a disputed pair
An old observation remains the latest usable priceA stale page can hold fresh peers or distort the basketThe last_seen_at freshness window and latest-observation stock check
A scrape run times out without a terminal statusResume and coverage logic can treat the market as permanently activeThe retailer run budget, partial status, and terminal update after health persistence
An empty basket keeps yesterday's computed rowsDownstream readers show an old value as currentTransactional delete plus explicit zero coverage
A snapshot write fails after a partial publishReaders can see mixed freshness or a false healthy runLast-good retention and non-zero job failure signaling

Key files

FileWhy to read it
scripts/…/food-prices-fetch.mjsShared source collection boundary, request limits, proxy routing, and final date checks
scripts/…/food-prices.mjsShared parsing, observation construction, report discovery, and bounded history merge rules
scripts/…/food-prices-seed.mjsLast-good merge, source retention, and seed health contract
scripts/…/index.mjsSource ID to adapter registration and generated country/cube adapters
scripts/…/monthly.mjsShared report, month, value, CSV, and cube normalization rules
scripts/…/fews.mjsFEWS monthly price selection and preservation of local market details
scripts/…/makstat.mjsMakStat food CPI context selection and label validation
scripts/…/food-price-sources.jsonSource cadence, freshness, context, attribution, and fetch policy
scripts/…/foag.mjsDimension-preserving consumer-price series and vocabulary checks
scripts/…/pricecatcher.mjsPremise-level grouping and median acceptance rule
scripts/…/mcba.mjsWholesale quote disambiguation and archive validation
consumer-prices-core/…/search.tsRetailer URL discovery, extraction policy, provider fallback, and candidate evidence
consumer-prices-core/…/validator.tsStructured product, class, and quantity validation contract
consumer-prices-core/…/scrape.tsObservation admission, match persistence, pin maintenance, and run accounting
consumer-prices-core/…/validate.tsCross-retailer median and review-hold behavior
consumer-prices-core/…/aggregate.tsFresh usable inputs and basket index rules
consumer-prices-core/…/publish.tsRedis snapshot handoff and freshness behavior
consumer-prices-core/…/migrate.tsMigration order and transaction contract
consumer-prices-core/…/012_observation_validator_columns.sqlDatabase-side validator admission fields
consumer-prices-core/…/scrape.test.tsPin disable and recovery guarantees
consumer-prices-core/…/validate.test.tsCross-retailer, freshness, and zero-coverage guarantees
consumer-prices-core/…/publish.test.tsSnapshot, metadata, and failure guarantees
src/…/ConsumerPricesPanel.tsDownstream market, health, and world-inflation presentation contract