Consumer-price validation pipeline
The consumer-price pipeline turns public food-price reports and retailer observations into data that downstream indexes, alerts, and views can use. It normalizes product and quantity facts, rejects weak or stale observations, keeps source history safe during refreshes, and exposes quality signals with the accepted data. See the data-ingestion parent page for the wider ingestion model. The seed-pipeline-orchestration and alerts-and-notification-delivery pages describe nearby downstream flows.
Pipeline boundary
System purpose
The consumer-price pipeline moves data through four quality boundaries: source collection, observation admission, cross-retailer review, and publication. Each boundary can reduce coverage instead of passing uncertain data forward. This keeps a small data set visible as partial rather than turning a bad price or stale report into an index value.
The main jobs and source steps run in order:
- Source collectors read public reports and return dated price observations or dated context.
scrapefinds retailer pages and stores product matches and observations.validatechecks accepted prices against fresh peer retailers.aggregatecomputes basket measures from usable matches.publishwrites compact snapshots for WorldMonitor readers.
This diagram shows how source reports and retailer observations reach downstream consumers.
flowchart LR
A[Public price report] --> B[Source adapter]
B --> C[Normalize and date-check]
C --> D[Retain last-good snapshot]
E[Retailer page] --> F[Normalize product and size]
F --> G[Admit observation]
G --> H[Cross-retailer price check]
D --> I[Consumer-price data]
H --> I
I --> J[Indexes, alerts, and panel views]
Quality boundary
The pipeline treats price quality as a publication decision, not only as a fetch result. A report can return rows and still fail because its schema, unit, date, currency, product identity, or context meaning changed. A retailer page can return a numeric price and still fail because the page names another product, uses another currency, lacks proof of the price, or falls outside the requested size range.
References: scripts/…/food-prices-fetch.mjs• L243-346 consumer-prices-core/…/scrape.ts• L201-255 consumer-prices-core/…/validate.ts• L130-182 consumer-prices-core/…/aggregate.ts• L52-65
Source collection
Each configured source has one adapter that converts its publisher format into a common report. The source registry supplies cadence, context-only status, freshness limits, fetch budget, and attribution. The adapter registry links source IDs to the correct collector, including country-specific FEWS feeds and separate Swiss FOAG cubes.
The common report shape separates market observations from context. Context-only sources can publish dated food CPI or benchmark information without pretending that an index or survey value is a market price. Observation sources retain market, market type, unit, currency, date, price, and origin so later consumers can compare like-for-like series. FEWS country feeds keep the local currency and market identity for the latest public monthly data, while MakStat supplies food CPI as dated context rather than market prices.
References: scripts/…/food-price-sources.json• L1-12 scripts/…/food-price-sources.json• L259-334 scripts/…/food-price-sources.json• L369-466 scripts/…/index.mjs• L15-29 scripts/…/fews.mjs• L8-24 scripts/…/makstat.mjs• L6-16 scripts/…/geostat.mjs• L7-23 scripts/…/foag.mjs• L18-41
Report normalization
Adapters reject structural changes instead of guessing how to read them. Shared helpers check required CSV columns, positive values, valid months, bounded cube dimensions, and non-future dates. A refreshed report replaces withdrawn quotes for its own dates while dates outside the requested report remain in history.
The source-specific rules preserve distinctions that affect meaning. Examples include pack units in national retail tables, market and source identity in FEWS data, production and origin qualifiers in FOAG series, modal wholesale quotes in Argentina, and premise medians in Malaysia. CPI and annual benchmark sources remain context because their values are indexes or survey measures rather than direct market prices. The MakStat adapter also checks that the selected cube labels still mean food CPI month-on-month and year-on-year changes before it publishes context.
References: scripts/…/monthly.mjs• L4-25 scripts/…/monthly.mjs• L28-49 scripts/…/fews.mjs• L8-24 scripts/…/makstat.mjs• L7-16 scripts/…/foag.mjs• L28-41 scripts/…/pricecatcher.mjs• L9-35 scripts/…/mcba.mjs• L11-33 scripts/…/food-prices.mjs• L27-78 scripts/…/food-prices.mjs• L114-195 scripts/…/food-prices.mjs• L221-255 scripts/…/food-prices.mjs• L272-309
Fetch boundary
Collection limits protect the job from publisher failures that could consume unbounded time or memory. Requests stay on the configured publisher hosts and protocols, follow only a small number of redirects, enforce response-size limits, and use the source fetch budget. PDF, workbook, archive, and Parquet readers apply their own page, sheet, expansion, row, and output limits.
Sources that cannot be reached reliably from the job environment can use the shared proxy route. The collector bounds proxy exits, preserves the same host and size checks, and tries a direct request only when the proxy did not reach the publisher. FEWS uses this route for its country feeds, and MakStat uses it for its statistical cube.
The collector also checks that observations and context dates are valid and no later than the collection date. A context-only source is accepted when it has valid context even when it has no observations. A normal source must return at least one dated observation.
References: scripts/…/food-prices-fetch.mjs• L22-72 scripts/…/food-prices-fetch.mjs• L108-127 scripts/…/food-prices-fetch.mjs• L129-174 scripts/…/food-prices-fetch.mjs• L194-259 scripts/…/food-prices-fetch.mjs• L262-365 scripts/…/food-prices.mjs• L1-22
Product normalization
Product identity
Normalization gives retailer data a stable form for matching. Titles are cleaned before token comparison, promotion words are removed, and brand aliases map common retailer spellings to one name. This reduces false differences between a canonical basket item and a retailer listing without changing the source price.
The canonical matcher combines identity and size evidence. Brand and category agreement raise confidence. Title overlap and size agreement provide further evidence. A strong result can become auto, a weaker result can enter review, and a clearly wrong result is not selected.
References: consumer-prices-core/…/title.ts• L9-36 consumer-prices-core/…/brand.ts• L30-46 consumer-prices-core/…/canonical.ts• L28-93
Quantity normalization
Size normalization converts pack sizes and common mass, volume, and count units into base quantities. It also handles local decimal and thousands separators. The result supports both unit-price calculation and a quantity window for a basket item. Unknown units return no size instead of inventing a quantity.
The validator treats content units differently from count units. It can accept a plausible mass-versus-volume match when the quantity could describe the same product, but it rejects a quantity that cannot fit the configured window.
References: consumer-prices-core/…/size.ts• L6-13 consumer-prices-core/…/size.ts• L129-192 consumer-prices-core/…/validator.ts• L210-250
Observation admission
Search-hit validation
Retailer search uses two bounded stages: it discovers candidate product URLs and then extracts a price and product facts from each allowed page. Stored pins and configured seed pages are tried first. If they fail, the flow can use normal discovery so one broken page does not leave the basket item uncovered. Host, path, and market-scope rules apply to pins, seeds, and discovered URLs.
An accepted candidate needs a positive finite price, a product name, the configured currency, and evidence that the price appears in the page content when rendered content is available. The validator checks identity tokens, negative product classes, non-food terms, and the quantity window. It returns a score from 0 to 1 as well as reasons and signals. The normal automatic admission threshold is 0.75.
Strict retailers reject failed validation. Other retailers can keep a failed result in shadow mode for diagnosis, while a weak but usable result becomes a candidate rather than entering an index. A direct pin receives the validator result again during scrape, so a page that has drifted to another product can be rejected or downgraded even when the pin itself was previously trusted.
Provider errors use bounded cooldowns and a single wider discovery retry. This limits paid calls during an outage while allowing a temporary empty or poorly ranked search result to recover. A failed provider path can still fall back to a valid seed candidate.
References: consumer-prices-core/…/validator.ts• L44-68 consumer-prices-core/…/validator.ts• L253-318 consumer-prices-core/…/search.ts• L197-203 consumer-prices-core/…/search.ts• L439-687 consumer-prices-core/…/search.ts• L729-999
Observation records
An admitted observation keeps the source payload with its price, currency, stock state, normalized size, validator evidence, and scrape run. Consecutive identical payloads are reused rather than added as new observations. This preserves a clean history while the retailer product's last-seen time still records that the page was checked.
The validator result is also available as stored scalar fields. Missing validator data keeps the legacy admission rule. Malformed validator data becomes unknown and fails the admission expression. This lets the database apply the same boundary during later validation without reading the full audit payload.
Match changes and observation writes share a transaction when a new match is admitted or a direct match is demoted. This prevents aggregation from seeing new evidence with an old match state, or an updated match state with an older observation.
References: consumer-prices-core/…/search.ts• L1002-1049 consumer-prices-core/…/observations.ts• L4-62 consumer-prices-core/…/012_observation_validator_columns.sql• L1-21 consumer-prices-core/…/scrape.ts• L214-359
Match states
Match state controls whether a product may affect a basket. auto and approved are usable states. candidate preserves a weak match for later work but does not publish it. review and rejected keep uncertain or blocked matches out of the aggregate.
Scrape updates do not overwrite a human approval, review hold, or rejection. A direct pin that later receives a weak validator result can move from auto to candidate; a human-approved match is not demoted by that machine path. This separation prevents a routine scrape from undoing curation.
References: consumer-prices-core/…/matches.ts• L9-23 consumer-prices-core/…/matches.ts• L25-83 consumer-prices-core/…/008_candidate_match_status.sql• L1-10
Scrape run accounting
The scrape job records page attempts, successes, errors, validator rejections, and classified failure reasons. A retailer run can finish as partial when its time budget stops the target loop. A match-admission persistence failure is stronger: it fails the run and prevents the health timestamp from claiming a successful publication boundary.
The job processes retailers in bounded parallel workers and keeps each retailer's target loop bounded. It writes health before the terminal run status, so resume logic cannot treat an unfinished health update as a completed cycle.
References: consumer-prices-core/…/scrape.ts• L163-194 consumer-prices-core/…/scrape.ts• L381-449 consumer-prices-core/…/scrape.ts• L454-479
Snapshot retention
Last-good merge
The seed job combines a new source report with the prior snapshot before publication. It refuses an empty incoming observation set when the prior snapshot has prices, refuses a date regression, and keeps older country context when a fetch fails or moves backward. This prevents a temporary publisher error from erasing usable downstream data.
Each source uses its configured fetch and content age limits. A successful publish records the source as activated. A failed first fetch records health without creating a false successful data receipt, while the seed lock and fetch margin allow the prior snapshot read, merge, cleanup, and freshness writes to finish after collection.
References: scripts/…/food-prices-seed.mjs• L9-31 scripts/…/food-prices-seed.mjs• L34-63 scripts/…/food-price-sources.json• L3-12 scripts/…/food-price-sources.json• L382-392
Downstream handoff
The publish job converts computed data into compact snapshots for overviews, movers, freshness, categories, retailer spreads, basket series, and coverage. It writes these snapshots to Redis with a TTL longer than the normal daily schedule. The payload also carries seed metadata for record count and scrape coverage.
Freshness controls whether a publication advances the seed clock. A stale scrape may replace a payload after a successful aggregate, but it retains the last fresh publication time. A failed snapshot build leaves the last good key in place and fails the job so the scheduler can report the fault.
The consumer-price panel reads overview, category, mover, retailer-spread, and freshness snapshots together. It shows a global market view only from market-independent data, and it loads the World inflation view from a separate IMF source. Upstream-unavailable responses produce a collection-in-progress state instead of repeated empty tab messages.
References: consumer-prices-core/…/publish.ts• L55-79 consumer-prices-core/…/publish.ts• L161-195 consumer-prices-core/…/publish.ts• L201-323 src/…/ConsumerPricesPanel.ts• L241-295 src/…/ConsumerPricesPanel.ts• L298-419
Manual seed fallback
home/…/seed-consumer-prices.mjs is a manual recovery path that reads the service API and writes short-lived placeholders or snapshots. It is not an authoritative scheduled writer. Running it as a cron would compete with the core publisher and replace its longer TTLs.
References: scripts/seed-consumer-prices.mjs• L1-17 scripts/seed-consumer-prices.mjs• L126-154 scripts/seed-consumer-prices.mjs• L266-282
Cross-retailer validation
Freshness boundary
Validation and aggregation use only products that the scrape job saw within the freshness window. The window is fourteen days. It uses the retailer product's last-seen time because an unchanged payload can reuse an older observation row while a new scrape still confirms the current price.
The latest observation must also be in stock before it can provide a priced peer. An out-of-stock latest observation can still prove that a retailer was recently seen, which matters when deciding whether a review hold is safe to release.
References: consumer-prices-core/…/observation-freshness.ts• L1-17 consumer-prices-core/…/validate.ts• L61-80 consumer-prices-core/…/aggregate.ts• L43-65
Price sanity checks
The validation job groups usable matches by basket item and gives each retailer one median price vote. It needs at least two retailers to compare prices. If two peers strongly disagree, it holds both sides instead of choosing one by majority. Otherwise it holds a price above 2.5x the peer median or below 0.35x the median.
The job writes evidence for each hold and preserves the owner of a human approval. It restores only holds created by the validation job. Restoration requires an admitted latest observation and either a fresh peer showing a normal range or no fresh peer left to support the old dispute.
References: consumer-prices-core/…/validate.ts• L24-33 consumer-prices-core/…/validate.ts• L130-212
Pin recovery
Pinned retailer pages improve repeat coverage, but repeated out-of-stock results or fetch errors can disable a pin. The scrape job probes disabled pins in bounded batches and restores a pin after three consecutive in-stock observations. An out-of-stock result resets the recovery streak, so intermittent recovery cannot silently re-enable a bad page.
The recovery migration clears old sticky disable state and its trigger counters once. Later scrapes can then disable genuinely bad pages again from current evidence. Recovery probes remain excluded from aggregate use until the disable marker is cleared.
References: consumer-prices-core/…/matches.ts• L137-167 consumer-prices-core/…/scrape.ts• L132-155 consumer-prices-core/…/scrape.ts• L361-378 consumer-prices-core/…/009_pin_auto_recovery.sql• L18-37
Basket aggregation
Index inputs
Aggregation reads only active basket items with auto or approved matches, fresh retailer products, and an in-stock latest observation. It measures each item against the first day on or after the basket base date when that item has data. If no item has a baseline, it does not publish a false index of 100.
The fixed basket index uses the average available price per item. The value basket index uses the cheapest available price. Both use basket weights. Coverage is written beside the index so downstream readers can tell a full result from a thin one.
References: consumer-prices-core/…/aggregate.ts• L81-163 consumer-prices-core/…/aggregate.ts• L218-249 consumer-prices-core/…/aggregate.test.ts• L36-75
Empty coverage
An empty fresh basket is a valid data-quality result, not a reason to keep showing an older price level as current. Aggregation clears that day's prior computed rows and writes coverage_pct = 0 in one transaction. This makes missing coverage visible to snapshot builders and health checks.
References: consumer-prices-core/…/aggregate.ts• L218-235 consumer-prices-core/…/validate.test.ts• L107-122
Schema evolution
Migration chain
The schema uses forward-only SQL migrations. The runner records each applied file in schema_migrations, runs files in sorted order, and wraps each migration in a transaction. A failed migration rolls back before the process stops.
The important quality changes are:
| Migration | Contract added |
|---|---|
| 001_initial.sql | Retailers, products, observations, matches, baskets, indices, and run health. |
| 007_pinning_columns.sql | Soft pin disable and failure counters without deleting match history. |
| 008_candidate_match_status.sql | A non-publishing state for weak matches. |
| 009_pin_auto_recovery.sql | Recovery streaks and the one-time reset of sticky pin state. |
| 010_scrape_run_coverage.sql | A non-negative count of validator-rejected pages. |
| 011_scrape_run_failure_reasons.sql | A bounded diagnostic object for terminal scrape failures. |
| 012_observation_validator_columns.sql | Stored validator results used by later admission checks. |
References: consumer-prices-core/…/migrate.ts• L1-57 consumer-prices-core/…/001_initial.sql• L69-120 consumer-prices-core/…/010_scrape_run_coverage.sql• L1-12 consumer-prices-core/…/011_scrape_run_failure_reasons.sql• L1-25
Migration contracts
Migration 009 is more than a column change. Its one-time data reset must run only once because a repeat would erase current failure counters. Migration 012 must treat old, missing, and malformed validator payloads differently so a table rewrite does not fail and malformed data does not pass the admission gate.
The SQL contract test checks both backfill and later inserts and updates. This protects the link between raw audit payloads and the scalar fields read by the validation job.
References: consumer-prices-core/…/009_pin_auto_recovery.sql• L39-61 consumer-prices-core/…/009_pin_auto_recovery.sql• L63-98 consumer-prices-core/…/observation-validator-columns.sql• L1-51
Job test contracts
Validation contracts
The validation tests protect the conditions that can silently change what reaches an index. They require one peer vote per retailer, hold both sides of a two-retailer dispute, preserve human review ownership, and restore only validation-owned holds. They also assert that stale observations do not re-enter the peer set.
References: consumer-prices-core/…/validate.test.ts• L20-87 consumer-prices-core/…/validate.test.ts• L88-151
Scrape contracts
The scrape tests protect the symmetric pin lifecycle. Three consecutive out-of-stock observations disable a pin. Three consecutive in-stock observations clear the disable marker. Each side resets the other streak, which prevents intermittent results from creating a false recovery. The contract also caps the recovery counter and makes the clear action safe to repeat.
References: consumer-prices-core/…/scrape.test.ts• L31-86 consumer-prices-core/…/scrape.test.ts• L88-138
Aggregate contracts
The aggregate tests protect the meaning of the published index. They require first-observation baselines, reject a fabricated 100 when no baseline exists, and keep the methodology version on computed rows. The validation tests also cover zero-coverage replacement as a transaction.
References: consumer-prices-core/…/aggregate.test.ts• L36-75 consumer-prices-core/…/validate.test.ts• L107-122
Publication contracts
The publish tests protect the handoff to Redis. They check coverage diagnostics, per-range snapshot keys, retained freshness metadata, skipped all-gated movers, last-good behavior after a build failure, and failure signaling when Redis does not confirm a write.
References: consumer-prices-core/…/publish.test.ts• L67-160 consumer-prices-core/…/publish.test.ts• L162-179 consumer-prices-core/…/publish.test.ts• L256-258
Change checklist
When these files or sections are changed, remember to consider these:
- Keep source IDs aligned across food-price-sources.json, the adapter registry, and the per-source seed entry point.
- Keep
contextOnlyaligned with whether an adapter returns observations or only dated context. Context must not enter market-price aggregates. - Keep report replacement dates aligned with
mergeHistoryso withdrawn quotes are removed without deleting unrequested history. - Keep units, currencies, market types, and series qualifiers intact when changing an adapter. A collapsed dimension can create false comparisons.
- Keep source fetch and content-age limits aligned with the source cadence and with the panel's freshness display.
- Keep proxy-required sources aligned with proxy resolution, bounded exits, and the direct-fallback rule. A source that works locally can still fail in the job environment.
- Keep last-good retention and future-date checks in place when changing report parsing or merge behavior.
- Keep the validator score threshold aligned across validator.ts, match admission, generated observation columns, and the validation query.
- Keep page host rules, optional path filters, and required path segments aligned for pins, seeds, and discovered URLs. A host can serve more than one market.
- Keep price-evidence handling aligned with provider output. A missing rendered page must remain visible as unverified rather than silently becoming proven evidence.
- Keep
candidateexcluded from every aggregate and snapshot path while retaining its evidence for later review. - Keep the fourteen-day
last_seen_atrule consistent between validation, aggregation, and freshness reporting. - Keep scrape failure reasons, rejected counts, terminal status, and health updates aligned when changing the target loop. A run that cannot persist admission must not refresh health.
- When changing the Redis envelope or seed metadata, update the publisher, the seed merge path, and the panel response contract together.
- Treat pin-disable and pin-recovery thresholds as a pair. Changing one without the other can make coverage decay or re-enable bad pages.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| A publisher changes a column, cube dimension, unit, or label | The source can publish misread prices or silently lose coverage | Required-field checks, bounded cube decoding, and adapter-specific format errors |
| A proxy exit cannot reach a source from the job environment | A valid source appears unavailable or misses its freshness limit | Bounded proxy retries, per-exit time sharing, and direct fallback only after no proxy exit reaches the publisher |
| A context-only CPI or survey value is treated as a market price | Downstream comparisons mix unlike measures | contextOnly source metadata and separate contexts handling |
| A source returns an empty or older report after a successful run | Current downstream data is erased or moves backward | Last-good merge rejects missing observations and date regression |
| A search result points to another host, market path, or product | A foreign or wrong-product price can enter a retailer match | Host allowlist, path policy, page-title checks, and validator gates |
| A provider returns a made-up price or no rendered content | An unproven value can look like a verified observation | Same-call price evidence, explicit no-content logging, and bounded provider fallback |
| A weak or malformed validator result is admitted | A wrong product can enter a match or index | Strict validation, the 0.75 admission threshold, and IS TRUE on stored validator fields |
| A pinned page drifts to another product | A trusted URL can write a wrong price repeatedly | Direct-hit validator enforcement and automatic demotion only for machine-owned auto matches |
| A two-retailer price disagreement is resolved by choosing one side | The index can use a bad price while hiding the conflict | The validation job holds both sides of a disputed pair |
| An old observation remains the latest usable price | A stale page can hold fresh peers or distort the basket | The last_seen_at freshness window and latest-observation stock check |
| A scrape run times out without a terminal status | Resume and coverage logic can treat the market as permanently active | The retailer run budget, partial status, and terminal update after health persistence |
| An empty basket keeps yesterday's computed rows | Downstream readers show an old value as current | Transactional delete plus explicit zero coverage |
| A snapshot write fails after a partial publish | Readers can see mixed freshness or a false healthy run | Last-good retention and non-zero job failure signaling |
Key files
| File | Why to read it |
|---|---|
| scripts/…/food-prices-fetch.mjs | Shared source collection boundary, request limits, proxy routing, and final date checks |
| scripts/…/food-prices.mjs | Shared parsing, observation construction, report discovery, and bounded history merge rules |
| scripts/…/food-prices-seed.mjs | Last-good merge, source retention, and seed health contract |
| scripts/…/index.mjs | Source ID to adapter registration and generated country/cube adapters |
| scripts/…/monthly.mjs | Shared report, month, value, CSV, and cube normalization rules |
| scripts/…/fews.mjs | FEWS monthly price selection and preservation of local market details |
| scripts/…/makstat.mjs | MakStat food CPI context selection and label validation |
| scripts/…/food-price-sources.json | Source cadence, freshness, context, attribution, and fetch policy |
| scripts/…/foag.mjs | Dimension-preserving consumer-price series and vocabulary checks |
| scripts/…/pricecatcher.mjs | Premise-level grouping and median acceptance rule |
| scripts/…/mcba.mjs | Wholesale quote disambiguation and archive validation |
| consumer-prices-core/…/search.ts | Retailer URL discovery, extraction policy, provider fallback, and candidate evidence |
| consumer-prices-core/…/validator.ts | Structured product, class, and quantity validation contract |
| consumer-prices-core/…/scrape.ts | Observation admission, match persistence, pin maintenance, and run accounting |
| consumer-prices-core/…/validate.ts | Cross-retailer median and review-hold behavior |
| consumer-prices-core/…/aggregate.ts | Fresh usable inputs and basket index rules |
| consumer-prices-core/…/publish.ts | Redis snapshot handoff and freshness behavior |
| consumer-prices-core/…/migrate.ts | Migration order and transaction contract |
| consumer-prices-core/…/012_observation_validator_columns.sql | Database-side validator admission fields |
| consumer-prices-core/…/scrape.test.ts | Pin disable and recovery guarantees |
| consumer-prices-core/…/validate.test.ts | Cross-retailer, freshness, and zero-coverage guarantees |
| consumer-prices-core/…/publish.test.ts | Snapshot, metadata, and failure guarantees |
| src/…/ConsumerPricesPanel.ts | Downstream market, health, and world-inflation presentation contract |