Introducing a new way to review your PRs.

Read the announcement

lockfale / osint-framework

OSINT Framework

View on GitHub

8 documents · Updated Nov 16, 2026

docs / enrichment-merge-flow

Enrichment Merge Flow

Overview

The enrichment merge flow is a one-off Python entrypoint in merge_enrichments.py that applies a structured enrichment batch from enrichment-batch4-domains.json into the OSINT catalog stored in public/arf.json. Run from the repository root, the script loads enrichment-batch4-domains.json, takes its top-level enrichments object, loads the full public/arf.json tree, recursively walks every dictionary node, and mutates any node whose name exactly matches an enrichment key.

The matching algorithm is deliberately simple: merge_enrichment_into_node(node, enrichments) checks whether the current dict has a name field and whether that string exists as a key in enrichments; if so, every key/value pair from that enrichment object is assigned onto the catalog node. The merge is overwrite-style and shallow: existing fields with the same names are replaced, fields not present in the enrichment are preserved, and nested structures are not merged recursively beyond the catalog traversal itself. After checking the current node, the function descends into children only when children exists and is a list.

enrichment-batch4-domains.json is expected to contain a JSON object shaped like { "enrichments": { "Tool Name": { ...metadata... } } }. The batch’s metadata fields include catalog annotations such as description, status, pricing, bestFor, input, output, opsec, opsecNote, localInstall, googleDork, registration, editUrl, api, invitationOnly, and deprecated. The batch does not include name, type, or url, so those identifying fields are preserved in public/arf.json for the matched resources.

public/arf.json is expected to be a hierarchical OSINT Framework tree with a root folder and nested children arrays. The batch currently targets domain-related catalog entries, especially the Domain Blacklists area containing resources such as Threatexpert.com Malicious URLs, Zeus C2 Tracker, Malware Domains Blacklist, Blackweb, DNS Sinkhole, DNS-BH Malware Domain Blocklist, MalwareURL (R), scumware.org, ZeuS Tracker, and Email Domain Validation. Because matching is global by exact name, the same enrichment also applies to every duplicate name elsewhere in the catalog; for example, Malware Domain List appears in both the broader malicious-domain reputation area and Domain Blacklists, so both nodes receive the same fields.

Expected execution flow:

  1. Run python3 merge_enrichments.py from the repository root so the script’s relative file paths resolve.
  2. merge_enrichments.py reads enrichment-batch4-domains.json and extracts enrichment_data["enrichments"].
  3. It reads public/arf.json into memory.
  4. It recursively walks the catalog tree and overwrites matched nodes with enrichment fields.
  5. It writes the entire updated catalog back to public/arf.json with two-space indentation.
  6. It prints a success message using len(enrichments), which reports the number of batch entries, not the number of catalog nodes actually modified.

The flow has no network calls, database writes, or external services. Its only state changes are local filesystem reads of enrichment-batch4-domains.json and public/arf.json, followed by an in-place rewrite of public/arf.json.

Routing / ownership

This doc owns the enrichment merge flow and the following covered paths/globs:

Coupling

  • If a resource is renamed in public/arf.json, update the corresponding key in enrichment-batch4-domains.json; merge_enrichments.py matches only exact name strings and silently skips non-matching enrichment keys.
  • If new enrichment fields are added to enrichment-batch4-domains.json, verify the frontend or downstream catalog consumers tolerate those fields in public/arf.json; the script copies every key from the enrichment object without an allowlist.
  • If duplicate name values in public/arf.json are intended to receive different metadata, do not use this script unchanged; the merge key is only name, so every duplicate node receives the same enrichment.
  • If public/arf.json formatting or generation expectations change, check merge_enrichments.py; it rewrites the complete file using Python json.dump(..., indent=2) and does not preserve prior whitespace or key formatting.

Risks

  • merge_enrichments.py writes directly to public/arf.json without a temporary file or backup. If serialization or disk write fails after opening the target for write, the catalog can be left truncated or partially written.
  • merge_enrichments.py reports Successfully merged enrichment data for {len(enrichments)} tools even when some names do not exist in public/arf.json. A typo or stale key in enrichment-batch4-domains.json therefore looks successful but leaves the target entry unchanged.
  • Exact-name global matching in merge_enrichments.py can update unintended duplicates in public/arf.json. This is visible for Malware Domain List, which occurs in more than one catalog location and is matched by one batch key.
  • public/arf.json has a long history of hand edits, enrichment batches, and syntax/structure fixes, including commits described as fixing a missing comma, removing a null entry, fixing trailing newline/array formatting, and reverting prior catalog changes. Treat large generated diffs from this flow carefully: review JSON validity and affected nodes rather than relying only on the script’s success message.