Introducing a new way to review your PRs.

Read the announcement

smicallef / spiderfoot

SpiderFoot automates OSINT for threat intelligence and mapping your attack surface.

View on GitHub

13 documents · Updated Nov 16, 2026

docs / correlation-rules

Correlation rules

Owned paths/globs:

Overview

The correlation engine is SpiderFootCorrelator in spiderfoot/correlation.py. It turns YAML rule strings into higher-level scan findings after a scan has finished. The correlator is constructed with a SpiderFootDb, a ruleset dict whose values are YAML rule bodies, and an optional scanId. During initialization it loads each YAML body with yaml.safe_load(), stores the original text back on the parsed rule as rawYaml, trims string values in meta, loads event type metadata from SpiderFootDb.eventTypes(), and validates rule shape before any scan processing runs.

run_correlations() is the scan-level entry point in spiderfoot/correlation.py. It refuses to run if SpiderFootDb.scanInstanceGet(scanId) does not exist or if the scan is still RUNNING, STARTING, or STARTED. For each loaded rule it calls process_rule(), then writes each surviving bucket as a correlation result through create_correlation(). Output is stored with SpiderFootDb.correlationResultCreate(scanId, rule id, meta name, meta description, meta risk, raw YAML, generated title, event IDs), so the persisted finding keeps both the display metadata and the original rule text.

Rule packs live under correlations/**. Each YAML rule has an id, version, meta, one or more collections, an optional aggregation, optional analysis, and a headline; correlations/template.yaml and correlations/README.md document the author-facing format. spiderfoot/correlation.py enforces mandatory top-level components meta, collections, and headline, allows only known top-level keys, requires meta.name, meta.description, and meta.risk, and validates collection methods/fields and analysis method names. The tests in test/…/test_spiderfootcorrelator.py cover constructor type checks, invalid YAML/rules, invalid scan IDs, invalid argument types, and basic validity failures, but they do not exercise the bundled YAML rules end to end.

Collection evaluation has two phases. The first matcher in each collect block is converted to database criteria by build_db_criteria() and fetched with SpiderFootDb.scanResultEvent(). That first matcher can query only top-level type, module, or data; dotted fields such as source.data, child.type, or entity.data are only available after the initial fetch. type can be exact or regex against known event type names; module supports only exact values; data is exact-only for the database query and is encoded before being passed as criteria. Later matchers in the same collect block run locally through refine_collection() and event_keep() against the fetched event objects.

The event objects created by collect_from_db() contain type, data, module, id, entity_type, and empty source, child, and entity arrays. Before collection, analyze_rule_scope() scans all collection, aggregation, and analysis fields. If a rule references source.*, child.*, or entity.*, the collector enriches fetched events from the database: enrich_event_sources() calls SpiderFootDb.scanElementSourcesDirect(), enrich_event_children() calls SpiderFootDb.scanResultEvent(sourceId=...), and enrich_event_entities() walks source chains until it finds event types classified as ENTITY or INTERNAL according to the eventTypes() metadata. Enrichment is chunked in 5,000-event batches.

Matching semantics are intentionally simple. event_extract() recursively follows dotted fields through source, child, or entity arrays and returns all matching values. event_keep() supports exact and case-insensitive regex, and values prefixed with not invert the match: a list of only negative patterns keeps values that match none of them, while positive patterns keep values that match at least one positive entry. Each collected event and enriched sub-event is stamped with _collection, allowing later analysis to distinguish multiple collect blocks.

Aggregation is performed by aggregate_events() when a rule has an aggregation section. It buckets events by every value extracted from the configured field; if the aggregation field is dotted, non-matching sub-events are stripped from the copied event placed in that bucket. Without aggregation, process_rule() uses a single default bucket containing all collected events. Analysis methods then mutate the bucket map in place:

  • threshold keeps only buckets whose extracted field counts fall between minimum and maximum; with count_unique_only, the threshold applies to the number of unique extracted values.
  • outlier keeps buckets whose share of all bucketed events is at or below maximum_percent, unless the average bucket percentage is below noisy_percent (default 10), in which case all buckets are dropped as too anomalous/noisy.
  • first_collection_only keeps values present in collection 0 and removes buckets where the same field value appears in any later collection.
  • match_all_to_first_collection builds a reference set from collection 0, then keeps later-collection events that match it by exact, substring contains, or IP-in-network subnet using netaddr.
  • both_collections is accepted by validation but is not implemented in analyze_events().

Correlation titles are generated by build_correlation_title(). headline can be a string or a dict with text; placeholders like {data}, {source.data}, or {entity.data} are replaced from the first event in the result bucket, with carriage returns removed and only the first line used. create_correlation() currently stores all top-level event IDs in the bucket as evidence for the result.

The bundled YAML rules fall into several groups:

Coupling

  • If event type names, entity classifications, or the tuple layout returned by SpiderFootDb.eventTypes() changes, update spiderfoot/correlation.py and relevant correlations/*.yaml rules together. SpiderFootCorrelator.__init__() reads type names from tuple index 1 and entity classification from index 3, then enrichment uses that map to decide which source-chain records can satisfy entity.* fields.
  • If SpiderFootDb.scanResultEvent() or SpiderFootDb.scanElementSourcesDirect() changes row order, verify collect_from_db(), enrich_event_sources(), enrich_event_children(), and enrich_event_entities() in spiderfoot/correlation.py. These methods address DB rows by numeric indices, so a silent tuple-shape change will attach wrong data, type, module, IDs, or sources to correlation evidence.
  • If adding a YAML rule under correlations/**, make the first matcher in every collect block query only top-level type, module, or exact data. build_db_criteria() rejects dotted first fields, regex-on-data as the first matcher, and regex-on-module; move those constraints to later matchers so they run locally after enrichment.
  • If adding a new analysis method to correlations/*.yaml, update both check_rule_validity() and analyze_events() in spiderfoot/correlation.py. Validation and execution use separate method allowlists/dispatch, so accepting a method does not make it execute.
  • If relying on headline.publish_collections from correlations/README.md, verify create_correlation() in spiderfoot/correlation.py. The current output path stores event IDs from every event in the bucket and does not filter by publish_collections.

Risks

  • correlations/README.md documents headline.publish_collections, and correlations/egress_ip_from_wikipedia.yaml sets it to publish only collection 1, but spiderfoot/correlation.py only reads headline.text in build_correlation_title() and ignores publish_collections in create_correlation(). If a rule uses collection 0 only as reference data, the persisted correlation can include reference event IDs as evidence.
  • both_collections is listed as a valid analysis method in check_rule_validity() in spiderfoot/correlation.py, but analyze_events() contains only a TODO/pass branch for it. A YAML rule using this method will load successfully but will not filter buckets, causing broader results than the rule author likely intended.
  • The bundled tests in test/…/test_spiderfootcorrelator.py mostly assert type errors and invalid-rule handling. If matching, enrichment, aggregation, or YAML pack behavior changes in spiderfoot/correlation.py or correlations/**, regressions can pass this test file unless additional fixture-backed correlation execution tests are added.