Correlation rules
Owned paths/globs:
- spiderfoot/correlation.py
correlations/**- test/…/test_spiderfootcorrelator.py
Overview
The correlation engine is SpiderFootCorrelator in spiderfoot/correlation.py. It turns YAML rule strings into higher-level scan findings after a scan has finished. The correlator is constructed with a SpiderFootDb, a ruleset dict whose values are YAML rule bodies, and an optional scanId. During initialization it loads each YAML body with yaml.safe_load(), stores the original text back on the parsed rule as rawYaml, trims string values in meta, loads event type metadata from SpiderFootDb.eventTypes(), and validates rule shape before any scan processing runs.
run_correlations() is the scan-level entry point in spiderfoot/correlation.py. It refuses to run if SpiderFootDb.scanInstanceGet(scanId) does not exist or if the scan is still RUNNING, STARTING, or STARTED. For each loaded rule it calls process_rule(), then writes each surviving bucket as a correlation result through create_correlation(). Output is stored with SpiderFootDb.correlationResultCreate(scanId, rule id, meta name, meta description, meta risk, raw YAML, generated title, event IDs), so the persisted finding keeps both the display metadata and the original rule text.
Rule packs live under correlations/**. Each YAML rule has an id, version, meta, one or more collections, an optional aggregation, optional analysis, and a headline; correlations/template.yaml and correlations/README.md document the author-facing format. spiderfoot/correlation.py enforces mandatory top-level components meta, collections, and headline, allows only known top-level keys, requires meta.name, meta.description, and meta.risk, and validates collection methods/fields and analysis method names. The tests in test/…/test_spiderfootcorrelator.py cover constructor type checks, invalid YAML/rules, invalid scan IDs, invalid argument types, and basic validity failures, but they do not exercise the bundled YAML rules end to end.
Collection evaluation has two phases. The first matcher in each collect block is converted to database criteria by build_db_criteria() and fetched with SpiderFootDb.scanResultEvent(). That first matcher can query only top-level type, module, or data; dotted fields such as source.data, child.type, or entity.data are only available after the initial fetch. type can be exact or regex against known event type names; module supports only exact values; data is exact-only for the database query and is encoded before being passed as criteria. Later matchers in the same collect block run locally through refine_collection() and event_keep() against the fetched event objects.
The event objects created by collect_from_db() contain type, data, module, id, entity_type, and empty source, child, and entity arrays. Before collection, analyze_rule_scope() scans all collection, aggregation, and analysis fields. If a rule references source.*, child.*, or entity.*, the collector enriches fetched events from the database: enrich_event_sources() calls SpiderFootDb.scanElementSourcesDirect(), enrich_event_children() calls SpiderFootDb.scanResultEvent(sourceId=...), and enrich_event_entities() walks source chains until it finds event types classified as ENTITY or INTERNAL according to the eventTypes() metadata. Enrichment is chunked in 5,000-event batches.
Matching semantics are intentionally simple. event_extract() recursively follows dotted fields through source, child, or entity arrays and returns all matching values. event_keep() supports exact and case-insensitive regex, and values prefixed with not invert the match: a list of only negative patterns keeps values that match none of them, while positive patterns keep values that match at least one positive entry. Each collected event and enriched sub-event is stamped with _collection, allowing later analysis to distinguish multiple collect blocks.
Aggregation is performed by aggregate_events() when a rule has an aggregation section. It buckets events by every value extracted from the configured field; if the aggregation field is dotted, non-matching sub-events are stripped from the copied event placed in that bucket. Without aggregation, process_rule() uses a single default bucket containing all collected events. Analysis methods then mutate the bucket map in place:
thresholdkeeps only buckets whose extracted field counts fall betweenminimumandmaximum; withcount_unique_only, the threshold applies to the number of unique extracted values.outlierkeeps buckets whose share of all bucketed events is at or belowmaximum_percent, unless the average bucket percentage is belownoisy_percent(default10), in which case all buckets are dropped as too anomalous/noisy.first_collection_onlykeeps values present in collection0and removes buckets where the same field value appears in any later collection.match_all_to_first_collectionbuilds a reference set from collection0, then keeps later-collection events that match it byexact, substringcontains, or IP-in-networksubnetusingnetaddr.both_collectionsis accepted by validation but is not implemented inanalyze_events().
Correlation titles are generated by build_correlation_title(). headline can be a string or a dict with text; placeholders like {data}, {source.data}, or {entity.data} are replaced from the first event in the result bucket, with carriage returns removed and only the first line used. create_correlation() currently stores all top-level event IDs in the bucket as evidence for the result.
The bundled YAML rules fall into several groups:
- Exposure and misconfiguration: database_exposed.yaml, remote_desktop_exposed.yaml, cloud_bucket_open.yaml, cloud_bucket_open_related.yaml, dns_zone_transfer_possible.yaml, root_path_needs_auth.yaml, internal_host.yaml, open_port_version.yaml, and cert_expired.yaml look for directly risky scan events or source/child combinations.
- Vulnerability and stale-host signals: vulnerability_critical.yaml, vulnerability_high.yaml, vulnerability_mediumlow.yaml, stale_host.yaml, and http_errors.yaml group vulnerability, HTTP, expired certificate, junk file, and error events by host/entity.
- Malicious and breach consensus: multiple_malicious.yaml, multiple_malicious_affiliate.yaml, multiple_malicious_cohost.yaml, and email_in_multiple_breaches.yaml use threshold analysis to require repeated reports from scan data.
- Outliers: outlier_cloud.yaml, outlier_country.yaml, outlier_email.yaml, outlier_hostname.yaml, outlier_ipaddress.yaml, outlier_registrar.yaml, and outlier_webserver.yaml aggregate common entity types or infrastructure attributes and keep rare buckets.
- Source-specific discoveries: host_only_from_bruteforce.yaml, host_only_from_certificatetransparency.yaml, email_only_from_pasteleak_site.yaml, name_only_from_pasteleak_site.yaml, email_in_whois.yaml, human_name_in_whois.yaml, data_from_base64.yaml, and data_from_docmeta.yaml reason about where a datum came from by using module, source, or child fields.
- Relationship-strength and cross-reference rules: strong_affiliate_certs.yaml, strong_similardomain_crossref.yaml, and egress_ip_from_wikipedia.yaml use source/entity relationships or
match_all_to_first_collectionto identify stronger ties to the target.
Coupling
- If event type names, entity classifications, or the tuple layout returned by
SpiderFootDb.eventTypes()changes, update spiderfoot/correlation.py and relevant correlations/*.yaml rules together.SpiderFootCorrelator.__init__()reads type names from tuple index1and entity classification from index3, then enrichment uses that map to decide which source-chain records can satisfyentity.*fields. - If
SpiderFootDb.scanResultEvent()orSpiderFootDb.scanElementSourcesDirect()changes row order, verifycollect_from_db(),enrich_event_sources(),enrich_event_children(), andenrich_event_entities()in spiderfoot/correlation.py. These methods address DB rows by numeric indices, so a silent tuple-shape change will attach wrongdata,type,module, IDs, or sources to correlation evidence. - If adding a YAML rule under
correlations/**, make the first matcher in everycollectblock query only top-leveltype,module, or exactdata.build_db_criteria()rejects dotted first fields, regex-on-dataas the first matcher, and regex-on-module; move those constraints to later matchers so they run locally after enrichment. - If adding a new analysis method to correlations/*.yaml, update both
check_rule_validity()andanalyze_events()in spiderfoot/correlation.py. Validation and execution use separate method allowlists/dispatch, so accepting a method does not make it execute. - If relying on
headline.publish_collectionsfrom correlations/README.md, verifycreate_correlation()in spiderfoot/correlation.py. The current output path stores event IDs from every event in the bucket and does not filter bypublish_collections.
Risks
- correlations/README.md documents
headline.publish_collections, and correlations/egress_ip_from_wikipedia.yaml sets it to publish only collection1, but spiderfoot/correlation.py only readsheadline.textinbuild_correlation_title()and ignorespublish_collectionsincreate_correlation(). If a rule uses collection0only as reference data, the persisted correlation can include reference event IDs as evidence. both_collectionsis listed as a valid analysis method incheck_rule_validity()in spiderfoot/correlation.py, butanalyze_events()contains only a TODO/pass branch for it. A YAML rule using this method will load successfully but will not filter buckets, causing broader results than the rule author likely intended.- The bundled tests in test/…/test_spiderfootcorrelator.py mostly assert type errors and invalid-rule handling. If matching, enrichment, aggregation, or YAML pack behavior changes in spiderfoot/correlation.py or
correlations/**, regressions can pass this test file unless additional fixture-backed correlation execution tests are added.