Introducing a new way to review your PRs.

Read the announcement

smicallef / spiderfoot

SpiderFoot automates OSINT for threat intelligence and mapping your attack surface.

View on GitHub

13 documents · Updated Nov 16, 2026

docs / threat-intel-and-reputation-modules

Threat intelligence and reputation modules

Overview

This doc owns the SpiderFoot modules that enrich or score observed indicators against threat intelligence, malware, phishing, abuse, blocklist, DNSBL, and reputation sources. Owned path globs are: modules/sfp_abuse*.py, modules/sfp_alienvault*.py, modules/sfp_bitcoinabuse.py, modules/sfp_blocklistde.py, modules/sfp_botscout.py, modules/sfp_botvrij.py, modules/sfp_cinsscore.py, modules/sfp_circllu.py, modules/sfp_citadel.py, modules/sfp_cleanbrowsing.py, modules/sfp_coinblocker.py, modules/sfp_comodo.py, modules/sfp_cybercrimetracker.py, modules/sfp_dronebl.py, modules/sfp_emergingthreats.py, modules/sfp_fortinet.py, modules/sfp_fraudguard.py, modules/sfp_fsecure_riddler.py, modules/sfp_google*.py, modules/sfp_greensnow.py, modules/sfp_greynoise*.py, modules/sfp_honeypot.py, modules/sfp_hybrid_analysis.py, modules/sfp_isc.py, modules/sfp_koodous.py, modules/sfp_maltiverse.py, modules/sfp_malwarepatrol.py, modules/sfp_metadefender.py, modules/sfp_openphish.py, modules/sfp_phishstats.py, modules/sfp_phishtank.py, modules/sfp_pulsedive.py, modules/sfp_sorbs.py, modules/sfp_spamcop.py, modules/sfp_spamhaus.py, modules/sfp_spur.py, modules/sfp_surbl.py, modules/sfp_talosintel.py, modules/sfp_threat*.py, modules/sfp_uceprotect.py, modules/sfp_urlscan.py, modules/sfp_virustotal.py, modules/sfp_voipbl.py, modules/sfp_vxvault.py, modules/sfp_xforce.py, plus their covered tests under test/…/test_sfp_abuse*.py, test/…/test_sfp_alienvault*.py, test/…/test_sfp_*threat*.py, test/…/test_sfp_virustotal.py, test/…/test_sfp_abuse*.py, and test/…/test_sfp_alienvault*.py.

All of these files are SpiderFootPlugin modules that receive SpiderFoot events, deduplicate queries in self.results, call an external feed/API/DNS service, and emit new SpiderFootEvent findings. The common watched inputs are IP_ADDRESS, IPV6_ADDRESS, AFFILIATE_IPADDR, AFFILIATE_IPV6_ADDRESS, INTERNET_NAME, AFFILIATE_INTERNET_NAME, CO_HOSTED_SITE, NETBLOCK_OWNER, and NETBLOCK_MEMBER; specialized modules add BITCOIN_ADDRESS in modules/sfp_bitcoinabuse.py, EMAILADDR in modules/sfp_botscout.py, modules/sfp_citadel.py, and modules/sfp_threatcrowd.py, and DOMAIN_NAME for search/enrichment modules such as modules/sfp_threatminer.py, modules/sfp_hybrid_analysis.py, modules/sfp_koodous.py, and modules/sfp_fsecure_riddler.py.

There are several lookup categories:

Malicious indicators are emitted by mapping the input event to the closest MALICIOUS_* and often parallel BLACKLISTED_* event type. IP and IPv6 inputs usually become MALICIOUS_IPADDR / BLACKLISTED_IPADDR; affiliate IP inputs become MALICIOUS_AFFILIATE_IPADDR; host inputs become MALICIOUS_INTERNET_NAME, MALICIOUS_AFFILIATE_INTERNET_NAME, or MALICIOUS_COHOST; owned netblocks and subnet/member netblocks become MALICIOUS_NETBLOCK or MALICIOUS_SUBNET. Many modules emit both malicious and blacklisted events for the same hit, for example modules/sfp_blocklistde.py, modules/sfp_botscout.py, modules/sfp_voipbl.py, modules/sfp_spamhaus.py, modules/sfp_surbl.py, and modules/sfp_threatfox.py; some sources only emit malicious or enrichment events, such as modules/sfp_abusech.py, modules/sfp_fraudguard.py, modules/sfp_greynoise.py, modules/sfp_pulsedive.py, modules/sfp_threatcrowd.py, and modules/sfp_virustotal.py.

API keys are declared in each module's opts and advertised through meta['flags'] or dataSource['apiKeyInstructions']. Most modules use a single api_key (modules/sfp_abuseipdb.py, modules/sfp_bitcoinabuse.py, modules/sfp_googlesafebrowsing.py, modules/sfp_greynoise.py, modules/sfp_hybrid_analysis.py, modules/sfp_koodous.py, modules/sfp_metadefender.py, modules/sfp_pulsedive.py, modules/sfp_spur.py, modules/sfp_threatjammer.py); username/password or basic-auth pairs are used by modules/sfp_circllu.py, modules/sfp_fraudguard.py, modules/sfp_fsecure_riddler.py, and modules/sfp_xforce.py. Header names are provider-specific: Key for AbuseIPDB, X-OTX-API-KEY for AlienVault OTX, apikey for MetaDefender, Authorization: Token for Koodous, Authorization: Bearer for ThreatJammer, and Basic auth for CIRCL, Fraudguard, and X-Force. Modules usually set self.errorState = True and return when credentials are missing or rejected, preventing repeated failing requests during the scan.

Rate and error handling is local to each module. API modules commonly special-case 401, 403, and 429, set errorState, and stop future work (modules/sfp_bitcoinabuse.py, modules/sfp_googlesafebrowsing.py, modules/sfp_metadefender.py, modules/sfp_pulsedive.py, modules/sfp_threatfox.py, modules/sfp_threatjammer.py, modules/sfp_xforce.py). Some providers are proactively throttled with time.sleep(): VirusTotal public API waits 15 seconds in modules/sfp_virustotal.py, BitcoinAbuse waits 2 seconds in modules/sfp_bitcoinabuse.py, MetaDefender defaults to 6 seconds in modules/sfp_metadefender.py, and Koodous/ThreatFox/AbuseIPDB/Hybrid Analysis/Pulsedive use short configurable or fixed delays. Feed-based modules generally treat non-200 fetch responses or empty content as a module-level failure and then rely on cached feed data when available.

Large netblock lookups are guarded with netblocklookup, subnetlookup, maxnetblock, maxsubnet, and IPv6 variants. Modules that iterate every IP in a CIDR include modules/sfp_alienvault.py, modules/sfp_fraudguard.py, modules/sfp_greynoise.py, modules/sfp_honeypot.py, modules/sfp_maltiverse.py, modules/sfp_phishstats.py, modules/sfp_pulsedive.py, modules/sfp_spamcop.py, modules/sfp_spamhaus.py, modules/sfp_spur.py, modules/sfp_surbl.py, modules/sfp_threatcrowd.py, modules/sfp_threatminer.py, modules/sfp_uceprotect.py, modules/sfp_virustotal.py, and modules/sfp_xforce.py; feed modules that scan a cached list against a netblock include modules/sfp_abusech.py, modules/sfp_alienvaultiprep.py, modules/sfp_blocklistde.py, modules/sfp_cinsscore.py, modules/sfp_emergingthreats.py, modules/sfp_talosintel.py, and modules/sfp_voipbl.py.

Coupling

Risks

  • modules/sfp_coinblocker.py declares BLACKLISTED_COHOST in producedEvents() but emits BACKLISTED_COHOST for CO_HOSTED_SITE hits. If a CoinBlocker co-host match occurs, downstream consumers expecting the declared event type will not receive it.
  • modules/sfp_alienvault.py emits INTERNET_NAME_UNRESOVLED for unresolved passive-DNS hosts, misspelling the usual INTERNET_NAME_UNRESOLVED event. If another module depends on unresolved-host events, AlienVault OTX passive DNS results can be missed.
  • modules/sfp_metadefender.py emits MALICIOUS_IPADDR and BLACKLISTED_IPADDR when a source assessment equals trustworthy, while domain logic treats non-trustworthy as malicious. If MetaDefender's API uses trustworthy literally, IP reputation results are inverted.
  • modules/sfp_phishtank.py parses CSV rows with line.strip().split(",") and then reads index 1. If a PhishTank URL contains a comma or the CSV format changes, host extraction can misparse or skip entries because it is not using a CSV parser.