Threat intelligence and reputation modules
Overview
This doc owns the SpiderFoot modules that enrich or score observed indicators against threat intelligence, malware, phishing, abuse, blocklist, DNSBL, and reputation sources. Owned path globs are: modules/sfp_abuse*.py, modules/sfp_alienvault*.py, modules/sfp_bitcoinabuse.py, modules/sfp_blocklistde.py, modules/sfp_botscout.py, modules/sfp_botvrij.py, modules/sfp_cinsscore.py, modules/sfp_circllu.py, modules/sfp_citadel.py, modules/sfp_cleanbrowsing.py, modules/sfp_coinblocker.py, modules/sfp_comodo.py, modules/sfp_cybercrimetracker.py, modules/sfp_dronebl.py, modules/sfp_emergingthreats.py, modules/sfp_fortinet.py, modules/sfp_fraudguard.py, modules/sfp_fsecure_riddler.py, modules/sfp_google*.py, modules/sfp_greensnow.py, modules/sfp_greynoise*.py, modules/sfp_honeypot.py, modules/sfp_hybrid_analysis.py, modules/sfp_isc.py, modules/sfp_koodous.py, modules/sfp_maltiverse.py, modules/sfp_malwarepatrol.py, modules/sfp_metadefender.py, modules/sfp_openphish.py, modules/sfp_phishstats.py, modules/sfp_phishtank.py, modules/sfp_pulsedive.py, modules/sfp_sorbs.py, modules/sfp_spamcop.py, modules/sfp_spamhaus.py, modules/sfp_spur.py, modules/sfp_surbl.py, modules/sfp_talosintel.py, modules/sfp_threat*.py, modules/sfp_uceprotect.py, modules/sfp_urlscan.py, modules/sfp_virustotal.py, modules/sfp_voipbl.py, modules/sfp_vxvault.py, modules/sfp_xforce.py, plus their covered tests under test/…/test_sfp_abuse*.py, test/…/test_sfp_alienvault*.py, test/…/test_sfp_*threat*.py, test/…/test_sfp_virustotal.py, test/…/test_sfp_abuse*.py, and test/…/test_sfp_alienvault*.py.
All of these files are SpiderFootPlugin modules that receive SpiderFoot events, deduplicate queries in self.results, call an external feed/API/DNS service, and emit new SpiderFootEvent findings. The common watched inputs are IP_ADDRESS, IPV6_ADDRESS, AFFILIATE_IPADDR, AFFILIATE_IPV6_ADDRESS, INTERNET_NAME, AFFILIATE_INTERNET_NAME, CO_HOSTED_SITE, NETBLOCK_OWNER, and NETBLOCK_MEMBER; specialized modules add BITCOIN_ADDRESS in modules/sfp_bitcoinabuse.py, EMAILADDR in modules/sfp_botscout.py, modules/sfp_citadel.py, and modules/sfp_threatcrowd.py, and DOMAIN_NAME for search/enrichment modules such as modules/sfp_threatminer.py, modules/sfp_hybrid_analysis.py, modules/sfp_koodous.py, and modules/sfp_fsecure_riddler.py.
There are several lookup categories:
- Downloaded plaintext/CSV feeds: modules/sfp_abuseipdb.py, modules/sfp_abusech.py, modules/sfp_alienvaultiprep.py, modules/sfp_blocklistde.py, modules/sfp_botvrij.py, modules/sfp_cinsscore.py, modules/sfp_coinblocker.py, modules/sfp_cybercrimetracker.py, modules/sfp_emergingthreats.py, modules/sfp_malwarepatrol.py, modules/sfp_openphish.py, modules/sfp_phishtank.py, modules/sfp_talosintel.py, modules/sfp_voipbl.py, and modules/sfp_vxvault.py fetch lists via
self.sf.fetchUrl(), cache content withself.sf.cacheGet()/cachePut(), parse host/IP entries, then match the incoming indicator or each IP in a netblock. - DNS reputation lookups: modules/sfp_dronebl.py, modules/sfp_honeypot.py, modules/sfp_sorbs.py, modules/sfp_spamcop.py, modules/sfp_spamhaus.py, modules/sfp_surbl.py, and modules/sfp_uceprotect.py reverse IPv4 addresses and query DNSBL zones through
self.sf.resolveHost(). modules/sfp_cleanbrowsing.py and modules/sfp_comodo.py use explicit DNS resolvers and treat a normally resolving host that does not resolve through the filtering resolver as blocked. - JSON REST reputation APIs: modules/sfp_alienvault.py, modules/sfp_bitcoinabuse.py, modules/sfp_fraudguard.py, modules/sfp_googlesafebrowsing.py, modules/sfp_greynoise.py, modules/sfp_greynoise_community.py, modules/sfp_maltiverse.py, modules/sfp_metadefender.py, modules/sfp_phishstats.py, modules/sfp_pulsedive.py, modules/sfp_spur.py, modules/sfp_threatfox.py, modules/sfp_threatjammer.py, and modules/sfp_xforce.py parse JSON scores, classifications, threats, or blacklist arrays before emitting findings.
- Passive DNS/search enrichment: modules/sfp_alienvault.py, modules/sfp_circllu.py, modules/sfp_fsecure_riddler.py, modules/sfp_hybrid_analysis.py, modules/sfp_koodous.py, modules/sfp_threatcrowd.py, modules/sfp_threatminer.py, modules/sfp_urlscan.py, modules/sfp_virustotal.py, and modules/sfp_xforce.py produce supporting events such as
INTERNET_NAME,INTERNET_NAME_UNRESOLVED,CO_HOSTED_SITE,DOMAIN_NAME,LINKED_URL_INTERNAL,APPSTORE_ENTRY,RAW_RIR_DATA,GEOINFO,BGP_AS_MEMBER, orCOMPANY_NAMEin addition to any malicious verdicts.
Malicious indicators are emitted by mapping the input event to the closest MALICIOUS_* and often parallel BLACKLISTED_* event type. IP and IPv6 inputs usually become MALICIOUS_IPADDR / BLACKLISTED_IPADDR; affiliate IP inputs become MALICIOUS_AFFILIATE_IPADDR; host inputs become MALICIOUS_INTERNET_NAME, MALICIOUS_AFFILIATE_INTERNET_NAME, or MALICIOUS_COHOST; owned netblocks and subnet/member netblocks become MALICIOUS_NETBLOCK or MALICIOUS_SUBNET. Many modules emit both malicious and blacklisted events for the same hit, for example modules/sfp_blocklistde.py, modules/sfp_botscout.py, modules/sfp_voipbl.py, modules/sfp_spamhaus.py, modules/sfp_surbl.py, and modules/sfp_threatfox.py; some sources only emit malicious or enrichment events, such as modules/sfp_abusech.py, modules/sfp_fraudguard.py, modules/sfp_greynoise.py, modules/sfp_pulsedive.py, modules/sfp_threatcrowd.py, and modules/sfp_virustotal.py.
API keys are declared in each module's opts and advertised through meta['flags'] or dataSource['apiKeyInstructions']. Most modules use a single api_key (modules/sfp_abuseipdb.py, modules/sfp_bitcoinabuse.py, modules/sfp_googlesafebrowsing.py, modules/sfp_greynoise.py, modules/sfp_hybrid_analysis.py, modules/sfp_koodous.py, modules/sfp_metadefender.py, modules/sfp_pulsedive.py, modules/sfp_spur.py, modules/sfp_threatjammer.py); username/password or basic-auth pairs are used by modules/sfp_circllu.py, modules/sfp_fraudguard.py, modules/sfp_fsecure_riddler.py, and modules/sfp_xforce.py. Header names are provider-specific: Key for AbuseIPDB, X-OTX-API-KEY for AlienVault OTX, apikey for MetaDefender, Authorization: Token for Koodous, Authorization: Bearer for ThreatJammer, and Basic auth for CIRCL, Fraudguard, and X-Force. Modules usually set self.errorState = True and return when credentials are missing or rejected, preventing repeated failing requests during the scan.
Rate and error handling is local to each module. API modules commonly special-case 401, 403, and 429, set errorState, and stop future work (modules/sfp_bitcoinabuse.py, modules/sfp_googlesafebrowsing.py, modules/sfp_metadefender.py, modules/sfp_pulsedive.py, modules/sfp_threatfox.py, modules/sfp_threatjammer.py, modules/sfp_xforce.py). Some providers are proactively throttled with time.sleep(): VirusTotal public API waits 15 seconds in modules/sfp_virustotal.py, BitcoinAbuse waits 2 seconds in modules/sfp_bitcoinabuse.py, MetaDefender defaults to 6 seconds in modules/sfp_metadefender.py, and Koodous/ThreatFox/AbuseIPDB/Hybrid Analysis/Pulsedive use short configurable or fixed delays. Feed-based modules generally treat non-200 fetch responses or empty content as a module-level failure and then rely on cached feed data when available.
Large netblock lookups are guarded with netblocklookup, subnetlookup, maxnetblock, maxsubnet, and IPv6 variants. Modules that iterate every IP in a CIDR include modules/sfp_alienvault.py, modules/sfp_fraudguard.py, modules/sfp_greynoise.py, modules/sfp_honeypot.py, modules/sfp_maltiverse.py, modules/sfp_phishstats.py, modules/sfp_pulsedive.py, modules/sfp_spamcop.py, modules/sfp_spamhaus.py, modules/sfp_spur.py, modules/sfp_surbl.py, modules/sfp_threatcrowd.py, modules/sfp_threatminer.py, modules/sfp_uceprotect.py, modules/sfp_virustotal.py, and modules/sfp_xforce.py; feed modules that scan a cached list against a netblock include modules/sfp_abusech.py, modules/sfp_alienvaultiprep.py, modules/sfp_blocklistde.py, modules/sfp_cinsscore.py, modules/sfp_emergingthreats.py, modules/sfp_talosintel.py, and modules/sfp_voipbl.py.
Coupling
- If a
watchedEvents()input is added or renamed in any owned module, verify the correspondingproducedEvents()andhandleEvent()mapping in the same file. Most reputation modules manually map each event type to aMALICIOUS_*andBLACKLISTED_*name; a missing branch silently drops findings after SpiderFoot routes the event. - If netblock defaults change in modules such as modules/sfp_virustotal.py, modules/sfp_spamhaus.py, modules/sfp_pulsedive.py, or modules/sfp_xforce.py, verify the CIDR guard still prevents accidental full-range iteration. These modules expand
IPNetwork(eventData)into individual addresses before querying external services. - If a provider changes its authentication scheme, update both the request construction and the missing-credential gate in the same module. Header/query-string key names differ across modules/sfp_abuseipdb.py, modules/sfp_alienvault.py, modules/sfp_metadefender.py, modules/sfp_koodous.py, modules/sfp_threatjammer.py, and modules/sfp_xforce.py, so credentials cannot be normalized by only changing an option name.
- If feed URLs or parse formats change, update the cache key/parse method together. Feed modules such as modules/sfp_abusech.py, modules/sfp_openphish.py, modules/sfp_phishtank.py, and modules/sfp_vxvault.py cache raw remote content and re-parse cached content on later scans; changing only the fetch URL can leave stale parse assumptions in place.
- If DNSBL result codes are updated, adjust both the result-code map and the emitted severity events. modules/sfp_dronebl.py, modules/sfp_sorbs.py, modules/sfp_spamhaus.py, and modules/sfp_honeypot.py decide whether to emit malicious, proxy/VPN, or only blacklist events from specific
127.x.x.xresponses.
Risks
- modules/sfp_coinblocker.py declares
BLACKLISTED_COHOSTinproducedEvents()but emitsBACKLISTED_COHOSTforCO_HOSTED_SITEhits. If a CoinBlocker co-host match occurs, downstream consumers expecting the declared event type will not receive it. - modules/sfp_alienvault.py emits
INTERNET_NAME_UNRESOVLEDfor unresolved passive-DNS hosts, misspelling the usualINTERNET_NAME_UNRESOLVEDevent. If another module depends on unresolved-host events, AlienVault OTX passive DNS results can be missed. - modules/sfp_metadefender.py emits
MALICIOUS_IPADDRandBLACKLISTED_IPADDRwhen a source assessment equalstrustworthy, while domain logic treats non-trustworthyas malicious. If MetaDefender's API usestrustworthyliterally, IP reputation results are inverted. - modules/sfp_phishtank.py parses CSV rows with
line.strip().split(",")and then reads index1. If a PhishTank URL contains a comma or the CSV format changes, host extraction can misparse or skip entries because it is not using a CSV parser.