External Tool and Cloud Modules
Overview
This doc owns the external scanner wrappers, cloud object-storage discovery modules, and blocklist/DNS reputation checks at these paths/globs:
- modules/sfp_azureblobstorage.py
- modules/sfp_digitaloceanspace.py
- modules/sfp_googleobjectstorage.py
- modules/sfp_s3bucket.py
- modules/sfp_tool_*.py
- modules/sfp_projectdiscovery.py
- modules/sfp_crxcavator.py
- modules/sfp_builtwith.py
- modules/sfp_whatcms.py
- modules/sfp_cookie.py
- modules/sfp_adblock.py
- modules/sfp_adguard_dns.py
- modules/sfp_dns_for_family.py
- modules/sfp_quad9.py
- modules/sfp_stevenblack_hosts.py
- test/…/test_sfp_azureblobstorage.py
- test/…/test_sfp_digitaloceanspace.py
- test/…/test_sfp_googleobjectstorage.py
- test/…/test_sfp_s3bucket.py
- test/…/test_sfp_tool_*.py
- test/…/test_sfp_projectdiscovery.py
- test/…/test_sfp_crxcavator.py
- test/…/test_sfp_builtwith.py
- test/…/test_sfp_whatcms.py
- test/…/test_sfp_cookie.py
- test/…/test_sfp_ad*.py
- test/…/test_sfp_tool_*.py
The modules/sfp_tool_*.py modules are SpiderFoot adapters around local command-line security tools. They all run from SpiderFoot events, validate required path options, deduplicate inputs with results, invoke a local binary with subprocess.Popen, parse stdout or a tool output file, and emit SpiderFoot events derived from tool findings. The wrappers do not install tools themselves: operators must configure paths such as whatweb_path, wappalyzer_path, wafw00f_path, trufflehog_path, testsslsh_path, snallygaster_path, retirejs_path, onesixtyone_path, nuclei_path, template_path, nmappath, nbtscan_path, dnstwistpath, and cmseekpath in each module's opts.
Web technology tool wrappers operate on host or URL events. modules/sfp_tool_whatweb.py watches INTERNET_NAME, executes Ruby plus the configured WhatWeb executable with JSON logging to stdout, and emits WEBSERVER_BANNER, WEBSERVER_TECHNOLOGY, and RAW_RIR_DATA after filtering noisy WhatWeb plugins such as headers and redirects. modules/sfp_tool_wappalyzer.py watches INTERNET_NAME, runs Node.js against the configured Wappalyzer cli.js using https://{host}, parses technologies, and maps category names to OPERATING_SYSTEM, WEBSERVER_TECHNOLOGY, or SOFTWARE_USED. modules/sfp_tool_wafw00f.py runs Python plus WAFW00F with JSON output and emits RAW_RIR_DATA plus WEBSERVER_TECHNOLOGY for non-generic firewall detections. modules/sfp_tool_cmseek.py runs CMSeeK in batch mode and reads the generated Result/…/cms.json file to emit a combined CMS name/version as WEBSERVER_TECHNOLOGY.
Vulnerability and exposure tool wrappers translate scanner-specific JSON/plaintext into SpiderFoot vulnerability events. modules/sfp_tool_nuclei.py watches INTERNET_NAME, IP_ADDRESS, and bounded NETBLOCK_OWNER ranges, feeds one or more targets to Nuclei stdin with -json, -no-interactsh, and excluded dos, fuzz, and misc tags, then emits VULNERABILITY_CVE_* via self.sf.cveInfo() when a CVE appears in the output line; informational Nuclei matches become WEBSERVER_TECHNOLOGY, and other matcher findings become VULNERABILITY_GENERAL. modules/sfp_tool_testsslsh.py runs testssl.sh with severity threshold mincve, writes JSON to a temporary file, and emits CVE or general vulnerability events; for netblock scans it only creates an IP_ADDRESS child event when at least one vulnerable result exists. modules/sfp_tool_retirejs.py only processes linked URLs containing .js, fetches the JavaScript into a temporary directory, runs retire --outputformat json -j, and emits either VULNERABILITY_GENERAL or VULNERABILITY_CVE_*. modules/sfp_tool_snallygaster.py runs snallygaster --nowww -j against INTERNET_NAME and turns each result with a cause into VULNERABILITY_GENERAL. modules/sfp_tool_trufflehog.py watches SOCIAL_MEDIA and optionally PUBLIC_CODE_REPO when allrepos is true, extracts GitHub/GitLab/Bitbucket repository URLs, runs TruffleHog with JSON and regex mode, and emits each parsed row as PASSWORD_COMPROMISED while suppressing large diff fields.
Network scanner wrappers are bounded by path validation, netblock size options, and self-event suppression. modules/sfp_tool_nmap.py watches IP_ADDRESS and NETBLOCK_OWNER, requires an nmappath ending in nmap or /, runs nmap -O --osscan-limit, and emits OPERATING_SYSTEM for reliable OS details, creating child IP_ADDRESS events for netblock hits. modules/sfp_tool_nbtscan.py watches IP_ADDRESS and NETBLOCK_OWNER, runs nbtscan -v, parses NetBIOS name table blocks, and emits UDP_PORT_OPEN on port 137 plus UDP_PORT_OPEN_INFO; for netblock output it extracts the scanned IP and emits it first. modules/sfp_tool_onesixtyone.py writes configured SNMP communities to a temporary file during setup, runs onesixtyone -c {communitiesFile}, and emits UDP_PORT_OPEN on port 161 plus UDP_PORT_OPEN_INFO for rows containing the target. modules/sfp_tool_dnstwist.py watches DOMAIN_NAME, optionally skips wildcard TLDs, prefers dnstwist from PATH before the configured dnstwistpath, parses JSON output across both domain-name and domain field variants, and emits SIMILARDOMAIN except for domains still matching the target.
The cloud-storage modules (modules/sfp_s3bucket.py, modules/sfp_digitaloceanspace.py, modules/sfp_googleobjectstorage.py, and modules/sfp_azureblobstorage.py) generate likely bucket names from a DOMAIN_NAME by combining eventData.replace('.', ''), self.sf.domainKeyword(...), provider endpoints, and configured suffixes. They also pass through externally linked provider URLs from LINKED_URL_EXTERNAL as CLOUD_STORAGE_BUCKET when they match provider hostnames. The S3, DigitalOcean Spaces, and Google Object Storage modules fetch candidate HTTPS URLs, treat NoSuchBucket as a miss, count <Key> elements in ListBucketResult, and emit CLOUD_STORAGE_BUCKET plus CLOUD_STORAGE_BUCKET_OPEN when file count is non-zero. The Azure Blob module treats any response code as a found bucket and only emits CLOUD_STORAGE_BUCKET. All four batch checks into _maxthreads chunks, spawn named Python threads, use a lock for shared result maps, and use noLog=True for storage fetches where implemented.
API-backed enrichment modules in this scope query hosted services rather than local tools. modules/sfp_projectdiscovery.py requires api_key, calls https://dns.projectdiscovery.io/dns/{domain}/subdomains, emits RAW_RIR_DATA, then emits INTERNET_NAME or INTERNET_NAME_UNRESOLVED depending on the verify option and DNS resolution. modules/sfp_crxcavator.py searches CRXcavator by domain keyword, fetches individual extension reports, filters Chrome extension metadata to records whose privacy/support/offered-by/website hosts match the target, emits APPSTORE_ENTRY, target/affiliate host events, internal linked URLs, addresses, and raw data, and sleeps one second between API calls. modules/sfp_builtwith.py requires api_key, calls BuiltWith relationship/domain endpoints, extracts emails, phones, names, subdomains, technologies younger than maxage, shared IPs/cohosts, analytics identifiers, and affiliate domains. modules/sfp_whatcms.py requires api_key, enforces WhatCMS delay/timeout settings, treats API error codes as fatal by setting errorState, and emits RAW_RIR_DATA plus WEBSERVER_TECHNOLOGY from the Technology endpoint.
The reputation and filtering modules compare URLs or hosts against external lists/resolvers. modules/sfp_adblock.py downloads and caches an AdBlock Plus list, parses it with adblockparser.AdblockRules, and emits URL_ADBLOCKED_EXTERNAL for third-party/script matches or URL_ADBLOCKED_INTERNAL for internal URL matches. modules/sfp_stevenblack_hosts.py downloads and caches Steven Black's hosts file, parses hostnames, optionally skips affiliates/cohosts, and emits both MALICIOUS_* and BLACKLISTED_* events. modules/sfp_adguard_dns.py, modules/sfp_dns_for_family.py, and modules/sfp_quad9.py use dns.resolver.Resolver with provider nameservers to decide whether a host is blocked: AdGuard checks default/family resolvers for 94.140.14.35, DNS for Family checks for 159.69.10.249, and Quad9 first verifies normal SpiderFoot resolution then treats failure through 9.9.9.9 as blocked/malicious.
modules/sfp_cookie.py is a local extraction helper in this group: it watches WEBSERVER_HTTPHEADERS, deduplicates by event.actualSource, refuses external hosts by checking self.getTarget().matches(self.sf.urlFQDN(eventSource)), parses the header JSON, and emits TARGET_WEB_COOKIE when a cookie field is present.
The unit tests under test/unit/modules/ are mostly smoke tests for option/description parity, setup, watched event lists, produced event lists, and selected safety failures. For example, test/…/test_sfp_tool_nuclei.py, test/…/test_sfp_tool_testsslsh.py, test/…/test_sfp_builtwith.py, and test/…/test_sfp_whatcms.py assert that missing paths or API keys set errorState; test/…/test_sfp_whatcms.py also documents which HTTP status paths are expected to be fatal.
Coupling
- If a local tool wrapper's command-line arguments change, also update its parser expectations in the same file: modules/sfp_tool_nuclei.py, modules/sfp_tool_whatweb.py, modules/sfp_tool_wafw00f.py, modules/sfp_tool_retirejs.py, modules/sfp_tool_snallygaster.py, modules/sfp_tool_testsslsh.py, modules/sfp_tool_wappalyzer.py, modules/sfp_tool_dnstwist.py, and modules/sfp_tool_cmseek.py all assume specific JSON keys or output files produced by the invoked command.
- If netblock handling changes in modules/sfp_tool_nuclei.py, modules/sfp_tool_testsslsh.py, modules/sfp_tool_nmap.py, modules/sfp_tool_nbtscan.py, or modules/sfp_tool_onesixtyone.py, verify
netblockscanandnetblockscanmaxbehavior together; these modules rely onnetaddr.IPNetworkcontainment checks to avoid rescanning subnets and to cap expensive/invasive scans. - If SpiderFoot event names change, update the hard-coded emitted and watched types across this scope: cloud modules emit
CLOUD_STORAGE_BUCKET(_OPEN), tool wrappers emitVULNERABILITY_*,WEBSERVER_*,UDP_PORT_*, andOPERATING_SYSTEM, and DNS/list modules emit pairedBLACKLISTED_*/MALICIOUS_*event families. - If
_fetchtimeout,_useragent,_internettlds, or_genericusersoption behavior changes globally, recheck modules/sfp_builtwith.py, modules/sfp_projectdiscovery.py, modules/sfp_crxcavator.py, modules/sfp_whatcms.py, modules/sfp_stevenblack_hosts.py, modules/sfp_tool_retirejs.py, and the cloud-storage modules because they read these shared options directly rather than owning local defaults. - If cache semantics change, verify modules/sfp_adblock.py and modules/sfp_stevenblack_hosts.py: both accept a
cacheperiodoption, but their cache reads currently pass a fixed24hour TTL rather than the option value.
Risks
- modules/sfp_s3bucket.py, modules/sfp_digitaloceanspace.py, and modules/sfp_googleobjectstorage.py build result strings as
https://host:filecountand later split on:. Becausehttps://contains a colon, fields are positional in a fragile way; if URL formatting changes,CLOUD_STORAGE_BUCKETandCLOUD_STORAGE_BUCKET_OPENcan contain malformed bucket names or incorrect file counts. - modules/sfp_azureblobstorage.py, modules/sfp_s3bucket.py, modules/sfp_digitaloceanspace.py, and modules/sfp_googleobjectstorage.py only flush a batch when the loop reaches
_maxthreads; a final partial batch is not processed. Targets with fewer candidates than_maxthreads, or the tail of any candidate list, can be silently skipped. - modules/sfp_tool_retirejs.py creates a temporary directory before fetching the JavaScript and returns without removing it when
fetchUrl()returns no content. Repeated fetch failures can leak files under the system temp directory. - modules/sfp_tool_cmseek.py reads Result/…/cms.json directly under the CMSeeK installation path. If
eventDatacontains URL punctuation or CMSeeK changes its output directory naming, SpiderFoot reportsFile does not existand emits no technology event even when CMSeeK succeeded.