Introducing a new way to review your PRs.

Read the announcement

smicallef / spiderfoot

SpiderFoot automates OSINT for threat intelligence and mapping your attack surface.

View on GitHub

13 documents · Updated Nov 16, 2026

docs / external-tool-and-cloud-modules

External Tool and Cloud Modules

Overview

This doc owns the external scanner wrappers, cloud object-storage discovery modules, and blocklist/DNS reputation checks at these paths/globs:

The modules/sfp_tool_*.py modules are SpiderFoot adapters around local command-line security tools. They all run from SpiderFoot events, validate required path options, deduplicate inputs with results, invoke a local binary with subprocess.Popen, parse stdout or a tool output file, and emit SpiderFoot events derived from tool findings. The wrappers do not install tools themselves: operators must configure paths such as whatweb_path, wappalyzer_path, wafw00f_path, trufflehog_path, testsslsh_path, snallygaster_path, retirejs_path, onesixtyone_path, nuclei_path, template_path, nmappath, nbtscan_path, dnstwistpath, and cmseekpath in each module's opts.

Web technology tool wrappers operate on host or URL events. modules/sfp_tool_whatweb.py watches INTERNET_NAME, executes Ruby plus the configured WhatWeb executable with JSON logging to stdout, and emits WEBSERVER_BANNER, WEBSERVER_TECHNOLOGY, and RAW_RIR_DATA after filtering noisy WhatWeb plugins such as headers and redirects. modules/sfp_tool_wappalyzer.py watches INTERNET_NAME, runs Node.js against the configured Wappalyzer cli.js using https://{host}, parses technologies, and maps category names to OPERATING_SYSTEM, WEBSERVER_TECHNOLOGY, or SOFTWARE_USED. modules/sfp_tool_wafw00f.py runs Python plus WAFW00F with JSON output and emits RAW_RIR_DATA plus WEBSERVER_TECHNOLOGY for non-generic firewall detections. modules/sfp_tool_cmseek.py runs CMSeeK in batch mode and reads the generated Result/…/cms.json file to emit a combined CMS name/version as WEBSERVER_TECHNOLOGY.

Vulnerability and exposure tool wrappers translate scanner-specific JSON/plaintext into SpiderFoot vulnerability events. modules/sfp_tool_nuclei.py watches INTERNET_NAME, IP_ADDRESS, and bounded NETBLOCK_OWNER ranges, feeds one or more targets to Nuclei stdin with -json, -no-interactsh, and excluded dos, fuzz, and misc tags, then emits VULNERABILITY_CVE_* via self.sf.cveInfo() when a CVE appears in the output line; informational Nuclei matches become WEBSERVER_TECHNOLOGY, and other matcher findings become VULNERABILITY_GENERAL. modules/sfp_tool_testsslsh.py runs testssl.sh with severity threshold mincve, writes JSON to a temporary file, and emits CVE or general vulnerability events; for netblock scans it only creates an IP_ADDRESS child event when at least one vulnerable result exists. modules/sfp_tool_retirejs.py only processes linked URLs containing .js, fetches the JavaScript into a temporary directory, runs retire --outputformat json -j, and emits either VULNERABILITY_GENERAL or VULNERABILITY_CVE_*. modules/sfp_tool_snallygaster.py runs snallygaster --nowww -j against INTERNET_NAME and turns each result with a cause into VULNERABILITY_GENERAL. modules/sfp_tool_trufflehog.py watches SOCIAL_MEDIA and optionally PUBLIC_CODE_REPO when allrepos is true, extracts GitHub/GitLab/Bitbucket repository URLs, runs TruffleHog with JSON and regex mode, and emits each parsed row as PASSWORD_COMPROMISED while suppressing large diff fields.

Network scanner wrappers are bounded by path validation, netblock size options, and self-event suppression. modules/sfp_tool_nmap.py watches IP_ADDRESS and NETBLOCK_OWNER, requires an nmappath ending in nmap or /, runs nmap -O --osscan-limit, and emits OPERATING_SYSTEM for reliable OS details, creating child IP_ADDRESS events for netblock hits. modules/sfp_tool_nbtscan.py watches IP_ADDRESS and NETBLOCK_OWNER, runs nbtscan -v, parses NetBIOS name table blocks, and emits UDP_PORT_OPEN on port 137 plus UDP_PORT_OPEN_INFO; for netblock output it extracts the scanned IP and emits it first. modules/sfp_tool_onesixtyone.py writes configured SNMP communities to a temporary file during setup, runs onesixtyone -c {communitiesFile}, and emits UDP_PORT_OPEN on port 161 plus UDP_PORT_OPEN_INFO for rows containing the target. modules/sfp_tool_dnstwist.py watches DOMAIN_NAME, optionally skips wildcard TLDs, prefers dnstwist from PATH before the configured dnstwistpath, parses JSON output across both domain-name and domain field variants, and emits SIMILARDOMAIN except for domains still matching the target.

The cloud-storage modules (modules/sfp_s3bucket.py, modules/sfp_digitaloceanspace.py, modules/sfp_googleobjectstorage.py, and modules/sfp_azureblobstorage.py) generate likely bucket names from a DOMAIN_NAME by combining eventData.replace('.', ''), self.sf.domainKeyword(...), provider endpoints, and configured suffixes. They also pass through externally linked provider URLs from LINKED_URL_EXTERNAL as CLOUD_STORAGE_BUCKET when they match provider hostnames. The S3, DigitalOcean Spaces, and Google Object Storage modules fetch candidate HTTPS URLs, treat NoSuchBucket as a miss, count <Key> elements in ListBucketResult, and emit CLOUD_STORAGE_BUCKET plus CLOUD_STORAGE_BUCKET_OPEN when file count is non-zero. The Azure Blob module treats any response code as a found bucket and only emits CLOUD_STORAGE_BUCKET. All four batch checks into _maxthreads chunks, spawn named Python threads, use a lock for shared result maps, and use noLog=True for storage fetches where implemented.

API-backed enrichment modules in this scope query hosted services rather than local tools. modules/sfp_projectdiscovery.py requires api_key, calls https://dns.projectdiscovery.io/dns/{domain}/subdomains, emits RAW_RIR_DATA, then emits INTERNET_NAME or INTERNET_NAME_UNRESOLVED depending on the verify option and DNS resolution. modules/sfp_crxcavator.py searches CRXcavator by domain keyword, fetches individual extension reports, filters Chrome extension metadata to records whose privacy/support/offered-by/website hosts match the target, emits APPSTORE_ENTRY, target/affiliate host events, internal linked URLs, addresses, and raw data, and sleeps one second between API calls. modules/sfp_builtwith.py requires api_key, calls BuiltWith relationship/domain endpoints, extracts emails, phones, names, subdomains, technologies younger than maxage, shared IPs/cohosts, analytics identifiers, and affiliate domains. modules/sfp_whatcms.py requires api_key, enforces WhatCMS delay/timeout settings, treats API error codes as fatal by setting errorState, and emits RAW_RIR_DATA plus WEBSERVER_TECHNOLOGY from the Technology endpoint.

The reputation and filtering modules compare URLs or hosts against external lists/resolvers. modules/sfp_adblock.py downloads and caches an AdBlock Plus list, parses it with adblockparser.AdblockRules, and emits URL_ADBLOCKED_EXTERNAL for third-party/script matches or URL_ADBLOCKED_INTERNAL for internal URL matches. modules/sfp_stevenblack_hosts.py downloads and caches Steven Black's hosts file, parses hostnames, optionally skips affiliates/cohosts, and emits both MALICIOUS_* and BLACKLISTED_* events. modules/sfp_adguard_dns.py, modules/sfp_dns_for_family.py, and modules/sfp_quad9.py use dns.resolver.Resolver with provider nameservers to decide whether a host is blocked: AdGuard checks default/family resolvers for 94.140.14.35, DNS for Family checks for 159.69.10.249, and Quad9 first verifies normal SpiderFoot resolution then treats failure through 9.9.9.9 as blocked/malicious.

modules/sfp_cookie.py is a local extraction helper in this group: it watches WEBSERVER_HTTPHEADERS, deduplicates by event.actualSource, refuses external hosts by checking self.getTarget().matches(self.sf.urlFQDN(eventSource)), parses the header JSON, and emits TARGET_WEB_COOKIE when a cookie field is present.

The unit tests under test/unit/modules/ are mostly smoke tests for option/description parity, setup, watched event lists, produced event lists, and selected safety failures. For example, test/…/test_sfp_tool_nuclei.py, test/…/test_sfp_tool_testsslsh.py, test/…/test_sfp_builtwith.py, and test/…/test_sfp_whatcms.py assert that missing paths or API keys set errorState; test/…/test_sfp_whatcms.py also documents which HTTP status paths are expected to be fatal.

Coupling

Risks