DNS, domain intelligence, WHOIS, and registry modules
Overview
Owned paths/globs for this doc: modules/sfp_dns*.py, modules/sfp_arin.py, modules/sfp_ripe.py, modules/sfp_whois.py, modules/sfp_whoisology.py, modules/sfp_whoxy.py, modules/sfp_jsonwhoiscom.py, modules/sfp_reversewhois.py, modules/sfp_tldsearch.py, modules/sfp_zonefiles.py, modules/sfp_zoneh.py, modules/sfp_crobat_api.py, modules/sfp_crt.py, modules/sfp_certspotter.py, modules/sfp_securitytrails.py, modules/sfp_sublist3r.py, modules/sfp_subdomain_takeover.py, and the matching unit/integration test globs under test/unit/modules/ and test/integration/modules/ named in the assignment.
This area is SpiderFoot's DNS and domain-intelligence event pipeline. Most modules are SpiderFootPlugin classes that subscribe to domain, hostname, IP, e-mail, WHOIS, or certificate events, query DNS or an external intelligence service, and emit normalized events such as INTERNET_NAME, DOMAIN_NAME, IP_ADDRESS, PROVIDER_DNS, DOMAIN_WHOIS, RAW_RIR_DATA, or affiliate/co-host variants. Shared state is usually per-scan temporary storage created with self.tempStorage() in each module's setup() to de-duplicate event data, API pages, hostnames, or certificate IDs.
Core DNS resolution starts in modules/sfp_dnsresolve.py. It watches hostnames, IP addresses, owned netblocks, and raw content events including WHOIS/RIR data, DNS records, certificates, banners, and web content. For hostnames it calls self.sf.resolveHost() and self.sf.resolveHost6() and classifies results as target IP_ADDRESS/IPV6_ADDRESS or affiliate addresses. For IP addresses it reverse-resolves with self.sf.resolveIP() and classifies names by self.getTarget().matches(). For owned netblocks it enumerates hosts only when netblocklookup is enabled and the prefix length is no larger than maxnetblock or maxv6netblock. For raw text it scans around target names to extract subdomains. processHost() emits unresolved variants when DNS verification fails and calls processDomain() to emit DOMAIN_NAME, DOMAIN_NAME_PARENT, or AFFILIATE_DOMAIN_NAME.
Raw DNS and active DNS enumeration are split by purpose. modules/sfp_dnsraw.py queries CNAME, MX, NS, and TXT records for INTERNET_NAME, DOMAIN_NAME, and DOMAIN_NAME_PARENT, emits the literal RAW_DNS_RECORDS, extracts PROVIDER_MAIL, PROVIDER_DNS, DNS_TEXT, and DNS_SPF, and turns CNAME/MX/NS/SPF include domains into target or affiliate host events after optional resolution. modules/sfp_dnscommonsrv.py tries a fixed list of common SRV names for INTERNET_NAME and DOMAIN_NAME, emits DNS_SRV, and reports SRV targets as INTERNET_NAME or AFFILIATE_INTERNET_NAME. modules/sfp_dnsbrute.py brute-forces subdomains from bundled dictionaries and optional numeric suffix permutations, uses thread batches capped by _maxthreads, skips wildcard DNS when configured, and emits INTERNET_NAME only for resolved names. modules/sfp_tldsearch.py derives the target keyword with self.sf.domainKeyword(), tests it against configured _internettlds, optionally fetches the web page when activeonly is set, and emits SIMILARDOMAIN. modules/sfp_dnsneighbor.py walks nearby IPv4 addresses around an IP_ADDRESS, reverse-resolves them, and emits neighboring IP_ADDRESS or AFFILIATE_IPADDR events.
Zone transfer and passive DNS providers feed the same hostname/IP event model. modules/sfp_dnszonexfer.py consumes PROVIDER_DNS, resolves the nameserver to an IP to avoid resolver leakage, attempts dns.query.xfr() for each target name, emits the transferred zone as RAW_DNS_RECORDS, and extracts A/C-style rows into INTERNET_NAME. modules/sfp_dnsgrep.py queries https://dns.bufferover.run/dns for a DOMAIN_NAME, emits the full JSON as RAW_RIR_DATA, and extracts target-matching FDNS/RDNS domains as INTERNET_NAME or INTERNET_NAME_UNRESOLVED. modules/sfp_dnsdumpster.py, modules/sfp_sublist3r.py, and modules/sfp_crobat_api.py query DNSDumpster, Sublist3r, and Crobat/SonarSearch for subdomains; they accept DOMAIN_NAME and sometimes INTERNET_NAME, suppress duplicate parent/child scans, validate that results remain under the target, and emit resolved or unresolved INTERNET_NAME events. modules/sfp_dnsdb.py queries Farsight DNSDB for DOMAIN_NAME, IP_ADDRESS, and IPV6_ADDRESS, emits raw NDJSON-derived data as RAW_RIR_DATA, extracts DNS providers, mail providers, TXT records, target IP/IPv6 records, target hostnames, and bounded CO_HOSTED_SITE events with optional current-DNS validation.
Certificate transparency modules turn CT logs into hostnames and certificate facts. modules/sfp_crt.py queries crt.sh for DOMAIN_NAME or INTERNET_NAME, emits the JSON result as RAW_RIR_DATA, extracts name_value names, optionally fetches each certificate by crt.sh ID, parses it with self.sf.parseCert(), emits SSL_CERTIFICATE_RAW, and reports target names as INTERNET_NAME/INTERNET_NAME_UNRESOLVED, non-target names as CO_HOSTED_SITE, and domain-level DOMAIN_NAME or CO_HOSTED_SITE_DOMAIN. modules/sfp_certspotter.py uses the CertSpotter issuances API with Basic auth from api_key, paginates with the last issuance ID up to max_pages, expands dns_names, issuer, and cert, emits raw page data, certificate raw text, issuer, issued date, expiry/expiring events, and the same target-vs-co-host hostname/domain events.
WHOIS and registry modules provide ownership, registrar, contact, and netblock context. modules/sfp_whois.py performs python-whois lookups for DOMAIN_NAME, DOMAIN_NAME_PARENT, co-host/affiliate domains, and similar domains, and uses RDAP via ipwhois.IPWhois.lookup_rdap() for NETBLOCK_OWNER and NETBLOCKV6_OWNER; it emits typed WHOIS blobs (DOMAIN_WHOIS, NETBLOCK_WHOIS, AFFILIATE_DOMAIN_WHOIS, CO_HOSTED_SITE_DOMAIN_WHOIS, SIMILARDOMAIN_WHOIS) and DOMAIN_REGISTRAR. modules/sfp_jsonwhoiscom.py uses JsonWHOIS.com to emit RAW_RIR_DATA, DOMAIN_WHOIS or AFFILIATE_DOMAIN_WHOIS, DOMAIN_REGISTRAR, PROVIDER_DNS, contact e-mail/phone/address facts, and AFFILIATE_DOMAIN_UNREGISTERED. modules/sfp_arin.py queries ARIN POC endpoints for DOMAIN_NAME and HUMAN_NAME, emits RAW_RIR_DATA, contact HUMAN_NAME, and follow-up contact raw records. modules/sfp_ripe.py uses RIPE Stat endpoints to map IPs to prefixes, prefixes to ASNs, ASNs to announced prefixes/neighbours, and owner strings to target keywords; it emits NETBLOCK_OWNER/NETBLOCK_MEMBER, IPv6 equivalents, BGP_AS_OWNER/BGP_AS_MEMBER, and RAW_RIR_DATA.
Reverse-WHOIS and domain search providers expand from identity artifacts. modules/sfp_whoisology.py and modules/sfp_whoxy.py consume EMAILADDR, require API keys, query reverse WHOIS by e-mail, de-duplicate returned domain_name values, and emit AFFILIATE_INTERNET_NAME plus AFFILIATE_DOMAIN_NAME when the result is a registrable domain. modules/sfp_reversewhois.py scrapes reversewhois.io for a DOMAIN_NAME, extracts table rows with date-like cells, emits affiliate internet/domain events for non-target domains, and emits registrars as DOMAIN_REGISTRAR. modules/sfp_securitytrails.py has three paths: IP/IPv6/netblock lookup for hosting providers and co-hosts, e-mail lookup for affiliate hostnames/domains, and domain lookup for subdomains. It requires an API key and stops after maxcohost co-host events. modules/sfp_zonefiles.py queries ZoneFiles.io for a DOMAIN_NAME, emits raw data, verified IPs, DNS providers, target-owned e-mail addresses, phones, and SOFTWARE_USED technologies.
Risk/reputation checks consume the domains discovered by the modules above. modules/sfp_subdomain_takeover.py fetches and caches the upstream subjack fingerprints.json list, watches affiliate hostnames and unresolved affiliate hostnames, and emits AFFILIATE_INTERNET_NAME_HIJACKABLE when CNAME/fingerprint or NXDOMAIN rules match. modules/sfp_zoneh.py caches Zone-H's special defacements RSS for 48 hours and maps target, affiliate, and co-host name/IP events to the corresponding DEFACED_* event type. modules/sfp_dns_for_family.py resolves names through DNS for Family's resolvers and emits blacklisted target, affiliate, or co-host events when the block address 159.69.10.249 is returned.
Tests under the owned test/…/test_sfp_*.py and test/…/test_sfp_*.py globs are mostly module-contract smoke tests: they instantiate the module, check option descriptions match options, run setup(), and assert watchedEvents() and producedEvents() return lists. A few tests encode behavioral guardrails, such as test/…/test_sfp_crt.py distinguishing fatal and non-fatal crt.sh response codes and test/…/test_sfp_securitytrails.py requiring errorState when no API key is configured.
Coupling
- If event type names are changed in modules/sfp_dnsresolve.py, also update all producers that feed it and consumers that rely on its normalized outputs: modules/sfp_dnsraw.py, modules/sfp_crt.py, modules/sfp_certspotter.py, modules/sfp_dnsdb.py, modules/sfp_securitytrails.py, modules/sfp_zoneh.py, and modules/sfp_subdomain_takeover.py all depend on exact strings such as
INTERNET_NAME,AFFILIATE_INTERNET_NAME,CO_HOSTED_SITE,DOMAIN_NAME, and unresolved variants. - If
self.getTarget().matches()semantics or_internettldsparsing changes, verify domain classification in modules/sfp_dnsresolve.py, modules/sfp_dnsraw.py, modules/sfp_dnsdumpster.py, modules/sfp_sublist3r.py, modules/sfp_crobat_api.py, modules/sfp_crt.py, modules/sfp_certspotter.py, modules/sfp_reversewhois.py, modules/sfp_whoxy.py, modules/sfp_whoisology.py, and modules/sfp_tldsearch.py; those modules decide whether an output is a target, affiliate, co-host, parent domain, or similar domain by these helpers. - If DNS helper behavior changes (
resolveHost,resolveHost6,resolveIP,validateIP,checkDnsWildcard), verify modules/sfp_dnsresolve.py, modules/sfp_dnsraw.py, modules/sfp_dnsbrute.py, modules/sfp_dnsneighbor.py, modules/sfp_tldsearch.py, modules/sfp_dnsdb.py, modules/sfp_zonefiles.py, modules/sfp_crt.py, modules/sfp_certspotter.py, modules/sfp_crobat_api.py, modules/sfp_dnsdumpster.py, and modules/sfp_sublist3r.py; these modules silently use those helpers for de-duplication, wildcard suppression, and resolved-vs-unresolved event selection. - If certificate parsing changes in
self.sf.parseCert(), verify modules/sfp_crt.py and modules/sfp_certspotter.py; both expecttext,altnames,issuer,issued,expired,expiring, andexpirystrkeys and convert SANs into DNS/co-host events. - If an external API schema changes, patch the module-specific parser before changing emitted event contracts: DNSDB NDJSON parsing is in modules/sfp_dnsdb.py, SecurityTrails record/subdomain parsing in modules/sfp_securitytrails.py, CertSpotter issuance parsing in modules/sfp_certspotter.py, crt.sh JSON/certificate fetching in modules/sfp_crt.py, JsonWHOIS contact fields in modules/sfp_jsonwhoiscom.py, ZoneFiles field extraction in modules/sfp_zonefiles.py, and Crobat/Sublist3r/DNSDumpster result handling in their respective modules.
- If API key or rate-limit handling is adjusted, preserve
errorStatesemantics in modules/sfp_dnsdb.py, modules/sfp_certspotter.py, modules/sfp_securitytrails.py, modules/sfp_jsonwhoiscom.py, modules/sfp_zonefiles.py, modules/sfp_whoisology.py, and modules/sfp_whoxy.py; once set, most handlers return early for the rest of the scan.
Risks
- modules/sfp_whois.py does RDAP with
ipwhois.IPWhois(ip)and has an inline TODO that it does not use the configured proxy. In proxied or restricted scan environments, netblock WHOIS lookups can bypass expected network controls or fail differently fromself.sf.fetchUrl()-based modules. - modules/sfp_dnsresolve.py and modules/sfp_dnsbrute.py can generate large DNS workloads: netblock enumeration walks every address up to
maxnetblock/maxv6netblock, and brute force uses dictionary plus numeric suffix threading. If option bounds are relaxed, scans can become very slow or noisy and may amplify wildcard DNS false positives. - modules/sfp_securitytrails.py watches
NETBLOCK_OWNERbut its IP branch checks"NETLBLOCK_OWNER"(misspelled). Netblock events will not use the intended SecurityTrails IP/co-host lookup path unless this string mismatch is corrected. - modules/sfp_dnsdb.py assumes a multi-line DNSDB response and returns
records[1:-1], effectively discarding the first and last parsed JSON lines. If DNSDB changes wrapper/metadata line conventions, valid records can be silently dropped or malformed responses can look like no data. - modules/sfp_subdomain_takeover.py matches service CNAME fingerprints by checking whether each configured CNAME substring appears in the event hostname, not by resolving and inspecting the host's actual CNAME chain. This can miss takeover conditions where the vulnerable provider appears only in DNS records and can produce false positives when a provider string appears in a hostname.