Network, IP, Port, SSL, and Infrastructure Discovery Modules
Overview
This document covers the SpiderFoot modules that turn IP addresses, domains, netblocks, web content, HTTP headers, certificates, coordinates, and analytics IDs into infrastructure intelligence. Owned implementation paths/globs are: modules/sfp_bgpview.py, modules/sfp_bingsharedip.py, modules/sfp_binaryedge.py, modules/sfp_censys.py, modules/sfp_hackertarget.py, modules/sfp_hosting.py, modules/sfp_hostio.py, modules/sfp_ip*.py, modules/sfp_networksdb.py, modules/sfp_onyphe.py, modules/sfp_opendns.py, modules/sfp_opennic.py, modules/sfp_portscan_tcp.py, modules/sfp_robtex.py, modules/sfp_shodan.py, modules/sfp_sslcert.py, modules/sfp_strangeheaders.py, modules/sfp_webanalytics.py, modules/sfp_webframework.py, modules/sfp_webserver.py, modules/sfp_viewdns.py, modules/sfp_wigle.py, and modules/sfp_zetalytics.py. Owned tests/globs are test/…/test_sfp_bgpview.py, test/…/test_sfp_bingsharedip.py, test/…/test_sfp_binaryedge.py, test/…/test_sfp_censys.py, test/…/test_sfp_hackertarget.py, test/…/test_sfp_host*.py, test/…/test_sfp_ip*.py, test/…/test_sfp_networksdb.py, test/…/test_sfp_onyphe.py, test/…/test_sfp_open*.py, test/…/test_sfp_portscan_tcp.py, test/…/test_sfp_robtex.py, test/…/test_sfp_shodan.py, test/…/test_sfp_sslcert.py, test/…/test_sfp_strangeheaders.py, test/…/test_sfp_web*.py, test/…/test_sfp_viewdns.py, test/…/test_sfp_wigle.py, and test/…/test_sfp_zetalytics.py.
The common execution model is SpiderFoot event fan-out: each module declares watchedEvents(), deduplicates input with self.results or related temp storage in setup(), queries an external source or local network primitive from handleEvent(), then emits new SpiderFootEvent objects with the triggering event as parent. This makes the scan graph explicit: an IP_ADDRESS can become a NETBLOCK_MEMBER, BGP_AS_MEMBER, GEOINFO, CO_HOSTED_SITE, TCP_PORT_OPEN, WEBSERVER_HTTPHEADERS, or vulnerability event, and those outputs are then consumed by downstream modules.
BGP and network ownership discovery is centered on modules/sfp_bgpview.py, modules/sfp_censys.py, modules/sfp_networksdb.py, and modules/sfp_shodan.py. modules/sfp_bgpview.py queries https://api.bgpview.io/ip/, /asn/, and /prefix/; IP_ADDRESS and IPV6_ADDRESS results emit RAW_RIR_DATA, BGP_AS_MEMBER, and NETBLOCK_MEMBER or NETBLOCKV6_MEMBER, while ASN and prefix lookups can emit owner PHYSICAL_ADDRESS. modules/sfp_censys.py uses Censys Search v2 with HTTP Basic auth for host lookups, emits raw host records, location, open TCP/UDP ports, TCP banners, SOFTWARE_USED, WEBSERVER_HTTPHEADERS, OPERATING_SYSTEM, ASN, and BGP prefix events. modules/sfp_networksdb.py uses NetworksDB POST endpoints for IP info, geolocation, reverse DNS, and forward DNS; it turns IP input into netblocks, geolocation, co-hosts, and raw records, and turns INTERNET_NAME or DOMAIN_NAME into IP_ADDRESS/IPV6_ADDRESS. modules/sfp_shodan.py queries Shodan host, hostname search, and HTML search endpoints; IP and netblock inputs become raw records, OS/device/geolocation, open ports, banners, products, ASNs, and CVE-derived vulnerability events, while DOMAIN_NAME and selected WEB_ANALYTICS_ID values produce raw Shodan search data for graph context.
Shared-IP and passive DNS discovery is split across several providers with similar guardrails. modules/sfp_bingsharedip.py, modules/sfp_hackertarget.py, modules/sfp_robtex.py, modules/sfp_viewdns.py, modules/sfp_binaryedge.py, modules/sfp_networksdb.py, and modules/sfp_onyphe.py all emit CO_HOSTED_SITE for hosts found on a queried IP when the host is outside the target domain or when cohostsamedomain permits same-domain co-hosting. Most of them enforce maxcohost and optionally verify the hostname still resolves to the queried IP using self.sf.validateIP(). For netblock inputs, modules/sfp_bingsharedip.py, modules/sfp_hackertarget.py, modules/sfp_robtex.py, modules/sfp_binaryedge.py, modules/sfp_censys.py, modules/sfp_shodan.py, and modules/sfp_portscan_tcp.py expand IPNetwork(eventData) only when module options allow netblock lookup and the prefix length is at least the configured maximum-size threshold.
Internet scanning and exposure sources are API-backed. modules/sfp_binaryedge.py queries BinaryEdge for email leak events, domain subdomains, passive DNS, torrents, CVEs, and IP port/banner data; it emits compromised email events, in-scope INTERNET_NAME and DOMAIN_NAME, co-hosts, malicious IP findings, vulnerability severities via self.sf.cveInfo(), and TCP/UDP port events with banners. modules/sfp_censys.py and modules/sfp_shodan.py perform similar exposure parsing from Censys and Shodan host records. modules/sfp_onyphe.py queries Onyphe geoloc, pastries, threatlist, and vulnscan endpoints, filters by age_limit_days, emits location/coordinates, leaked content, malicious IPs, CVEs, and domain/co-host data embedded in geolocation results. modules/sfp_hostio.py queries https://host.io/api/full/<domain> and maps ipinfo, related.email, and web sections into IPs, coordinates, geolocation, emails, analytics IDs, web technologies, description titles, and raw records. modules/sfp_zetalytics.py queries Zonecruncher/Zetalytics endpoints for subdomains, hostname relationships, email domains, and email addresses, emitting in-scope names or affiliate domains plus raw data when an event is generated.
Geolocation, reputation, and hosting classification modules enrich IP-derived graph nodes. modules/sfp_ipinfo.py, modules/sfp_ipstack.py, modules/sfp_ipapicom.py, and modules/sfp_ipapico.py turn IP_ADDRESS and, where supported, IPV6_ADDRESS into GEOINFO, coordinates, and raw RIR-style data. modules/sfp_ipregistry.py adds PHYSICAL_COORDINATES, GEOINFO, MALICIOUS_IPADDR, and raw data from ipregistry security/location records. modules/sfp_ipqualityscore.py handles IP_ADDRESS, DOMAIN_NAME, EMAILADDR, and PHONE_NUMBER, emitting malicious IP/name/email/phone events when fraud_score, abuse, or bot signals cross abuse_score_threshold, plus disposable/compromised email, phone type, and geolocation. modules/sfp_hosting.py downloads and caches client9/ipcat datacenter ranges under cache key sfipcat, then emits PROVIDER_HOSTING when an IP_ADDRESS falls inside a known hosting provider range. modules/sfp_opendns.py resolves names through OpenDNS FamilyShield resolvers and maps OpenDNS sinkhole IPs to blacklisted and malicious internet-name/cohost events. modules/sfp_opennic.py resolves OpenNIC/peer TLDs through hard-coded OpenNIC resolvers and emits target or affiliate IPv4/IPv6 address events.
Active and semi-active surface discovery is handled by modules/sfp_portscan_tcp.py, modules/sfp_sslcert.py, and web fingerprint modules. modules/sfp_portscan_tcp.py accepts IP_ADDRESS and optionally NETBLOCK_OWNER, loads ports from an inline list, URL, or @ file via self.sf.optValueToData(), scans with self.sf.safeSocket() in batches of maxthreads, and emits TCP_PORT_OPEN plus TCP_PORT_OPEN_BANNER when data is read from the socket. modules/sfp_sslcert.py watches INTERNET_NAME, LINKED_URL_INTERNAL, and IP_ADDRESS, opens an SSL socket through self.sf.safeSSLSocket(), parses the certificate with self.sf.parseCert(), emits TCP_PORT_OPEN for direct host/IP inputs, raw certificate text, issuer/issued metadata, mismatch/expiry findings, SAN-derived INTERNET_NAME/INTERNET_NAME_UNRESOLVED or CO_HOSTED_SITE, and domain/co-host-domain events. modules/sfp_webserver.py consumes JSON WEBSERVER_HTTPHEADERS, limits processing to target FQDNs via event.actualSource, emits server banners, technology hints, and internal/external links from location and CSP headers. modules/sfp_strangeheaders.py consumes the same header event and emits WEBSERVER_STRANGEHEADER for non-standard header names. modules/sfp_webframework.py consumes target TARGET_WEB_CONTENT from sfp_spider only and emits URL_WEB_FRAMEWORK for JavaScript/CMS regex matches. modules/sfp_webanalytics.py extracts WEB_ANALYTICS_ID from TARGET_WEB_CONTENT and DNS_TEXT, including Google Analytics/AdSense/Tag Manager/site verification and many DNS TXT domain-verification formats; modules/sfp_shodan.py can later pivot selected Google analytics/site-verification IDs into Shodan HTML searches.
Location-to-secondary-network discovery is handled by modules/sfp_wigle.py. It watches PHYSICAL_COORDINATES, validates the configured base64 WiGLE credential, queries https://api.wigle.net/api/v2/network/search with a small latitude/longitude variance and optional days_limit, and emits WIFI_ACCESS_POINT records as SSID plus net ID. Because PHYSICAL_COORDINATES can come from modules/sfp_hostio.py, modules/sfp_ipregistry.py, modules/sfp_ipapicom.py, and modules/sfp_onyphe.py, WiGLE findings are downstream of both domain and IP enrichment flows.
The unit tests under the owned test/…/test_sfp_*.py paths mostly validate module option metadata, setup, watched/produced event declarations, and no-API-key errorState behavior for API-backed modules. Some tests exercise transformations directly, such as test/…/test_sfp_webanalytics.py verifying TARGET_WEB_CONTENT and DNS_TEXT analytics extraction, and test/…/test_sfp_censys.py distinguishing Censys nonfatal (200, 400, 404) and fatal (401, 402, 403, 429, 500, 502, 503) response handling.
Coupling
- If a provider module changes
CO_HOSTED_SITEemission rules, check modules/sfp_bingsharedip.py, modules/sfp_hackertarget.py, modules/sfp_robtex.py, modules/sfp_viewdns.py, modules/sfp_binaryedge.py, modules/sfp_networksdb.py, modules/sfp_onyphe.py, and modules/sfp_sslcert.py together. They each enforce target-domain suppression, verification, andmaxcohostslightly differently, so changing one module's semantics can silently alter graph density compared with the others. - If the shape of
WEBSERVER_HTTPHEADERSchanges, update both modules/sfp_webserver.py and modules/sfp_strangeheaders.py; both expectevent.datato be a JSON object string and useevent.actualSourceto decide whether the headers belong to an in-scope target URL/host. modules/sfp_censys.py and modules/sfp_hackertarget.py also create this event type, so their emitted JSON/header casing must stay compatible with the consumers. - If
WEB_ANALYTICS_IDformatting changes in modules/sfp_webanalytics.py or modules/sfp_hostio.py, verify modules/sfp_shodan.py; Shodan parses IDs by splitting on": "and only supportsGoogle AdSense,Google Analytics, andGoogle Site Verificationprefixes for HTML search pivots. - If netblock scanning thresholds or option names change, check every module that expands
IPNetwork(eventData): modules/sfp_binaryedge.py, modules/sfp_censys.py, modules/sfp_hackertarget.py, modules/sfp_portscan_tcp.py, modules/sfp_robtex.py, modules/sfp_shodan.py, and modules/sfp_bingsharedip.py. They all rely on prefix-length comparisons to prevent unbounded API calls or active scans, but they use separate option names such asmaxnetblock,maxv6netblock,maxsubnet, andnetblockscanmax. - If
self.sf.cveInfo()changes its return type, update modules/sfp_binaryedge.py, modules/sfp_onyphe.py, and modules/sfp_shodan.py; each expects(event_type, event_text)and immediately emits vulnerability events from provider CVE identifiers. - If certificate parsing output changes in
self.sf.parseCert(), check modules/sfp_sslcert.py; it expects keys includingtext,issued,issuer,mismatch,hosts,altnames,expired,expiring, andexpirystr, and missing or renamed keys will suppress certificate graph edges. - If event resolution helpers change behavior, check modules/sfp_opennic.py, modules/sfp_networksdb.py, modules/sfp_sslcert.py, modules/sfp_binaryedge.py, modules/sfp_onyphe.py, and modules/sfp_zetalytics.py; these modules distinguish resolved vs unresolved internet names and target vs affiliate assets using
resolveHost(),resolveHost6(),validateIP(),validIP(),validIP6(),validIpNetwork(),isDomain(), andgetTarget().matches().
Risks
- modules/sfp_portscan_tcp.py, modules/sfp_binaryedge.py, modules/sfp_censys.py, modules/sfp_hackertarget.py, modules/sfp_robtex.py, and modules/sfp_shodan.py can multiply one netblock event into per-IP scans or API calls. If prefix-length guard logic or defaults are loosened, a single
NETBLOCK_OWNERcan cause excessive active scanning, rate-limit exhaustion, or very large scan graphs. - modules/sfp_zetalytics.py has a fragile hostname parser in
generate_hostname_events(): it checksisinstance("qname", str)instead of checking theqnamevalue. If a Zetalytics result lacksqname,Nonecan be added to the hostname set and passed intoverify_emit_internet_name(), where target matching and event emission may fail or suppress valid sibling hostnames. - modules/sfp_onyphe.py compares
res["code"] == 400while the other provider modules treat response codes as strings such as"429"and"403". Ifself.sf.fetchUrl()returns string HTTP codes consistently, an Onyphe400response will skip the intended API-key/request error path and fall through to JSON parsing, producing a less precise error and possibly leavingerrorStateinconsistent. - modules/sfp_strangeheaders.py and modules/sfp_webserver.py trust
event.actualSourcefor in-scope filtering. If a producer emitsWEBSERVER_HTTPHEADERSwithout a usable URL/host inactualSource, the consumers may either skip legitimate target headers or process headers against the wrong target FQDN. - modules/sfp_sslcert.py deduplicates only by
fqdn, not byfqdn:port. If the same host appears through multipleLINKED_URL_INTERNALevents with different HTTPS ports, only the first certificate is inspected, so alternate-port certificates and port-specific SANs can be missed.