Introducing a new way to review your PRs.

Read the announcement

smicallef / spiderfoot

SpiderFoot automates OSINT for threat intelligence and mapping your attack surface.

View on GitHub

13 documents · Updated Nov 16, 2026

docs / search-crawling-and-content-modules

Search, Crawling, Archive, Paste, Code, and Content Modules

Overview

This doc owns these paths:

These modules extend SpiderFoot scans by turning names, domains, hostnames, URLs, page bodies, public indexes, code search results, paste sites, archives, and reputation services into follow-on events. The main recursive crawl entry point is modules/sfp_spider.py: it watches INTERNET_NAME and LINKED_URL_INTERNAL, probes configured URL prefixes for hostnames, fetches pages with self.sf.fetchUrl, emits HTTP headers/status/content/content-type, extracts links via SpiderFootHelpers.extractLinksFromHtml, emits internal/external link events, then loops through same-target links until maxpages or maxlevels is reached. It stores urlEvents, fetchedPages, and per-site cookies in temp storage so later content and link events are parented back to the event that discovered each URL. modules/sfp_spider.py also gates recursion with target matching, optional subdomain exclusion, optional /~user filtering, optional robots.txt-derived rules, filtered file extensions, filtered MIME prefixes, and duplicate-link suppression.

Several modules consume the spider's URL and content events. modules/sfp_pageinfo.py only trusts TARGET_WEB_CONTENT emitted by sfp_spider, uses event.actualSource as the page URL, skips CSS/JS assets, classifies pages with regex groups into URL_JAVASCRIPT, URL_FORM, URL_PASSWORD, URL_UPLOAD, URL_JAVA_APPLET, URL_FLASH, or URL_STATIC, and emits PROVIDER_JAVASCRIPT for external script sources. modules/sfp_intfiles.py watches LINKED_URL_INTERNAL and flags configured document/archive extensions as INTERESTING_FILE. modules/sfp_filemeta.py then fetches LINKED_URL_INTERNAL or INTERESTING_FILE targets with a larger timeout and a 10 MB size limit and extracts metadata from PDFs (PyPDF2), DOCX (docx), PPTX (pptx), and images (exifread), emitting RAW_FILE_META_DATA plus SOFTWARE_USED when producer/creator/application/software fields are found. modules/sfp_junkfiles.py is the invasive companion to crawling: for each internal URL it HEAD-probes likely backup extensions, well-known filenames in the containing directory, and directory archive variants, then validates positives by checking that a randomized sibling URL returns 404 before emitting JUNK_FILE.

Public search and archive modules seed more internal URLs and descriptive data. modules/sfp_googlesearch.py and modules/sfp_bingsearch.py require API keys, search site:<internet-name> through the framework's Google/Bing iterator helpers, emit only URLs whose FQDN ends with the searched name as LINKED_URL_INTERNAL, and attach the raw iterator response as RAW_RIR_DATA if any internal link was found. modules/sfp_commoncrawl.py first scrapes the latest Common Crawl index IDs from https://index.commoncrawl.org/, queries the most recent configured indexes for <target>/* JSON lines, normalizes a trailing-dot host quirk, and emits each unique indexed URL as LINKED_URL_INTERNAL. modules/sfp_archiveorg.py is downstream of content classifiers and interesting-file detection: for selected event types such as INTERESTING_FILE, URL_PASSWORD, URL_FORM, and URL_STATIC, it calls the Wayback availability API for each configured farback day offset and emits the closest unique snapshot as the corresponding *_HISTORIC event.

Code and repository search modules extract emails, hosts, links, and repository evidence from public source indexes. modules/sfp_grep_app.py searches grep.app for a DOMAIN_NAME, stores each hit as RAW_RIR_DATA, strips <mark> tags from snippets, extracts target-related URLs as LINKED_URL_INTERNAL, extracts target-related email addresses as EMAILADDR or EMAILADDR_GENERIC, then DNS-resolves discovered hosts before emitting INTERNET_NAME, DOMAIN_NAME, or INTERNET_NAME_UNRESOLVED. modules/sfp_searchcode.py queries searchcode pages for a DOMAIN_NAME, extracts emails from the result set, extracts URLs from each result's lines, emits matching links and host events, and emits PUBLIC_CODE_REPO plus RAW_RIR_DATA for repositories whose result contains the original domain. modules/sfp_github.py searches GitHub repositories and users for a domain-derived keyword or USERNAME, optionally requires repository names to match exactly, emits PUBLIC_CODE_REPO, and can enrich GitHub SOCIAL_MEDIA profile events with possible full names (RAW_RIR_DATA) and GEOINFO by calling the GitHub users API.

Paste, leak, discussion, and public-content modules find mentions rather than crawl the target site directly. modules/sfp_pastebin.py uses Google Custom Search to find pastebin.com pages containing a domain, hostname, or email, verifies the fetched page still contains the query with a bounded regex, and emits LEAKSITE_URL followed by full LEAKSITE_CONTENT. modules/sfp_psbdmp.py calls psbdmp's email/domain API, translates returned paste IDs into Pastebin URLs, and verifies fetched content before emitting LEAKSITE_CONTENT. modules/sfp_wikileaks.py builds a quoted query against WikiLeaks search, applies a daysback lower bound and optional external sources, follows pagination up to a fail-safe limit, extracts wikileaks.org and cryptome.org links from result pages, and emits nested leak document URLs. modules/sfp_stackoverflow.py queries StackExchange search excerpts for a domain, emits each result as RAW_RIR_DATA, extracts emails, fetches question owners to emit usernames or possible names, and attempts to extract affiliate IP addresses from the excerpt/body text. modules/sfp_wikipediaedits.py calls Wikipedia's feedcontributions API for IP_ADDRESS or USERNAME, optionally constrained by days_limit, parses <link> tags from the feed, skips contribution-profile links, and emits WIKIPEDIA_PAGE_EDIT links.

Darknet search modules all search for target mentions and optionally verify fetched onion content before emitting DARKNET_MENTION_URL and a nearby DARKNET_MENTION_CONTENT snippet. modules/sfp_ahmia.py, modules/sfp_onionsearchengine.py, and modules/sfp_torch.py query their respective Tor search engines for DOMAIN_NAME, HUMAN_NAME, or EMAILADDR, skip human-name searches when fullnames is disabled, filter to .onion results, and only emit verified results when fetchlinks is true and the fetched page still contains the event data. modules/sfp_onioncity.py searches site:onion.link through Google Custom Search, stores the rendered Google result page as RAW_RIR_DATA, converts .onion.link URLs back to .onion, and performs the same optional content verification.

Other public-content/reputation modules in this scope enrich or qualify discovered names. modules/sfp_duckduckgo.py queries DuckDuckGo Instant Answer for domains, internet names, and affiliate internet names, optionally normalizing affiliate hostnames to parent domains, then emits abstract and related-topic descriptions with affiliate event variants. modules/sfp_emailcrawlr.py requires an API key, queries EmailCrawlr's domain endpoint, emits raw response data, target-owned emails (EMAILADDR or EMAILADDR_GENERIC), phone numbers, possible full names as raw data, and city/country GEOINFO. modules/sfp_grayhatwarfare.py requires an API key, derives a keyword from the domain, pages through open bucket results, and emits buckets whose first label starts or ends with the keyword as CLOUD_STORAGE_BUCKET, CLOUD_STORAGE_BUCKET_OPEN, and RAW_RIR_DATA. modules/sfp_punkspider.py hashes an INTERNET_NAME, calls PunkSpider's partial-hash API, and emits non-zero vulnerability counts as VULNERABILITY_GENERAL. modules/sfp_yandexdns.py resolves internet names, affiliate names, or co-hosted sites through Yandex family DNS servers and maps known sinkhole answers to blacklist and malicious events.

The unit tests in the owned test/…/test_sfp_*.py paths mostly assert setup, option/description parity, and watched/produced event list shape. A few tests pin important error or event behavior: Google/Bing/Pastebin/EmailCrawlr no-key paths set errorState, EmailCrawlr distinguishes nonfatal 200/404 from fatal HTTP responses, GitHub ignores non-GitHub SOCIAL_MEDIA, and Intfiles emits only for configured interesting extensions. The spider integration test in test/…/test_sfp_spider.py is currently skipped.

Coupling

Risks

  • modules/sfp_spider.py is high-churn and central to multiple downstream modules; git history shows repeated spider changes for false-positive links, complexity reduction, robots parsing, content typing, cookie/redirect handling, and filtered files. A regression in processUrl, cleanLinks, linkNotify, or contentNotify can either stop recursive crawling or silently starve sfp_pageinfo, sfp_intfiles, sfp_filemeta, sfp_junkfiles, and sfp_archiveorg of events.
  • modules/sfp_spider.py has a fragile robots.txt check in cleanLinks: type(blocked).lower(blocked) is not a normal string-lower call. If robotsonly is enabled and a link base has rules, this can break filtering rather than safely skipping blocked paths.
  • modules/sfp_stackoverflow.py has extractor bugs that can surface only when real API snippets contain matching data: extractIP4s and extractIP6s initialize ips as a list but call ips.add, and the IPv4 event type emitted in handleEvent does not match the declared produced event. If StackOverflow returns IP-like excerpts, IP extraction can fail or downstream modules can miss the event.
  • modules/sfp_wikipediaedits.py parses feed links with HTMLParser().unescape, an API removed from modern Python versions. If this path runs under a runtime without that method, Wikipedia edit discovery raises during response processing and emits no WIKIPEDIA_PAGE_EDIT events.
  • modules/sfp_filemeta.py depends on third-party parsers (PyPDF2, docx, pptx, exifread) over untrusted remote content. If a parser API changes or raises on malformed but fetchable files, the module returns from the whole event after logging an error, so later metadata fields or extensions for that same URL are not attempted.