Knowledge Base updated: 2026-11-16
Knowledge Base Index
1. Orientation
SpiderFoot is an OSINT automation framework: scans start from CLI or web UI entrypoints, instantiate the core scan/runtime, run SpiderFoot modules, emit events, persist results, and optionally derive correlations.
Main parts:
spiderfoot/— core framework primitives: helpers, logging, DB, events, plugins, targets, threadpool, and correlation engine.modules/— SpiderFoot plugin modules that enrich targets through DNS, threat intel, crawling, identity, network, cloud, and external-tool integrations.correlations/— YAML rule packs used to derive higher-level findings from scan output.- sf*.py — executable CLI, scanner, library, and web UI entrypoints.
spiderfoot/templates/andspiderfoot/static/— web UI templates and browser assets.test/— unit, integration, and acceptance coverage for core flows and modules.
Critical state lives primarily in the SQLite persistence layer managed by spiderfoot/db.py; it stores scan metadata, scan configuration, events/results, logs, and derived correlation output. Runtime scan state is passed as SpiderFootEvent objects through plugin callbacks and threadpool execution, while module configuration/API credentials are loaded from scan options rather than a separate service.
2. Architecture diagrams
flowchart TD
subgraph Entrypoints
CLI[sf.py / sfcli.py / sfscan.py]
Web[sfwebui.py]
end
subgraph Core[spiderfoot/]
Helpers[helpers.py]
Runtime[plugin.py / event.py / target.py / threadpool.py]
DB[db.py]
Corr[correlation.py]
end
subgraph Knowledge[Rules and assets]
Rules[correlations/*.yaml]
Dicts[spiderfoot/dicts/**]
UI[templates + static assets]
end
subgraph Modules[modules/]
DNS[DNS / WHOIS modules]
Net[Network / IP modules]
Threat[Threat intel modules]
Search[Search / crawling modules]
Identity[Identity / social modules]
Tools[External tool / cloud modules]
Storage[Storage modules]
end
External[External APIs / tools / public data]
CLI --> Runtime
Web --> Runtime
Web --> UI
Runtime --> Helpers
Runtime --> Modules
Modules --> External
Modules --> Storage
Storage --> DB
Runtime --> DB
Corr --> Rules
Corr --> DB
Modules --> Dicts
flowchart LR
Target[Scan target] --> Entrypoint[CLI or Web UI]
Entrypoint --> Configure[Load options and module set]
Configure --> Runtime[Plugin runtime]
Runtime --> Event[SpiderFootEvent]
Event --> Module[Interested modules]
Module --> NewEvents[New enrichment events]
NewEvents --> Runtime
Runtime --> Store[Storage module / db.py]
Store --> Results[Scan results]
flowchart TD
RawEvents[Persisted scan events] --> Correlator[spiderfoot/correlation.py]
Rules[correlations/*.yaml] --> Correlator
Correlator --> Matches[Matched conditions]
Matches --> Findings[Correlation output]
Findings --> DB[spiderfoot/db.py]
DB --> UI[Web UI / reports]
DB --> CLI[CLI reports]
sequenceDiagram
participant Browser
participant WebUI as sfwebui.py
participant Runtime as Scan runtime
participant Modules as SpiderFoot modules
participant DB as spiderfoot/db.py
Browser->>WebUI: configure/start scan
WebUI->>Runtime: create scan with target/options
Runtime->>Modules: dispatch events to watched modules
Modules-->>Runtime: emit enrichment events
Runtime->>DB: persist scan events/logs
Browser->>WebUI: poll scan status/results
WebUI->>DB: query scan data
DB-->>WebUI: rows/results
WebUI-->>Browser: rendered templates or JSON
3. Glossary
- SpiderFoot module /
sfp_*— a plugin undermodules/that consumes and emits SpiderFoot event types. SpiderFootEvent— the event object passed between modules during a scan.- Storage module — a special module such as sfp__stor_db.py or sfp__stor_stdout.py that records or emits scan events.
- Correlation rule — a YAML definition under
correlations/that derives a higher-level finding from existing scan events. - Target — the scan subject modeled by spiderfoot/target.py, such as a domain, IP, subnet, email, or human name.
4. Codebase-wide invariants
- SpiderFoot plugin modules live as flat Python files named modules/sfp_*.py; storage plugins are distinguished by the double-underscore prefix pattern shown by modules/sfp__stor_db.py and modules/sfp__stor_stdout.py.
- Core framework primitives are centralized under spiderfoot/*.py, while plugin implementations are outside that package under modules/*.py.
- Correlation definitions are YAML files under correlations/*.yaml, with correlations/README.md and correlations/template.yaml documenting and exemplifying the rule pack shape.
- Web UI presentation assets are split between server code in sfwebui.py, templates in
spiderfoot/templates/*.tmpl, and browser assets inspiderfoot/static/js/**andspiderfoot/static/css/**. - Module tests mirror module filenames using test/…/test_sfp_*.py and test/…/test_sfp_*.py, matching the plugin naming convention under modules/sfp_*.py.
5. Routing table
- spiderfoot/__init__.py, spiderfoot/__version__.py, spiderfoot/helpers.py, spiderfoot/logger.py,
spiderfoot/dicts/**, README.md, requirements.txt,setup.cfg,VERSION-> docs/core-engine.md - spiderfoot/plugin.py, spiderfoot/event.py, spiderfoot/target.py, spiderfoot/threadpool.py, modules/sfp_template.py, related unit tests under
test/unit/spiderfoot/test_spiderfoot{plugin,event,target,threadpool}.py-> docs/plugin-event-target-runtime.md - spiderfoot/db.py, modules/sfp__stor_db.py, modules/sfp__stor_stdout.py, related storage and DB tests -> docs/database-and-storage.md
- spiderfoot/correlation.py,
correlations/**, test/…/test_spiderfootcorrelator.py -> docs/correlation-rules.md - sf.py, sfcli.py, sfscan.py, sflib.py, CLI/scanner integration and unit tests -> docs/cli-and-scan-entrypoints.md
- sfwebui.py,
spiderfoot/templates/**,spiderfoot/static/js/**,spiderfoot/static/css/**,spiderfoot/static/img/**, spiderfoot/…/package.json, test/…/test_spiderfootwebui.py, test/…/test_sfwebui.py,test/acceptance/**-> docs/web-ui.md - modules/sfp_dns*.py, modules/sfp_arin.py, modules/sfp_ripe.py, modules/sfp_whois*.py, modules/sfp_jsonwhoiscom.py, modules/sfp_reversewhois.py, modules/sfp_tldsearch.py, modules/sfp_zone*.py, modules/sfp_crobat_api.py, modules/sfp_crt.py, modules/sfp_certspotter.py, modules/sfp_securitytrails.py, modules/sfp_sublist3r.py, modules/sfp_subdomain_takeover.py, and matching module tests -> docs/dns-domain-and-whois-modules.md
- modules/sfp_bgpview.py, modules/sfp_bingsharedip.py, modules/sfp_binaryedge.py, modules/sfp_censys.py, modules/sfp_hackertarget.py, modules/sfp_host*.py, modules/sfp_ip*.py, modules/sfp_networksdb.py, modules/sfp_onyphe.py, modules/sfp_opendns.py, modules/sfp_opennic.py, modules/sfp_portscan_tcp.py, modules/sfp_robtex.py, modules/sfp_shodan.py, modules/sfp_sslcert.py, modules/sfp_strangeheaders.py, modules/sfp_web*.py, modules/sfp_viewdns.py, modules/sfp_wigle.py, modules/sfp_zetalytics.py, and matching module tests -> docs/network-ip-and-port-modules.md
- modules/sfp_abuse*.py, modules/sfp_alienvault*.py, modules/sfp_bitcoinabuse.py, modules/sfp_blocklistde.py, modules/sfp_botscout.py, modules/sfp_botvrij.py, modules/sfp_cinsscore.py, modules/sfp_circllu.py, modules/sfp_citadel.py, modules/sfp_cleanbrowsing.py, modules/sfp_coinblocker.py, modules/sfp_comodo.py, modules/sfp_cybercrimetracker.py, modules/sfp_dronebl.py, modules/sfp_emergingthreats.py, modules/sfp_fortinet.py, modules/sfp_fraudguard.py, modules/sfp_fsecure_riddler.py, modules/sfp_google*.py, modules/sfp_greensnow.py, modules/sfp_greynoise*.py, modules/sfp_honeypot.py, modules/sfp_hybrid_analysis.py, modules/sfp_isc.py, modules/sfp_koodous.py, modules/sfp_maltiverse.py, modules/sfp_malwarepatrol.py, modules/sfp_metadefender.py, modules/sfp_openphish.py, modules/sfp_phishstats.py, modules/sfp_phishtank.py, modules/sfp_pulsedive.py, modules/sfp_sorbs.py, modules/sfp_spamcop.py, modules/sfp_spamhaus.py, modules/sfp_spur.py, modules/sfp_surbl.py, modules/sfp_talosintel.py, modules/sfp_threat*.py, modules/sfp_uceprotect.py, modules/sfp_urlscan.py, modules/sfp_virustotal.py, modules/sfp_voipbl.py, modules/sfp_vxvault.py, modules/sfp_xforce.py, and matching module tests -> docs/threat-intel-and-reputation-modules.md
- modules/sfp_ahmia.py, modules/sfp_archiveorg.py, modules/sfp_bingsearch.py, modules/sfp_commoncrawl.py, modules/sfp_duckduckgo.py, modules/sfp_emailcrawlr.py, modules/sfp_filemeta.py, modules/sfp_github.py, modules/sfp_googlesearch.py, modules/sfp_grayhatwarfare.py, modules/sfp_grep_app.py, modules/sfp_intfiles.py, modules/sfp_junkfiles.py, modules/sfp_onioncity.py, modules/sfp_onionsearchengine.py, modules/sfp_pageinfo.py, modules/sfp_pastebin.py, modules/sfp_psbdmp.py, modules/sfp_punkspider.py, modules/sfp_searchcode.py, modules/sfp_spider.py, modules/sfp_stackoverflow.py, modules/sfp_torch.py, modules/sfp_wikileaks.py, modules/sfp_wikipediaedits.py, modules/sfp_yandexdns.py, and matching module tests -> docs/search-crawling-and-content-modules.md
- modules/sfp_accounts.py, modules/sfp_abstractapi.py, modules/sfp_apple_itunes.py, modules/sfp_callername.py, modules/sfp_clearbit.py, modules/sfp_company.py, modules/sfp_crossref.py, modules/sfp_debounce.py, modules/sfp_dehashed.py, modules/sfp_email*.py, modules/sfp_fullcontact.py, modules/sfp_gleif.py, modules/sfp_gravatar.py, modules/sfp_h1nobbdde.py, modules/sfp_haveibeenpwned.py, modules/sfp_hunter.py, modules/sfp_keybase.py, modules/sfp_leakix.py, modules/sfp_myspace.py, modules/sfp_nameapi.py, modules/sfp_names.py, modules/sfp_neutrinoapi.py, modules/sfp_numverify.py, modules/sfp_opencorporates.py, modules/sfp_openstreetmap.py, modules/sfp_phone.py, modules/sfp_skymem.py, modules/sfp_slideshare.py, modules/sfp_snov.py, modules/sfp_social*.py, modules/sfp_textmagic.py, modules/sfp_trumail.py, modules/sfp_twilio.py, modules/sfp_twitter.py, modules/sfp_venmo.py, and matching module tests -> docs/identity-email-account-and-social-modules.md
- modules/sfp_azureblobstorage.py, modules/sfp_digitaloceanspace.py, modules/sfp_googleobjectstorage.py, modules/sfp_s3bucket.py, modules/sfp_tool_*.py, modules/sfp_projectdiscovery.py, modules/sfp_crxcavator.py, modules/sfp_builtwith.py, modules/sfp_whatcms.py, modules/sfp_cookie.py, modules/sfp_ad*.py, modules/sfp_dns_for_family.py, modules/sfp_quad9.py, modules/sfp_stevenblack_hosts.py, and matching module tests -> docs/external-tool-and-cloud-modules.md
- modules/sfp_base64.py, modules/sfp_binstring.py, modules/sfp_hashes.py, modules/sfp_pgp.py, modules/sfp_errors.py, and other content-decoding or parser-style modules -> docs/search-crawling-and-content-modules.md
- modules/sfp_bitcoin.py, modules/sfp_blockchain.py, modules/sfp_ethereum.py, modules/sfp_etherscan.py, and cryptocurrency/address enrichment modules -> docs/threat-intel-and-reputation-modules.md
- modules/sfp_creditcard.py, modules/sfp_countryname.py, modules/sfp_iban.py, modules/sfp_mnemonic.py, modules/sfp_similar.py, modules/sfp_trashpanda.py, and specialized entity/attribute enrichment modules -> docs/identity-email-account-and-social-modules.md
Dockerfile*, docker-compose*.yml,.dockerignore,generate-certificate-> docs/cli-and-scan-entrypoints.mddocs/**, excluding this knowledge base, -> docs/core-engine.md- Any change to module lifecycle hooks, watched/produced events, target matching, or scan concurrency -> docs/plugin-event-target-runtime.md
- Any change to scan persistence, report queries, result storage, or stdout/database output behavior -> docs/database-and-storage.md
- Any new modules/sfp_*.py plugin -> route by behavior to the relevant module-family doc above; if it introduces a new family, start from docs/plugin-event-target-runtime.md and add/update a family doc.
- Any change to correlation semantics or YAML rule shape -> docs/correlation-rules.md
- Any change to web routes, web API responses, templates, or browser-side scan interactions -> docs/web-ui.md