Introducing a new way to review your PRs.

Read the announcement

smicallef / spiderfoot

SpiderFoot automates OSINT for threat intelligence and mapping your attack surface.

View on GitHub

13 documents · Updated Nov 16, 2026

overview

Knowledge Base updated: 2026-11-16

Knowledge Base Index

1. Orientation

SpiderFoot is an OSINT automation framework: scans start from CLI or web UI entrypoints, instantiate the core scan/runtime, run SpiderFoot modules, emit events, persist results, and optionally derive correlations.

Main parts:

  • spiderfoot/ — core framework primitives: helpers, logging, DB, events, plugins, targets, threadpool, and correlation engine.
  • modules/ — SpiderFoot plugin modules that enrich targets through DNS, threat intel, crawling, identity, network, cloud, and external-tool integrations.
  • correlations/ — YAML rule packs used to derive higher-level findings from scan output.
  • sf*.py — executable CLI, scanner, library, and web UI entrypoints.
  • spiderfoot/templates/ and spiderfoot/static/ — web UI templates and browser assets.
  • test/ — unit, integration, and acceptance coverage for core flows and modules.

Critical state lives primarily in the SQLite persistence layer managed by spiderfoot/db.py; it stores scan metadata, scan configuration, events/results, logs, and derived correlation output. Runtime scan state is passed as SpiderFootEvent objects through plugin callbacks and threadpool execution, while module configuration/API credentials are loaded from scan options rather than a separate service.

2. Architecture diagrams

flowchart TD
  subgraph Entrypoints
    CLI[sf.py / sfcli.py / sfscan.py]
    Web[sfwebui.py]
  end

  subgraph Core[spiderfoot/]
    Helpers[helpers.py]
    Runtime[plugin.py / event.py / target.py / threadpool.py]
    DB[db.py]
    Corr[correlation.py]
  end

  subgraph Knowledge[Rules and assets]
    Rules[correlations/*.yaml]
    Dicts[spiderfoot/dicts/**]
    UI[templates + static assets]
  end

  subgraph Modules[modules/]
    DNS[DNS / WHOIS modules]
    Net[Network / IP modules]
    Threat[Threat intel modules]
    Search[Search / crawling modules]
    Identity[Identity / social modules]
    Tools[External tool / cloud modules]
    Storage[Storage modules]
  end

  External[External APIs / tools / public data]

  CLI --> Runtime
  Web --> Runtime
  Web --> UI
  Runtime --> Helpers
  Runtime --> Modules
  Modules --> External
  Modules --> Storage
  Storage --> DB
  Runtime --> DB
  Corr --> Rules
  Corr --> DB
  Modules --> Dicts
flowchart LR
  Target[Scan target] --> Entrypoint[CLI or Web UI]
  Entrypoint --> Configure[Load options and module set]
  Configure --> Runtime[Plugin runtime]
  Runtime --> Event[SpiderFootEvent]
  Event --> Module[Interested modules]
  Module --> NewEvents[New enrichment events]
  NewEvents --> Runtime
  Runtime --> Store[Storage module / db.py]
  Store --> Results[Scan results]
flowchart TD
  RawEvents[Persisted scan events] --> Correlator[spiderfoot/correlation.py]
  Rules[correlations/*.yaml] --> Correlator
  Correlator --> Matches[Matched conditions]
  Matches --> Findings[Correlation output]
  Findings --> DB[spiderfoot/db.py]
  DB --> UI[Web UI / reports]
  DB --> CLI[CLI reports]
sequenceDiagram
  participant Browser
  participant WebUI as sfwebui.py
  participant Runtime as Scan runtime
  participant Modules as SpiderFoot modules
  participant DB as spiderfoot/db.py

  Browser->>WebUI: configure/start scan
  WebUI->>Runtime: create scan with target/options
  Runtime->>Modules: dispatch events to watched modules
  Modules-->>Runtime: emit enrichment events
  Runtime->>DB: persist scan events/logs
  Browser->>WebUI: poll scan status/results
  WebUI->>DB: query scan data
  DB-->>WebUI: rows/results
  WebUI-->>Browser: rendered templates or JSON

3. Glossary

  • SpiderFoot module / sfp_* — a plugin under modules/ that consumes and emits SpiderFoot event types.
  • SpiderFootEvent — the event object passed between modules during a scan.
  • Storage module — a special module such as sfp__stor_db.py or sfp__stor_stdout.py that records or emits scan events.
  • Correlation rule — a YAML definition under correlations/ that derives a higher-level finding from existing scan events.
  • Target — the scan subject modeled by spiderfoot/target.py, such as a domain, IP, subnet, email, or human name.

4. Codebase-wide invariants

5. Routing table