Introducing a new way to review your PRs.

Read the announcement

smicallef / spiderfoot

SpiderFoot automates OSINT for threat intelligence and mapping your attack surface.

View on GitHub

13 documents · Updated Nov 16, 2026

docs / core-engine

Core Engine Primitives

Overview

Owned paths/globs: spiderfoot/__init__.py, spiderfoot/__version__.py, spiderfoot/helpers.py, spiderfoot/logger.py, spiderfoot/dicts/**, README.md, requirements.txt, setup.cfg, and VERSION.

This area contains the shared primitives that other SpiderFoot scans, modules, CLI/web frontends, and exporters build on. spiderfoot/__init__.py is the package facade: importing spiderfoot exposes core classes from sibling modules (SpiderFootDb, SpiderFootEvent, SpiderFootThreadPool, SpiderFootPlugin, SpiderFootTarget, SpiderFootHelpers, and SpiderFootCorrelator) plus __version__. That makes helper and core class imports stable even when their implementation files live elsewhere.

Version metadata is deliberately duplicated in simple files. spiderfoot/__version__.py defines VERSION = (4, 0, 0) and derives __version__ = "4.0.0"; root VERSION stores the display string SpiderFoot 4.0.0; README.md advertises version 4.0 and describes SpiderFoot as a Python 3 OSINT automation tool with CLI, embedded web UI, SQLite backend, graph exports, correlation rules, and a publisher/subscriber module model. Packaging and runtime dependency expectations are captured by requirements.txt, while setup.cfg holds lint/docstring policy and per-file ignores for the package facade and selected high-complexity legacy areas.

spiderfoot/helpers.py is the main shared utility surface. SpiderFootHelpers.dataPath(), cachePath(), and logPath() resolve writable runtime directories from SPIDERFOOT_DATA, SPIDERFOOT_CACHE, and SPIDERFOOT_LOGS, defaulting to ~/.spiderfoot/, ~/.spiderfoot/cache, and ~/.spiderfoot/logs; each function creates the directory before returning it. loadModulesAsDict() dynamically imports files named sfp_*.py from a supplied module directory as modules.<name>, instantiates the class with the same name, merges its asdict() metadata, and enforces exactly one known category. loadCorrelationRulesRaw() reads *.yaml files from a supplied correlations path into a rule-name-to-YAML-string dictionary.

Target and content normalization helpers in spiderfoot/helpers.py are used by scan setup and extractor modules. targetTypeFromString() maps seed strings to SpiderFoot event types using ordered regexes for IPv4, CIDR, email, phone, quoted human names and usernames, ASN, IPv6, IPv6 CIDR, DNS names, and Bitcoin addresses. URL helpers (urlRelativeToAbsolute(), urlBaseDir(), urlBaseUrl()) normalize relative paths and derive base URL components. genScanInstanceId() returns an uppercase short UUID segment for scan IDs. Validators include validLEI(), validEmail(), validPhoneNumber(), and sanitiseInput(); the last is an allowlist-based guard for command-like strings and rejects .., leading dashes, short inputs, and characters outside alphanumerics, -, ., plus optional extras.

Extractor helpers in spiderfoot/helpers.py convert raw content into event candidates without performing network I/O. extractLinksFromHtml() parses selected HTML tags with BeautifulSoup/lxml, filters JavaScript/data/mail/in-page/dynamic fragments, expands protocol-relative, root-relative, domain-mentioned, and directory-relative links, and returns absolute URLs keyed to their source and original form. Text extractors return de-duplicated findings for hashes, robots.txt disallow paths, PGP keys, email addresses, IBANs, credit card numbers, and HTTP(S) URLs. sslDerToPem() wraps DER-to-PEM conversion for certificates.

Graph/export helpers in spiderfoot/helpers.py transform scan result rows for frontends and exports. buildGraphData() expects rows of length 15 and uses row positions for entity value, parent value, event type, parent entity ID, and row class (ENTITY/INTERNAL) to derive parent-child entity edges, optionally filtered by event type. buildGraphJson() turns those edges into SigmaJS-style random-position node/edge JSON and highlights nodes whose labels appear in the scan root. buildGraphGexf() turns the same mapping into a NetworkX graph and serializes it through GEXFWriter. dataParentChildToTree() converts a parent-to-children dictionary into a D3-friendly nested tree by selecting the first key that is not listed as another node's child as the root.

Dictionary helpers in spiderfoot/helpers.py load package data from spiderfoot/dicts/** using importlib.resources. dictionaryWordsFromWordlists() reads spiderfoot/dicts/ispell/{english,german,french,spanish}.dict by default, lowercases entries, and strips ispell suffixes after /. humanNamesFromWordlists() reads spiderfoot/dicts/ispell/names.dict. usernamesFromWordlists() reads spiderfoot/…/generic-usernames.txt. The dictionary package also owns spiderfoot/…/subdomains.txt and spiderfoot/…/subdomains-10000.txt, which are line-oriented subdomain brute-force wordlists, plus spiderfoot/dicts/ispell/LICENSE for the bundled ispell dictionaries.

spiderfoot/logger.py centralizes process-safe logging. Workers call logWorkerSetup(loggingQueue) to attach a QueueHandler to the spiderfoot logger only once and emit all log records at DEBUG level into a main-process queue. The main process calls logListenerSetup(loggingQueue, opts) early to create and start a QueueListener. When logging is enabled via opts.get("__logging", True), the listener writes formatted messages to stderr, daily rotating spiderfoot.debug.log, daily rotating spiderfoot.error.log, and SQLite scan logs. The file handlers use SpiderFootHelpers.logPath() for their directory. The SQLite handler batches records carrying a scanId attribute, maps INFO to STATUS, records the emitting module name from the log record, and inserts batches through SpiderFootDb.scanLogEvents(); an atexit hook flushes remaining records and stop_listener() stops the queue listener on shutdown.

Coupling

  • If spiderfoot/__version__.py or root VERSION changes, also verify README.md release badges and install snippets; these files independently advertise the same version and can drift without code failing.
  • If SpiderFootHelpers.loadModulesAsDict() changes its dynamic import naming or category validation in spiderfoot/helpers.py, verify every module still lives under the modules import package with a class matching its sfp_*.py filename and that module metadata uses one of the hard-coded category names.
  • If scan result row layout changes, update SpiderFootHelpers.buildGraphData() in spiderfoot/helpers.py and all callers that feed graph exports; the helper indexes fixed row positions and raises only on total row length, so semantic column drift can produce wrong graphs without an obvious exception.
  • If dictionary files move or are renamed under spiderfoot/dicts/**, update the importlib.resources.open_text() package and filename strings in dictionaryWordsFromWordlists(), humanNamesFromWordlists(), and usernamesFromWordlists(); these helpers load package resources by literal package names rather than paths.
  • If log record attributes or scan log schema change, verify SpiderFootSqliteLogHandler in spiderfoot/logger.py and SpiderFootDb.scanLogEvents() together; the handler only persists records with record.scanId and passes (scanId, level, message, component, timestamp) tuples.
  • If runtime directory environment variables change, update both spiderfoot/helpers.py and spiderfoot/logger.py expectations; logger file handlers depend on SpiderFootHelpers.logPath() creating the log directory before handler construction.

Risks

  • spiderfoot/helpers.py has broad static utility responsibilities and a high complexity threshold is allowed in setup.cfg; changes to validators, extractors, or graph builders can affect many modules at once because callers import SpiderFootHelpers through the package facade in spiderfoot/__init__.py.
  • SpiderFootHelpers.extractLinksFromHtml() in spiderfoot/helpers.py iterates for domain in domains after only converting strings to one-element lists; if callers pass None, link extraction can fail when it reaches domain expansion. Callers that do not need domain matching should pass an empty list rather than None.
  • SpiderFootSqliteLogHandler.logBatch() in spiderfoot/logger.py inserts the current batch even when it is empty, including from the atexit path; SpiderFootDb.scanLogEvents() must tolerate empty batches or shutdown logging can produce avoidable errors.