Introducing a new way to review your PRs.

Read the announcement

smicallef / spiderfoot

SpiderFoot automates OSINT for threat intelligence and mapping your attack surface.

View on GitHub

13 documents · Updated Nov 16, 2026

docs / cli-and-scan-entrypoints

CLI and Scan Entrypoints

Owned paths/globs: sf.py, sfcli.py, sfscan.py, sflib.py, test/…/test_spiderfoot.py, test/…/test_spiderfootcli.py, test/…/test_spiderfootscanner.py, test/…/test_sf.py, test/…/test_sfcli.py.

Overview

This area covers the local executable paths for running SpiderFoot outside the web UI and the runtime classes they use to configure and execute scans.

sf.py is the main command-line entrypoint. Its main() builds the default global sfConfig, loads module metadata from modules/ with SpiderFootHelpers.loadModulesAsDict(), loads YAML correlation rules from correlations/, initializes SpiderFootDb, and stores the module/rule registries into sfConfig['__modules__'] and sfConfig['__correlationrules__']. It has three high-level modes:

  • metadata/listing mode: -M prints loaded modules, -T opens the DB with init=True and prints event types, and -V prints the version;
  • correlation mode: -C <scanID> constructs SpiderFootCorrelator with the DB handle and raw rules, then runs correlations against an existing scan;
  • runtime mode: -l IP:port starts the CherryPy web UI through start_web_server(), while scan arguments such as -s, -m, -t, and -u call start_scan().

sf.py:start_scan() is the local non-web scan orchestrator. It opens SpiderFootDb(sfConfig, init=True), creates a SpiderFoot helper, validates scan-only options, normalizes quoted/name targets, asks SpiderFootHelpers.targetTypeFromString() for the target type, and builds the module list. Module selection can be explicit (-m), type-driven (-t, expanded through SpiderFoot.modulesProducing() and SpiderFoot.eventsToModules()), use-case-driven (-u against loaded module groups), or strict (-x, limiting modules to those that consume the target type and provide requested types). It always appends sfp__stor_db and sfp__stor_stdout; sfp__stor_stdout is configured in-place with output format, filters, headers, source-data inclusion, maximum data length, newline stripping, and CSV delimiter settings. Before execution, saved DB configuration from dbh.configGet() is merged back into defaults with SpiderFoot.configUnserialize(), debug and strict output filtering are reapplied, and a scan ID is generated with SpiderFootHelpers.genScanInstanceId().

Actual local scan execution from sf.py runs in a child multiprocessing.Process targeting sfscan.startSpiderFootScanner(loggingQueue, scanName, scanId, target, targetType, modlist, cfg). The parent polls SpiderFootDb.scanInstanceGet(scanId) once per second until the scan status becomes ERROR-FAILED, ABORT-REQUESTED, ABORTED, or FINISHED; for JSON output it prints [ before the child starts and ] at completion. Ctrl-C is handled by sf.py:handle_abort(), which marks the scan row ABORTED through the DB before exiting.

sf.py:start_web_server() is still part of this entrypoint because sf.py -l starts the local web runtime that sfcli.py talks to. It configures CherryPy host/port/static paths, loads optional digest-auth credentials from SpiderFootHelpers.dataPath()/passwd, enables TLS if spiderfoot.key and spiderfoot.crt exist under the data path, installs CORS origins for the chosen URL, and launches SpiderFootWebUi(sfWebUiConfig, sfConfig, loggingQueue). The script guard in sf.py refuses to run with no arguments, requires Python 3.7+, and blocks legacy spiderfoot.db or passwd files in the application directory because the runtime expects them in the user data directory.

sfscan.py owns the in-process scan controller. startSpiderFootScanner() installs worker logging and instantiates SpiderFootScanner. SpiderFootScanner.__init__() deep-copies global options, opens SpiderFootDb, validates scan name/id/target/module inputs, creates a SpiderFoot helper bound to the DB and scan ID, records the scan with scanInstanceCreate(), creates a SpiderFootTarget, serializes the scan configuration with _modulesenabled, and prepares global runtime settings. Proxy settings in _socks1type/_socks2addr/_socks3port/_socks4user/_socks5pwd are converted into a proxy URL on SpiderFoot.socksProxy; _dnsserver overrides the system resolver via dns.resolver; _useragent and _internettlds may be loaded from literal strings, files, or URLs through SpiderFoot.optValueToData(), with TLD data cached using SpiderFoot.cacheGet() and cachePut(). The scanner sets status to INITIALIZING, creates a shared SpiderFootThreadPool, and starts immediately unless constructed with start=False.

SpiderFootScanner.__startScan() loads and wires modules. For each requested module name, it verifies the module is present in __config['__modules__'], imports modules.<modName>, instantiates the plugin class, merges module-specific options with global scan options, clears old listeners, sets scan ID/shared thread pool/DB handle, calls setup(), applies proxy socket and output filters, lets the module enrich the target, sets the final target, and attaches incoming/outgoing queues. Loaded modules are sorted by _priority; a pseudo SpiderFootPlugin named SpiderFoot UI emits the initial ROOT event and the target event, with an extra DOMAIN_NAME event when an INTERNET_NAME target is also a domain according to the TLD list. waitForThreads() starts each module thread, routes every SpiderFootEvent from the shared outgoing queue to modules whose watchedEvents() match, periodically checks the DB row for ABORT-REQUESTED, sends FINISHED sentinel values during final passes, and shuts down the shared thread pool in finally. Successful scans are marked FINISHED, post-scan rules run through runCorrelations(), and the DB handle is closed; aborted and failed scans are marked ABORTED or ERROR-FAILED.

sflib.py provides the SpiderFoot runtime helper used by both sf.py and sfscan.py. For this flow, the important methods are configuration serialization/deserialization (configSerialize(), configUnserialize()), module graph selection (modulesProducing(), modulesConsuming(), eventsFromModules(), eventsToModules()), option indirection (optValueToData()), scan-aware logging (error(), status(), info(), debug()), cache storage (cachePut(), cacheGet()), target/domain helpers (validHost(), isDomain()), and HTTP/DNS/proxy support (getSession(), useProxyForUrl(), fetchUrl()). SpiderFoot carries the DB handle, scan ID, and proxy string that scanner-created modules use indirectly during runtime.

sfcli.py is a separate interactive and batch client for an already-running SpiderFoot web server, not a direct scanner. The SpiderFootCli class subclasses cmd.Cmd and maintains local CLI options in ownopts, including server URL, output format, history/spool files, color, digest-auth username/password, and TLS verification. Its request() method sends JSON-accepting GET or POST requests with User-Agent SpiderFoot-CLI/<version> and optional HTTP Digest auth. The executable guard maps command-line flags to ownopts, can read commands from -e FILE or stdin without prompts, loads history from ~/.spiderfoot_history unless disabled, pings the server, then enters cmdloop().

sfcli.py commands are thin wrappers around web endpoints. start posts scanname, scantarget, modulelist, typelist, and usecase to /startscan, optionally following with logs <sid> -w; stop calls /stopscan; delete calls /scandelete; scans, scaninfo, data, summary, find, correlations, logs, and export read server-side DB-backed scan state through endpoints such as /scanlist, /scanopts, /scaneventresults, /scansummary, /search, /scancorrelations, /scanlog, /scanexportjsonmulti, /scaneventresultexportmulti, and /scanvizmulti. modules, types, and correlationrules populate completion caches or print server metadata. set has two paths: local cli.* values are updated in memory, while server-side settings are read from /optsraw, edited, then submitted to /savesettingsraw with the server token. Output is parsed as JSON by send_output(), rendered as pretty tables or JSON, and may be piped through built-in str/grep, regex, top, last, or file filters.

The tests in the owned paths exercise these entrypoints at two levels. test/…/test_spiderfootscanner.py documents scanner construction invariants, accepted status values, proxy validation, and the start=False path for creating scan DB state without launching modules. test/…/test_spiderfoot.py covers the SpiderFoot helpers that local scan setup relies on: config serialization, module/event graph lookup, cache access, DNS/target helpers, proxy decisions, and HTTP fetch behavior. test/…/test_spiderfootcli.py focuses on CLI parsing, printing, history/spool toggles, output formatting, local set, and command stubs. test/…/test_sf.py executes sf.py as a subprocess for no-arg, help, module/type listing, debug/quiet, invalid-target, invalid-module/type, and successful scan cases. test/…/test_sfcli.py verifies sfcli.py -h starts argparse help cleanly.

Coupling

  • If scan status strings change in sfscan.py, update every reader that treats them as protocol values: sf.py:start_scan() polls for ERROR-FAILED, ABORT-REQUESTED, ABORTED, and FINISHED, while test/…/test_spiderfootscanner.py asserts initialization and failure statuses.
  • If module metadata keys loaded into sfConfig['__modules__'] change, verify sf.py:start_scan() and sflib.py together. The CLI scan path expects descr, group, provides, consumes, and opts; SpiderFoot.modulesProducing(), modulesConsuming(), eventsToModules(), and scanner module setup read those keys directly.
  • If storage/output modules are renamed or their options change, update sf.py:start_scan(). It unconditionally appends sfp__stor_db and sfp__stor_stdout, then mutates sfConfig['__modules__']['sfp__stor_stdout']['opts'] using option names such as _format, _requested, _showonlyrequested, _showsource, _maxlength, and _csvdelim.
  • If web UI endpoint names or payloads change, update sfcli.py command methods at the same time. The CLI does not use a generated client; it hard-codes endpoint paths and form fields for starting scans, reading results, exporting data, editing settings, and stopping/deleting scans.
  • If config option types or storage names change, verify both sflib.py and sfcli.py. Local scans use SpiderFoot.configUnserialize(dbh.configGet(), sfConfig) and scanner snapshots use configSerialize(), while remote CLI settings strip module prefixes before posting /savesettingsraw.
  • If logging setup changes, keep sf.py and sfscan.py aligned. The parent creates a multiprocessing logging queue and calls logListenerSetup()/logWorkerSetup(), while child scans enter through startSpiderFootScanner() solely to call logger.logWorkerSetup(loggingQueue) before constructing SpiderFootScanner.

Risks

  • sfcli.py:do_shell() executes arbitrary local shell text with os.popen(line). Any context that feeds untrusted command files through sfcli.py -e can execute local commands, not just SpiderFoot CLI commands.
  • sfcli.py:do_stop() prints scan {id} instead of the selected scan_id, so successful stop messages can show Python's built-in id object rather than the scan identifier; this can mislead operators when multiple scans are running.
  • sf.py:start_scan() mutates sfConfig['__modules__']['sfp__stor_stdout']['opts'] before copying DB-backed config. Because module metadata is shared inside the process, repeated scans in the same process or tests reusing sfConfig can inherit prior stdout output filters.
  • sfscan.py:SpiderFootScanner.__init__() overrides the process-wide DNS resolver with dns.resolver.override_system_resolver() and restores it only when _dnsserver is blank during scanner construction. Concurrent scans with different DNS settings in the same process can affect each other's DNS behavior.