Introducing a new way to review your PRs.

Read the announcement

smicallef / spiderfoot

SpiderFoot automates OSINT for threat intelligence and mapping your attack surface.

View on GitHub

13 documents · Updated Nov 16, 2026

docs / web-ui

Web UI

Overview

Owned paths/globs: sfwebui.py, spiderfoot/templates/**, spiderfoot/static/js/**, spiderfoot/static/css/**, spiderfoot/static/img/**, spiderfoot/…/package.json, test/…/test_spiderfootwebui.py, test/…/test_sfwebui.py, and test/acceptance/**.

The Web UI is a CherryPy application implemented by SpiderFootWebUi in sfwebui.py. Construction loads persisted configuration through SpiderFootDb.configGet(), merges it with defaults via SpiderFoot.configUnserialize(), configures logging queues, installs custom error handlers, and applies secure response headers including a CSP that still allows inline scripts/styles because the Mako templates embed page logic directly. The app uses self.docroot from the web config as the URL prefix; every template receives docroot and builds links under that prefix.

Page routes in sfwebui.py render Mako templates from spiderfoot/templates/:

  • / (index) renders spiderfoot/templates/scanlist.tmpl, which loads spiderfoot/…/spiderfoot.scanlist.js to fetch /scanlist and build the scan-history table in the browser.
  • /newscan (newscan) renders spiderfoot/templates/newscan.tmpl, passing event types from SpiderFootDb.eventTypes() and module metadata from self.config['__modules__']. spiderfoot/…/spiderfoot.newscan.js switches between use-case, event-type, and module selection tabs and fills hidden form fields before POSTing to /startscan.
  • /clonescan (clonescan) also renders newscan.tmpl, but preselects modules from SpiderFootDb.scanConfigGet(id) and quotes unknown target types so human names/usernames remain valid UI input.
  • /scaninfo (scaninfo) validates a scan with SpiderFootDb.scanInstanceGet(id) and renders spiderfoot/templates/scaninfo.tmpl. That template contains most review UI logic: summary, correlations, browse, graph, settings, logs, search, export, and false-positive actions are driven by AJAX calls back into sfwebui.py.
  • /opts (opts) renders spiderfoot/templates/opts.tmpl, creates a per-page CSRF token in self.token, and lists global plus per-module options. spiderfoot/…/spiderfoot.opts.js serializes form inputs into allopts, triggers config-file import, resets settings, and exports API keys via /optsexport?pattern=api_key.
  • /error pages are not a route, but error(), error_page_404(), and spiderfoot/templates/error.tmpl provide HTML failures used by many routes.

Scan launch and lifecycle control also live in sfwebui.py. /startscan sanitizes scanname and scantarget, validates target type with SpiderFootHelpers.targetTypeFromString(), derives modlist from explicit modules, requested event types, or use-case groups, forces sfp__stor_db, removes sfp__stor_stdout, normalizes target casing, and starts a spawned multiprocessing.Process running startSpiderFootScanner. It waits until SpiderFootDb.scanInstanceGet(scanId) appears, then returns JSON when Accept: application/json is requested or redirects to /scaninfo?id=<scanId> otherwise. /rerunscan and /rerunscanmulti repeat the same process from a saved scan config. /stopscan refuses finished/aborted/non-running scans and sets ABORT-REQUESTED; /scandelete refuses to delete running scans and deletes scan instances from the DB.

The JSON/API surface used by the browser is exposed from sfwebui.py and mostly returns list-shaped records expected by the JavaScript templates:

  • Scan inventory/status: /scanlist, /scanstatus, /scanerrors, /scanlog, /scanopts.
  • Scan results: /scansummary, /scaneventresults, /scaneventresultsunique, /search, /scanhistory, /scanelementtypediscovery, /scancorrelations.
  • Metadata for API/CLI consumers and UI selectors: /eventtypes, /modules, /correlationrules, /ping, /query.
  • Mutations: /resultsetfp, /savesettings, /savesettingsraw, /vacuum, /stopscan, /scandelete.
  • Downloads: /scanexportlogs, /scancorrelationsexport, /scaneventresultexport, /scaneventresultexportmulti, /scansearchresultexport, /scanexportjsonmulti, /scanviz, and /scanvizmulti set content-disposition/content-type headers and return CSV, Excel, JSON, graph JSON, or GEXF bytes/strings.

The shared page frame in spiderfoot/templates/HEADER.tmpl loads spiderfoot/…/spiderfoot.js, spiderfoot/…/viz.js, Bootstrap, jQuery, Alertify, tablesorter, D3, CSS from spiderfoot/…/spiderfoot.css, optional dark CSS from spiderfoot/…/dark.css, and images from spiderfoot/static/img/. spiderfoot/…/package.json pins the browser dependencies loaded from static/node_modules: AlertifyJS, Bootstrap 3, D3 v3, jQuery, Sigma, and tablesorter. spiderfoot/templates/FOOTER.tmpl closes the shared container and renders a random promotional footer message.

Browser responsibilities are split by page. spiderfoot/…/spiderfoot.js provides the global sf helper namespace: AJAX via sf.fetchData(), scan delete/stop wrappers, search, SFURL tag rendering/removal, tooltip setup, console logging, and the dark-mode toggle backed by localStorage. spiderfoot/…/spiderfoot.scanlist.js renders the scan table from /scanlist, handles filtering by scan status, multi-select with shift-click, stop/delete/rerun actions, and multi-scan exports through a hidden iframe. spiderfoot/templates/scaninfo.tmpl contains its own page-specific JavaScript for summary polling every five seconds until the status is terminal, browse/search rendering, correlation expansion, log tables, export iframe URLs, Sigma graph rendering from /scanviz?gexf=0, false-positive updates through /resultsetfp, and D3 visualizations via viz.js. spiderfoot/…/viz.js supplies the D3 vertical bar chart, bubble chart, and dendrogram used by the scan summary and event browse visualization modes.

Coverage is split across unit, integration, and browser acceptance tests. test/…/test_spiderfootwebui.py instantiates SpiderFootWebUi with test configs and checks constructor validation, error rendering, input cleaning, invalid-scan behavior, settings helpers, query restrictions, start-scan validation, and list-shaped JSON providers. Many export and full scan-start tests in that file are explicitly skipped. test/…/test_sfwebui.py mounts the CherryPy app with a test DB and static directory, then verifies route status/body behavior for page routes, static assets, invalid scan IDs, selectors, metadata endpoints, and a redirect from a minimal /startscan. test/acceptance/scan.robot is the end-to-end browser suite: it assumes a live server on 127.0.0.1:5001, drives New Scan, Scans, Settings, and Scan Info navigation with Selenium, creates module/use-case scans, waits for scan completion, and verifies Browse/Summary/Graph/Settings/Log tab elements plus DNS resolve/reverse-resolve visible results. test/acceptance/run and test/…/requirements.txt document the Robot Framework invocation and dependencies.

Coupling

  • If sfwebui.py changes any JSON tuple/list shape returned by /scanlist, /scanstatus, /scansummary, /scaneventresults, /scaneventresultsunique, /scanopts, /scanlog, or /scancorrelations, update the index-based consumers in spiderfoot/…/spiderfoot.scanlist.js and the inline JavaScript in spiderfoot/templates/scaninfo.tmpl; these clients read numeric array positions rather than named fields.
  • If scan status names change in the scanner or database layer, update all badge/action checks in sfwebui.py, spiderfoot/…/spiderfoot.scanlist.js, and spiderfoot/templates/scaninfo.tmpl; running/terminal behavior is hard-coded as strings such as RUNNING, STARTING, INITIALIZING, FINISHED, ABORTED, and ERROR-FAILED.
  • If route names or docroot behavior change in sfwebui.py, update spiderfoot/templates/HEADER.tmpl, spiderfoot/templates/newscan.tmpl, spiderfoot/templates/opts.tmpl, spiderfoot/templates/scanlist.tmpl, spiderfoot/templates/scaninfo.tmpl, and the external JS files because they construct URLs directly rather than via a shared router.
  • If module metadata or option schema changes, verify both spiderfoot/templates/newscan.tmpl and spiderfoot/templates/opts.tmpl: they expect self.config['__modules__'][mod] to expose name, descr, group, opts, optdescs, optional cats, labels, and meta.dataSource fields.
  • If false-positive semantics change in the DB layer, update /resultsetfp in sfwebui.py and the selection/FP display logic in spiderfoot/templates/scaninfo.tmpl; the server cascades FP changes to children and blocks unsetting when a direct parent remains FP, while the browser assumes returned row positions include result ID and FP flag.
  • If static dependency versions in spiderfoot/…/package.json are upgraded, verify spiderfoot/templates/HEADER.tmpl, spiderfoot/templates/scaninfo.tmpl, and spiderfoot/…/viz.js; the UI uses Bootstrap 3 APIs, D3 v3 APIs (d3.svg.axis, d3.layout.pack, d3.layout.cluster), Sigma v1 plugin paths, and tablesorter pager/widget APIs.
  • If the settings CSRF flow changes, update both sfwebui.py (opts, optsraw, savesettings, savesettingsraw) and spiderfoot/…/spiderfoot.opts.js; the token is stored on the SpiderFootWebUi instance and submitted as a hidden field or JSON parameter.

Risks

  • spiderfoot/…/spiderfoot.newscan.js has a malformed loop in submitForm() (for (i = 0; i < tabs.length; tabs++)). If this path executes, submitting a new scan can fail to clear inactive hidden fields correctly or throw in the browser before the form reaches /startscan.
  • spiderfoot/…/spiderfoot.js is an external static file but its theme-toggle code assigns link.href = "${docroot}/static/css/..."; unlike Mako templates, that placeholder is not rendered. If the pre-reload stylesheet append starts to matter, the browser can request a literal ${docroot} URL instead of the mounted static path.
  • sfwebui.py starts scans in /startscan, /rerunscan, and /rerunscanmulti and then busy-waits on SpiderFootDb.scanInstanceGet(scanId) with no timeout. If startSpiderFootScanner fails before creating the DB row, the request can hang indefinitely.
  • sfwebui.py stores the settings CSRF token as self.token on the shared SpiderFootWebUi instance. If two settings pages are opened concurrently, the later /opts or /optsraw call overwrites the token and the earlier form submission to /savesettings or /savesettingsraw fails as invalid.
  • test/…/test_spiderfootwebui.py skips many export and mutation tests, and test/…/test_sfwebui.py mostly exercises invalid-scan paths. Changes to successful CSV/Excel/GEXF/JSON export behavior in sfwebui.py can regress without local test failures unless test/acceptance/scan.robot or manual browser export flows are run.