Web UI
Overview
Owned paths/globs: sfwebui.py, spiderfoot/templates/**, spiderfoot/static/js/**, spiderfoot/static/css/**, spiderfoot/static/img/**, spiderfoot/…/package.json, test/…/test_spiderfootwebui.py, test/…/test_sfwebui.py, and test/acceptance/**.
The Web UI is a CherryPy application implemented by SpiderFootWebUi in sfwebui.py. Construction loads persisted configuration through SpiderFootDb.configGet(), merges it with defaults via SpiderFoot.configUnserialize(), configures logging queues, installs custom error handlers, and applies secure response headers including a CSP that still allows inline scripts/styles because the Mako templates embed page logic directly. The app uses self.docroot from the web config as the URL prefix; every template receives docroot and builds links under that prefix.
Page routes in sfwebui.py render Mako templates from spiderfoot/templates/:
/(index) rendersspiderfoot/templates/scanlist.tmpl, which loads spiderfoot/…/spiderfoot.scanlist.js to fetch/scanlistand build the scan-history table in the browser./newscan(newscan) rendersspiderfoot/templates/newscan.tmpl, passing event types fromSpiderFootDb.eventTypes()and module metadata fromself.config['__modules__']. spiderfoot/…/spiderfoot.newscan.js switches between use-case, event-type, and module selection tabs and fills hidden form fields before POSTing to/startscan./clonescan(clonescan) also rendersnewscan.tmpl, but preselects modules fromSpiderFootDb.scanConfigGet(id)and quotes unknown target types so human names/usernames remain valid UI input./scaninfo(scaninfo) validates a scan withSpiderFootDb.scanInstanceGet(id)and rendersspiderfoot/templates/scaninfo.tmpl. That template contains most review UI logic: summary, correlations, browse, graph, settings, logs, search, export, and false-positive actions are driven by AJAX calls back into sfwebui.py./opts(opts) rendersspiderfoot/templates/opts.tmpl, creates a per-page CSRF token inself.token, and lists global plus per-module options. spiderfoot/…/spiderfoot.opts.js serializes form inputs intoallopts, triggers config-file import, resets settings, and exports API keys via/optsexport?pattern=api_key./errorpages are not a route, buterror(),error_page_404(), andspiderfoot/templates/error.tmplprovide HTML failures used by many routes.
Scan launch and lifecycle control also live in sfwebui.py. /startscan sanitizes scanname and scantarget, validates target type with SpiderFootHelpers.targetTypeFromString(), derives modlist from explicit modules, requested event types, or use-case groups, forces sfp__stor_db, removes sfp__stor_stdout, normalizes target casing, and starts a spawned multiprocessing.Process running startSpiderFootScanner. It waits until SpiderFootDb.scanInstanceGet(scanId) appears, then returns JSON when Accept: application/json is requested or redirects to /scaninfo?id=<scanId> otherwise. /rerunscan and /rerunscanmulti repeat the same process from a saved scan config. /stopscan refuses finished/aborted/non-running scans and sets ABORT-REQUESTED; /scandelete refuses to delete running scans and deletes scan instances from the DB.
The JSON/API surface used by the browser is exposed from sfwebui.py and mostly returns list-shaped records expected by the JavaScript templates:
- Scan inventory/status:
/scanlist,/scanstatus,/scanerrors,/scanlog,/scanopts. - Scan results:
/scansummary,/scaneventresults,/scaneventresultsunique,/search,/scanhistory,/scanelementtypediscovery,/scancorrelations. - Metadata for API/CLI consumers and UI selectors:
/eventtypes,/modules,/correlationrules,/ping,/query. - Mutations:
/resultsetfp,/savesettings,/savesettingsraw,/vacuum,/stopscan,/scandelete. - Downloads:
/scanexportlogs,/scancorrelationsexport,/scaneventresultexport,/scaneventresultexportmulti,/scansearchresultexport,/scanexportjsonmulti,/scanviz, and/scanvizmultiset content-disposition/content-type headers and return CSV, Excel, JSON, graph JSON, or GEXF bytes/strings.
The shared page frame in spiderfoot/templates/HEADER.tmpl loads spiderfoot/…/spiderfoot.js, spiderfoot/…/viz.js, Bootstrap, jQuery, Alertify, tablesorter, D3, CSS from spiderfoot/…/spiderfoot.css, optional dark CSS from spiderfoot/…/dark.css, and images from spiderfoot/static/img/. spiderfoot/…/package.json pins the browser dependencies loaded from static/node_modules: AlertifyJS, Bootstrap 3, D3 v3, jQuery, Sigma, and tablesorter. spiderfoot/templates/FOOTER.tmpl closes the shared container and renders a random promotional footer message.
Browser responsibilities are split by page. spiderfoot/…/spiderfoot.js provides the global sf helper namespace: AJAX via sf.fetchData(), scan delete/stop wrappers, search, SFURL tag rendering/removal, tooltip setup, console logging, and the dark-mode toggle backed by localStorage. spiderfoot/…/spiderfoot.scanlist.js renders the scan table from /scanlist, handles filtering by scan status, multi-select with shift-click, stop/delete/rerun actions, and multi-scan exports through a hidden iframe. spiderfoot/templates/scaninfo.tmpl contains its own page-specific JavaScript for summary polling every five seconds until the status is terminal, browse/search rendering, correlation expansion, log tables, export iframe URLs, Sigma graph rendering from /scanviz?gexf=0, false-positive updates through /resultsetfp, and D3 visualizations via viz.js. spiderfoot/…/viz.js supplies the D3 vertical bar chart, bubble chart, and dendrogram used by the scan summary and event browse visualization modes.
Coverage is split across unit, integration, and browser acceptance tests. test/…/test_spiderfootwebui.py instantiates SpiderFootWebUi with test configs and checks constructor validation, error rendering, input cleaning, invalid-scan behavior, settings helpers, query restrictions, start-scan validation, and list-shaped JSON providers. Many export and full scan-start tests in that file are explicitly skipped. test/…/test_sfwebui.py mounts the CherryPy app with a test DB and static directory, then verifies route status/body behavior for page routes, static assets, invalid scan IDs, selectors, metadata endpoints, and a redirect from a minimal /startscan. test/acceptance/scan.robot is the end-to-end browser suite: it assumes a live server on 127.0.0.1:5001, drives New Scan, Scans, Settings, and Scan Info navigation with Selenium, creates module/use-case scans, waits for scan completion, and verifies Browse/Summary/Graph/Settings/Log tab elements plus DNS resolve/reverse-resolve visible results. test/acceptance/run and test/…/requirements.txt document the Robot Framework invocation and dependencies.
Coupling
- If sfwebui.py changes any JSON tuple/list shape returned by
/scanlist,/scanstatus,/scansummary,/scaneventresults,/scaneventresultsunique,/scanopts,/scanlog, or/scancorrelations, update the index-based consumers in spiderfoot/…/spiderfoot.scanlist.js and the inline JavaScript inspiderfoot/templates/scaninfo.tmpl; these clients read numeric array positions rather than named fields. - If scan status names change in the scanner or database layer, update all badge/action checks in sfwebui.py, spiderfoot/…/spiderfoot.scanlist.js, and
spiderfoot/templates/scaninfo.tmpl; running/terminal behavior is hard-coded as strings such asRUNNING,STARTING,INITIALIZING,FINISHED,ABORTED, andERROR-FAILED. - If route names or
docrootbehavior change in sfwebui.py, updatespiderfoot/templates/HEADER.tmpl,spiderfoot/templates/newscan.tmpl,spiderfoot/templates/opts.tmpl,spiderfoot/templates/scanlist.tmpl,spiderfoot/templates/scaninfo.tmpl, and the external JS files because they construct URLs directly rather than via a shared router. - If module metadata or option schema changes, verify both
spiderfoot/templates/newscan.tmplandspiderfoot/templates/opts.tmpl: they expectself.config['__modules__'][mod]to exposename,descr,group,opts,optdescs, optionalcats,labels, andmeta.dataSourcefields. - If false-positive semantics change in the DB layer, update
/resultsetfpin sfwebui.py and the selection/FP display logic inspiderfoot/templates/scaninfo.tmpl; the server cascades FP changes to children and blocks unsetting when a direct parent remains FP, while the browser assumes returned row positions include result ID and FP flag. - If static dependency versions in spiderfoot/…/package.json are upgraded, verify
spiderfoot/templates/HEADER.tmpl,spiderfoot/templates/scaninfo.tmpl, and spiderfoot/…/viz.js; the UI uses Bootstrap 3 APIs, D3 v3 APIs (d3.svg.axis,d3.layout.pack,d3.layout.cluster), Sigma v1 plugin paths, and tablesorter pager/widget APIs. - If the settings CSRF flow changes, update both sfwebui.py (
opts,optsraw,savesettings,savesettingsraw) and spiderfoot/…/spiderfoot.opts.js; the token is stored on theSpiderFootWebUiinstance and submitted as a hidden field or JSON parameter.
Risks
- spiderfoot/…/spiderfoot.newscan.js has a malformed loop in
submitForm()(for (i = 0; i < tabs.length; tabs++)). If this path executes, submitting a new scan can fail to clear inactive hidden fields correctly or throw in the browser before the form reaches/startscan. - spiderfoot/…/spiderfoot.js is an external static file but its theme-toggle code assigns
link.href = "${docroot}/static/css/..."; unlike Mako templates, that placeholder is not rendered. If the pre-reload stylesheet append starts to matter, the browser can request a literal${docroot}URL instead of the mounted static path. - sfwebui.py starts scans in
/startscan,/rerunscan, and/rerunscanmultiand then busy-waits onSpiderFootDb.scanInstanceGet(scanId)with no timeout. IfstartSpiderFootScannerfails before creating the DB row, the request can hang indefinitely. - sfwebui.py stores the settings CSRF token as
self.tokenon the sharedSpiderFootWebUiinstance. If two settings pages are opened concurrently, the later/optsor/optsrawcall overwrites the token and the earlier form submission to/savesettingsor/savesettingsrawfails as invalid. - test/…/test_spiderfootwebui.py skips many export and mutation tests, and test/…/test_sfwebui.py mostly exercises invalid-scan paths. Changes to successful CSV/Excel/GEXF/JSON export behavior in sfwebui.py can regress without local test failures unless
test/acceptance/scan.robotor manual browser export flows are run.