Introducing a new way to review your PRs.

Read the announcement

smicallef / spiderfoot

SpiderFoot automates OSINT for threat intelligence and mapping your attack surface.

View on GitHub

13 documents · Updated Nov 16, 2026

docs / plugin-event-target-runtime

Plugin/Event/Target Runtime

Owned paths/globs for this doc:

Overview

This area defines the core runtime contract for SpiderFoot modules: SpiderFootPlugin in spiderfoot/plugin.py is the base class module authors subclass, SpiderFootEvent in spiderfoot/event.py is the data envelope passed between modules, SpiderFootTarget in spiderfoot/target.py models the original scan target and aliases used for scope checks, and SpiderFootThreadPool in spiderfoot/threadpool.py executes module work concurrently. modules/sfp_template.py is the canonical author-facing example showing how a plugin declares metadata, options, watched/produced events, external queries, duplicate suppression, event creation, and listener notification.

Plugin instances are stateful. SpiderFootPlugin holds the current scan id (setScanId() / getScanId()), current target (setTarget() / getTarget()), database handle (setDbh()), listener modules, optional input/output event queues, an optional shared thread pool, and stop/error flags. Module authors override setup(sf, userOpts), watchedEvents(), producedEvents(), handleEvent(event), and optionally finish() and running. The template in modules/sfp_template.py shows setup() storing the SpiderFoot API object in self.sf, merging userOpts into class-level defaults, initializing per-module duplicate tracking via self.tempStorage(), and setting self.__dataSource__ when the module wants a data-source label distinct from the module name.

Events are created by constructing SpiderFootEvent(eventType, data, module, sourceEvent) in spiderfoot/event.py. A ROOT event is special: it has no parent, its sourceEventHash and hash are both effectively the literal root identity (sourceEventHash == "ROOT", hash == "ROOT"), and its module may be empty. Non-root events require a non-empty module name and a parent SpiderFootEvent; the constructor records creation time, defaults confidence and visibility to 100, defaults risk to 0, and builds a random per-instance id that is hashed with SHA-256 for hash. asDict() emits the reduced event view used by consumers: integer generated, type, data, module, and parent source data when present.

Event dispatch is handled by SpiderFootPlugin.notifyListeners() in spiderfoot/plugin.py. It first validates that the argument is a SpiderFootEvent, applies __outputFilter__ unless the event is ROOT or the target's own event type, drops empty data, and stops early if checkForStop() says the scan should abort. It walks the source-event chain to detect repeated event type/data pairs; if the same item already appears upstream, it marks the event as storeOnly, meaning only storage modules whose __module__ contains __stor should see it. If outgoingEventQueue is configured, dispatch is asynchronous and the event is placed onto that queue. Otherwise listeners are sorted by _priority, filtered by listener.watchedEvents() (or *), assigned _currentEvent, and invoked synchronously through listener.handleEvent(sfEvent) in the current thread.

Threaded module execution is layered on top of queue-based dispatch. SpiderFootPlugin.start() starts a module-owned Python thread whose target is threadWorker() in spiderfoot/plugin.py. The worker creates a fresh SpiderFootDb handle for the thread, requires both incomingEventQueue and outgoingEventQueue, polls incoming events, submits normal events to handleEvent, and submits the sentinel string FINISHED to finish. poolExecute() runs storage modules whose names start with sfp__stor_ inline, but sends all other callbacks to the shared SpiderFootThreadPool with a task name of <module_name>_threadWorker and the module's maxThreads limit. SpiderFootPlugin.running reports whether the shared pool still has queued or running work for that task name.

SpiderFootThreadPool in spiderfoot/threadpool.py owns fixed worker threads and per-task queues. start() creates ThreadPoolWorker threads once; submit(callback, *args, taskName='default', maxThreads=100, ...) blocks while that task has maxThreads queued/running calls, then places (callback, args, kwargs) into that task's input queue. Workers round-robin across all task queues, execute callbacks, and optionally place return values into the task's output queue when saveResult=True. map() feeds an iterable through a background inputThread, starts the pool, and yields results(taskName, wait=True). shutdown(wait=True) drains saved results until finished, sets stop, clears input/output queues, and returns results grouped by task name. The unit coverage in test/…/test_spiderfootthreadpool.py exercises map(), direct submit(), saved results, named task queues, and mixed map/submit use.

Targets in spiderfoot/target.py constrain what a scan is about. SpiderFootTarget(targetValue, typeName) accepts only target types in _validTypes such as IP_ADDRESS, IPV6_ADDRESS, NETBLOCK_OWNER, NETBLOCKV6_OWNER, INTERNET_NAME, EMAILADDR, HUMAN_NAME, BGP_AS_OWNER, PHONE_NUMBER, USERNAME, and BITCOIN_ADDRESS. Aliases are stored as lowercase dictionaries with type and value through setAlias(), ignoring invalid or duplicate inputs. getNames() returns INTERNET_NAME aliases plus the target value for INTERNET_NAME and EMAILADDR targets. getAddresses() returns IP address aliases plus the current value for IP targets. matches(value, includeParents=False, includeChildren=True) is the runtime scope check: human names, phone numbers, usernames, and bitcoin addresses match everything; IP values match aliases or membership in IP/netblock targets using netaddr; domain-like values match exact names, optional parent domains, and optional child/subdomain names.

For module authors, modules/sfp_template.py is the practical lifecycle reference. A plugin subclass sets meta, opts, and optdescs; initializes mutable state in setup() rather than relying on class attributes; declares input types in watchedEvents() and output types in producedEvents(); keeps external API code in a helper such as query(); and in handleEvent() reads event.eventType, event.module, and event.data, suppresses duplicates in self.results, checks options before expensive expansion, creates child events with SpiderFootEvent(..., self.__name__, parent_event), calls self.notifyListeners(child_event), and checks self.checkForStop() inside loops so aborts can take effect. The template also shows how parser/extractor modules can preserve upstream data-source attribution by copying event.moduleDataSource to events they produce.

The unit tests under test/…/test_spiderfootplugin.py, test/…/test_spiderfootevent.py, and test/…/test_spiderfoottarget.py mostly assert constructor validation, simple accessors, dispatch type checks, target matching behavior, and the basic stop path. Several plugin tests are smoke tests with placeholder assertions, so behavior such as listener ordering, store-only delivery, threaded queue drainage, and output-filter edge cases is more strongly defined by spiderfoot/plugin.py itself than by precise unit assertions.

Coupling

  • If adding or renaming event types emitted by a module template or base-plugin flow, update both producedEvents()/watchedEvents() implementations and any target-type assumptions in SpiderFootPlugin.notifyListeners(); the output filter in spiderfoot/plugin.py compares strings against ROOT, self.getTarget().targetType, and __outputFilter__, so event-type drift silently drops dispatch.
  • If changing the SpiderFootEvent constructor or validation in spiderfoot/event.py, update module examples in modules/sfp_template.py; module authors are expected to pass the current event as sourceEvent, and non-root events will fail immediately if the parent is not a SpiderFootEvent.
  • If changing threaded execution in spiderfoot/plugin.py, verify spiderfoot/threadpool.py task naming and SpiderFootPlugin.running; both use the exact <module_name>_threadWorker task-name convention for queue/running counts.
  • If changing queue payloads in SpiderFootPlugin.threadWorker(), verify producers and consumers handle both SpiderFootEvent objects and the literal FINISHED sentinel; threadWorker() treats that string as the cleanup hook trigger and submits finish() instead of handleEvent().
  • If changing SpiderFootTarget._validTypes in spiderfoot/target.py, update test/…/test_spiderfoottarget.py and audit any modules that expect the target's type to be a possible event type; SpiderFootPlugin.notifyListeners() explicitly allows events whose type equals self.getTarget().targetType through the output filter.
  • If changing result saving in SpiderFootThreadPool, keep saveResult, taskName, and maxThreads as reserved kwargs; ThreadPoolWorker.run() removes taskName and saveResult before invoking callbacks, while SpiderFootThreadPool.submit() removes maxThreads, so callbacks should not receive those control fields.
  • If changing logging wrappers in spiderfoot/plugin.py, preserve the extra={'scanId': self.__scanId__} shape used by debug(), info(), and error(); the custom SpiderFootPluginLogger exists so module logs keep the caller's filename and line number instead of reporting the base wrapper.

Risks

  • spiderfoot/plugin.py has historically fragile threading paths: recent history includes 418ef64 Added spiderfoot-integrated thread pool, 2c39b7f slaying the threading dragons, and 1f35646 Shared threadpool. If threadWorker(), poolExecute(), or queue cleanup changes without exercising aborted scans and crashing handleEvent() callbacks, scans can hang because stale objects remain in incomingEventQueue or incomingEventQueue is left usable after errorState.
  • spiderfoot/threadpool.py workers swallow callback exceptions after logging and then continue the outer loop. If callers rely on saveResult=True, a callback failure produces no result object and no exception to shutdown()/results(), so code that assumes one output per submitted item can silently lose data.
  • spiderfoot/target.py implements getAddresses() by first assigning IPv4 equivalents and then replacing that list with IPv6 equivalents before returning. If code expects both IPv4 and IPv6 aliases to be considered together, IPv4 aliases can be discarded when the target itself is not IP_ADDRESS; changes here need focused tests because current tests only assert a list is returned for address targets.
  • spiderfoot/plugin.py uses class-level defaults for mutable fields such as _listenerModules and opts, while clearListeners() and module setup() are expected to reset per-scan state. If a module keeps mutable state outside setup() or forgets to clear it, data can leak between scans or threaded instances; modules/sfp_template.py explicitly warns authors to reset class member variables in setup().