Plugin/Event/Target Runtime
Owned paths/globs for this doc:
- spiderfoot/plugin.py
- spiderfoot/event.py
- spiderfoot/target.py
- spiderfoot/threadpool.py
- modules/sfp_template.py
- test/…/test_spiderfootplugin.py
- test/…/test_spiderfootevent.py
- test/…/test_spiderfoottarget.py
- test/…/test_spiderfootthreadpool.py
Overview
This area defines the core runtime contract for SpiderFoot modules: SpiderFootPlugin in spiderfoot/plugin.py is the base class module authors subclass, SpiderFootEvent in spiderfoot/event.py is the data envelope passed between modules, SpiderFootTarget in spiderfoot/target.py models the original scan target and aliases used for scope checks, and SpiderFootThreadPool in spiderfoot/threadpool.py executes module work concurrently. modules/sfp_template.py is the canonical author-facing example showing how a plugin declares metadata, options, watched/produced events, external queries, duplicate suppression, event creation, and listener notification.
Plugin instances are stateful. SpiderFootPlugin holds the current scan id (setScanId() / getScanId()), current target (setTarget() / getTarget()), database handle (setDbh()), listener modules, optional input/output event queues, an optional shared thread pool, and stop/error flags. Module authors override setup(sf, userOpts), watchedEvents(), producedEvents(), handleEvent(event), and optionally finish() and running. The template in modules/sfp_template.py shows setup() storing the SpiderFoot API object in self.sf, merging userOpts into class-level defaults, initializing per-module duplicate tracking via self.tempStorage(), and setting self.__dataSource__ when the module wants a data-source label distinct from the module name.
Events are created by constructing SpiderFootEvent(eventType, data, module, sourceEvent) in spiderfoot/event.py. A ROOT event is special: it has no parent, its sourceEventHash and hash are both effectively the literal root identity (sourceEventHash == "ROOT", hash == "ROOT"), and its module may be empty. Non-root events require a non-empty module name and a parent SpiderFootEvent; the constructor records creation time, defaults confidence and visibility to 100, defaults risk to 0, and builds a random per-instance id that is hashed with SHA-256 for hash. asDict() emits the reduced event view used by consumers: integer generated, type, data, module, and parent source data when present.
Event dispatch is handled by SpiderFootPlugin.notifyListeners() in spiderfoot/plugin.py. It first validates that the argument is a SpiderFootEvent, applies __outputFilter__ unless the event is ROOT or the target's own event type, drops empty data, and stops early if checkForStop() says the scan should abort. It walks the source-event chain to detect repeated event type/data pairs; if the same item already appears upstream, it marks the event as storeOnly, meaning only storage modules whose __module__ contains __stor should see it. If outgoingEventQueue is configured, dispatch is asynchronous and the event is placed onto that queue. Otherwise listeners are sorted by _priority, filtered by listener.watchedEvents() (or *), assigned _currentEvent, and invoked synchronously through listener.handleEvent(sfEvent) in the current thread.
Threaded module execution is layered on top of queue-based dispatch. SpiderFootPlugin.start() starts a module-owned Python thread whose target is threadWorker() in spiderfoot/plugin.py. The worker creates a fresh SpiderFootDb handle for the thread, requires both incomingEventQueue and outgoingEventQueue, polls incoming events, submits normal events to handleEvent, and submits the sentinel string FINISHED to finish. poolExecute() runs storage modules whose names start with sfp__stor_ inline, but sends all other callbacks to the shared SpiderFootThreadPool with a task name of <module_name>_threadWorker and the module's maxThreads limit. SpiderFootPlugin.running reports whether the shared pool still has queued or running work for that task name.
SpiderFootThreadPool in spiderfoot/threadpool.py owns fixed worker threads and per-task queues. start() creates ThreadPoolWorker threads once; submit(callback, *args, taskName='default', maxThreads=100, ...) blocks while that task has maxThreads queued/running calls, then places (callback, args, kwargs) into that task's input queue. Workers round-robin across all task queues, execute callbacks, and optionally place return values into the task's output queue when saveResult=True. map() feeds an iterable through a background inputThread, starts the pool, and yields results(taskName, wait=True). shutdown(wait=True) drains saved results until finished, sets stop, clears input/output queues, and returns results grouped by task name. The unit coverage in test/…/test_spiderfootthreadpool.py exercises map(), direct submit(), saved results, named task queues, and mixed map/submit use.
Targets in spiderfoot/target.py constrain what a scan is about. SpiderFootTarget(targetValue, typeName) accepts only target types in _validTypes such as IP_ADDRESS, IPV6_ADDRESS, NETBLOCK_OWNER, NETBLOCKV6_OWNER, INTERNET_NAME, EMAILADDR, HUMAN_NAME, BGP_AS_OWNER, PHONE_NUMBER, USERNAME, and BITCOIN_ADDRESS. Aliases are stored as lowercase dictionaries with type and value through setAlias(), ignoring invalid or duplicate inputs. getNames() returns INTERNET_NAME aliases plus the target value for INTERNET_NAME and EMAILADDR targets. getAddresses() returns IP address aliases plus the current value for IP targets. matches(value, includeParents=False, includeChildren=True) is the runtime scope check: human names, phone numbers, usernames, and bitcoin addresses match everything; IP values match aliases or membership in IP/netblock targets using netaddr; domain-like values match exact names, optional parent domains, and optional child/subdomain names.
For module authors, modules/sfp_template.py is the practical lifecycle reference. A plugin subclass sets meta, opts, and optdescs; initializes mutable state in setup() rather than relying on class attributes; declares input types in watchedEvents() and output types in producedEvents(); keeps external API code in a helper such as query(); and in handleEvent() reads event.eventType, event.module, and event.data, suppresses duplicates in self.results, checks options before expensive expansion, creates child events with SpiderFootEvent(..., self.__name__, parent_event), calls self.notifyListeners(child_event), and checks self.checkForStop() inside loops so aborts can take effect. The template also shows how parser/extractor modules can preserve upstream data-source attribution by copying event.moduleDataSource to events they produce.
The unit tests under test/…/test_spiderfootplugin.py, test/…/test_spiderfootevent.py, and test/…/test_spiderfoottarget.py mostly assert constructor validation, simple accessors, dispatch type checks, target matching behavior, and the basic stop path. Several plugin tests are smoke tests with placeholder assertions, so behavior such as listener ordering, store-only delivery, threaded queue drainage, and output-filter edge cases is more strongly defined by spiderfoot/plugin.py itself than by precise unit assertions.
Coupling
- If adding or renaming event types emitted by a module template or base-plugin flow, update both
producedEvents()/watchedEvents()implementations and any target-type assumptions inSpiderFootPlugin.notifyListeners(); the output filter in spiderfoot/plugin.py compares strings againstROOT,self.getTarget().targetType, and__outputFilter__, so event-type drift silently drops dispatch. - If changing the
SpiderFootEventconstructor or validation in spiderfoot/event.py, update module examples in modules/sfp_template.py; module authors are expected to pass the current event assourceEvent, and non-root events will fail immediately if the parent is not aSpiderFootEvent. - If changing threaded execution in spiderfoot/plugin.py, verify spiderfoot/threadpool.py task naming and
SpiderFootPlugin.running; both use the exact<module_name>_threadWorkertask-name convention for queue/running counts. - If changing queue payloads in
SpiderFootPlugin.threadWorker(), verify producers and consumers handle bothSpiderFootEventobjects and the literalFINISHEDsentinel;threadWorker()treats that string as the cleanup hook trigger and submitsfinish()instead ofhandleEvent(). - If changing
SpiderFootTarget._validTypesin spiderfoot/target.py, update test/…/test_spiderfoottarget.py and audit any modules that expect the target's type to be a possible event type;SpiderFootPlugin.notifyListeners()explicitly allows events whose type equalsself.getTarget().targetTypethrough the output filter. - If changing result saving in
SpiderFootThreadPool, keepsaveResult,taskName, andmaxThreadsas reserved kwargs;ThreadPoolWorker.run()removestaskNameandsaveResultbefore invoking callbacks, whileSpiderFootThreadPool.submit()removesmaxThreads, so callbacks should not receive those control fields. - If changing logging wrappers in spiderfoot/plugin.py, preserve the
extra={'scanId': self.__scanId__}shape used bydebug(),info(), anderror(); the customSpiderFootPluginLoggerexists so module logs keep the caller's filename and line number instead of reporting the base wrapper.
Risks
- spiderfoot/plugin.py has historically fragile threading paths: recent history includes
418ef64 Added spiderfoot-integrated thread pool,2c39b7f slaying the threading dragons, and1f35646 Shared threadpool. IfthreadWorker(),poolExecute(), or queue cleanup changes without exercising aborted scans and crashinghandleEvent()callbacks, scans can hang because stale objects remain inincomingEventQueueorincomingEventQueueis left usable aftererrorState. - spiderfoot/threadpool.py workers swallow callback exceptions after logging and then continue the outer loop. If callers rely on
saveResult=True, a callback failure produces no result object and no exception toshutdown()/results(), so code that assumes one output per submitted item can silently lose data. - spiderfoot/target.py implements
getAddresses()by first assigning IPv4 equivalents and then replacing that list with IPv6 equivalents before returning. If code expects both IPv4 and IPv6 aliases to be considered together, IPv4 aliases can be discarded when the target itself is notIP_ADDRESS; changes here need focused tests because current tests only assert a list is returned for address targets. - spiderfoot/plugin.py uses class-level defaults for mutable fields such as
_listenerModulesandopts, whileclearListeners()and modulesetup()are expected to reset per-scan state. If a module keeps mutable state outsidesetup()or forgets to clear it, data can leak between scans or threaded instances; modules/sfp_template.py explicitly warns authors to reset class member variables insetup().