Introducing a new way to review your PRs.

Read the announcement

smicallef / spiderfoot

SpiderFoot automates OSINT for threat intelligence and mapping your attack surface.

View on GitHub

13 documents · Updated Nov 16, 2026

docs / identity-email-account-and-social-modules

Identity Email, Account, and Social Enrichment Modules

Overview

This doc owns the identity-enrichment modules that turn text, domains, email addresses, phone numbers, usernames, human names, company names, social URLs, and physical addresses into richer SpiderFoot events. Owned paths/globs: modules/sfp_accounts.py, modules/sfp_abstractapi.py, modules/sfp_apple_itunes.py, modules/sfp_callername.py, modules/sfp_clearbit.py, modules/sfp_company.py, modules/sfp_crossref.py, modules/sfp_debounce.py, modules/sfp_dehashed.py, modules/sfp_email.py, modules/sfp_emailformat.py, modules/sfp_emailrep.py, modules/sfp_fullcontact.py, modules/sfp_gleif.py, modules/sfp_gravatar.py, modules/sfp_h1nobbdde.py, modules/sfp_haveibeenpwned.py, modules/sfp_hunter.py, modules/sfp_keybase.py, modules/sfp_leakix.py, modules/sfp_myspace.py, modules/sfp_nameapi.py, modules/sfp_names.py, modules/sfp_neutrinoapi.py, modules/sfp_numverify.py, modules/sfp_opencorporates.py, modules/sfp_openstreetmap.py, modules/sfp_phone.py, modules/sfp_skymem.py, modules/sfp_slideshare.py, modules/sfp_snov.py, modules/sfp_social.py, modules/sfp_sociallinks.py, modules/sfp_socialprofiles.py, modules/sfp_textmagic.py, modules/sfp_trumail.py, modules/sfp_twilio.py, modules/sfp_twitter.py, modules/sfp_venmo.py, test/…/test_sfp_accounts.py, test/…/test_sfp_email*.py, test/…/test_sfp_haveibeenpwned.py, test/…/test_sfp_hunter.py, test/…/test_sfp_names.py, test/…/test_sfp_phone.py, test/…/test_sfp_social*.py, test/…/test_sfp_accounts.py, and test/…/test_sfp_email*.py.

The extraction layer starts from content already gathered by other modules. modules/sfp_email.py scans web content, WHOIS/RIR blobs, leak content, raw DNS/file metadata, certificates, and banners with SpiderFootHelpers.extractEmailsFromText, validates the mail domain through self.sf.validHost, and classifies each address as EMAILADDR, EMAILADDR_GENERIC, or AFFILIATE_EMAILADDR based on target matching and _genericusers. modules/sfp_names.py emits HUMAN_NAME either by converting firstname.lastname@domain email usernames or by scoring capitalized word pairs from content against dictionary/name wordlists; it only trusts RAW_RIR_DATA from an allowlist of identity-rich modules such as sfp_clearbit, sfp_fullcontact, sfp_hunter, sfp_opencorporates, sfp_slideshare, sfp_twitter, sfp_gravatar, and sfp_keybase. modules/sfp_company.py extracts COMPANY_NAME or AFFILIATE_COMPANY_NAME by looking for legal suffixes in target/affiliate content, certificates, and WHOIS data. modules/sfp_phone.py parses phone numbers from web and WHOIS content with phonenumbers.PhoneNumberMatcher, emits normalized E.164 PHONE_NUMBER values, and looks up local libphonenumber carrier names as PROVIDER_TELCO.

Email discovery modules query domain-oriented sources and feed the same email event stream. modules/sfp_emailformat.py fetches https://www.email-format.com/d/{domain}/, parses the result table or page body, filters extracted addresses back to the target domain, skips masked hashes, and emits EMAILADDR or EMAILADDR_GENERIC. modules/sfp_skymem.py searches Skymem and then follows up to 20 domain-result pages, extracting only target-matching addresses. modules/sfp_hunter.py requires a Hunter API key, pages domain-search results in batches of 10, emits discovered addresses, and passes first_name/last_name through RAW_RIR_DATA as “Possible full name” for sfp_names.py to consume. modules/sfp_snov.py obtains an OAuth token from Snov using api_key_client_id and api_key_client_secret, pages domain-emails-with-info using lastId, stores the response as RAW_RIR_DATA, validates every email with SpiderFootHelpers.validEmail, and emits target addresses. modules/sfp_fullcontact.py enriches domains and emails through FullContact v3; domain responses produce emails, phones, locations, physical addresses, and key-person names, while email responses only emit a possible full name via RAW_RIR_DATA. modules/sfp_clearbit.py enriches an EMAILADDR through Clearbit, emitting possible full names, person/company physical addresses, company aliases as INTERNET_NAME or AFFILIATE_INTERNET_NAME, company site phone numbers, and company site email addresses.

Email verification, reputation, disposable-address, and breach modules attach risk context to addresses already found. modules/sfp_debounce.py, modules/sfp_trumail.py, and modules/sfp_nameapi.py all watch EMAILADDR, call their disposable-email APIs, and emit EMAILADDR_DISPOSABLE plus RAW_RIR_DATA only when the service reports disposable status. modules/sfp_emailrep.py queries EmailRep for each address, optionally without an API key but with a warning, and emits EMAILADDR_COMPROMISED when details.credentials_leaked is true, MALICIOUS_EMAILADDR when details.malicious_activity is true, and RAW_RIR_DATA only when one of those conditions exists. modules/sfp_haveibeenpwned.py requires a HIBP key, checks both EMAILADDR and PHONE_NUMBER breach endpoints, emits EMAILADDR_COMPROMISED or PHONE_NUMBER_COMPROMISED per breach name, and for email addresses also queries paste data, reconstructs supported paste URLs, refetches them, verifies the original email appears in the paste content, and emits LEAKSITE_URL plus LEAKSITE_CONTENT. modules/sfp_dehashed.py requires both Dehashed username and API key, searches email:"addr" or email:"@domain", and emits discovered domain email addresses, compromised email/password/hash events, and raw rows. modules/sfp_h1nobbdde.py searches the unofficial HackerOne disclosure timeline for a DOMAIN_NAME and emits VULNERABILITY_DISCLOSURE links whose title contains the queried domain. modules/sfp_leakix.py is host/domain oriented but contributes to the breach flow by querying LeakIX for IP_ADDRESS or DOMAIN_NAME, emitting host/service details and LEAKSITE_CONTENT from leak records that still belong to the target host/domain.

Phone enrichment begins with PHONE_NUMBER events from modules/sfp_phone.py, modules/sfp_gravatar.py, modules/sfp_fullcontact.py, modules/sfp_sociallinks.py, or other modules. modules/sfp_numverify.py, modules/sfp_abstractapi.py, and modules/sfp_neutrinoapi.py validate or enrich phone numbers with GEOINFO and carrier/provider data; AbstractAPI additionally emits PROVIDER_TELCO only when its response is valid. modules/sfp_callername.py is US-only (+1), scrapes CallerName for caller location and emits MALICIOUS_PHONE_NUMBER if unsafe votes exceed safe votes. modules/sfp_textmagic.py emits PHONE_NUMBER_TYPE from TextMagic lookup responses. modules/sfp_twilio.py uses Twilio lookup with the caller-name add-on and emits the returned caller name as COMPANY_NAME plus RAW_RIR_DATA. HIBP also watches phone numbers for breach findings in modules/sfp_haveibeenpwned.py.

Account and username discovery is centered on modules/sfp_accounts.py. It downloads and caches the WhatsMyName wmn-data.json site list as sfaccountsv2, filters invalid sites, and accepts EMAILADDR, DOMAIN_NAME, HUMAN_NAME, and USERNAME. Non-username inputs are converted into candidate usernames: email local parts, domain keywords, and human-name forms with spaces removed or replaced by dots. Candidates are filtered by generic users, common human names, dictionary words, and minimum length, then emitted as USERNAME. Account checks only run for USERNAME events so that candidates have passed that filtering stage; checkSites() fans out over the WhatsMyName definitions using _maxthreads, records positive matches as ACCOUNT_EXTERNAL_OWNED, and optionally checks typo/permutation variants as SIMILAR_ACCOUNT_EXTERNAL. Before the first real lookup, sfp_accounts.py tests a random username and caches always-positive sites in sfaccounts_state_v3 so noisy services can be distrusted for later scans.

Social profile discovery has two shapes: recognizing URLs already seen, and querying external services for profiles. modules/sfp_social.py watches LINKED_URL_EXTERNAL, matches fixed regexes for LinkedIn, GitHub, Bitbucket, Gitlab, Facebook, MySpace, YouTube, Twitter, SlideShare, Instagram, and Google+, emits a normalized SOCIAL_MEDIA string like Network: <SFURL>url</SFURL>, and emits a USERNAME for every matched network except Google+. modules/sfp_socialprofiles.py takes HUMAN_NAME, uses Bing or Google Custom Search to look for Facebook/Google+/LinkedIn URLs, emits the raw search response as RAW_RIR_DATA, and optionally “tightens” candidate profiles by fetching each profile and requiring one of the target domain keywords to appear before emitting SOCIAL_MEDIA. modules/sfp_sociallinks.py is a commercial Social Links integration over USERNAME, EMAILADDR, and PHONE_NUMBER; it queries Telegram by phone or alias, Flickr/Skype/LinkedIn by email, and emits human names, usernames, phone numbers, social URLs, job titles, company names, locations, account IDs, and raw results. modules/sfp_gravatar.py maps an EMAILADDR to an MD5 Gravatar profile and can emit usernames, possible names, phone numbers, related email addresses, instant-messenger accounts, and social accounts. modules/sfp_keybase.py accepts usernames, Keybase URLs, or domains; verifies username responses match the requested username; and emits raw profile data, newly discovered usernames, SOCIAL_MEDIA proof links, GEOINFO, BITCOIN_ADDRESS, and PGP_KEY blocks.

Network-specific profile enrichers consume either SOCIAL_MEDIA or USERNAME events. modules/sfp_myspace.py can search MySpace by email to emit a MySpace SOCIAL_MEDIA profile, and later reads MySpace profile pages for location. modules/sfp_slideshare.py and modules/sfp_twitter.py parse profile pages identified by SOCIAL_MEDIA, emitting possible full names as RAW_RIR_DATA and valid locations as GEOINFO. modules/sfp_venmo.py queries the public Venmo user API for a USERNAME, derives a display name or first/last name, emits HUMAN_NAME, and stores raw user data. modules/sfp_apple_itunes.py is domain-to-app identity enrichment: it reverses a domain (example.com to com.example), searches iTunes software results, requires the bundle id to contain the reversed target domain, emits APPSTORE_ENTRY, internal/affiliate seller hosts, internal seller URLs, and raw result JSON.

Organization and address enrichment modules connect company names, legal identifiers, domains, and locations. modules/sfp_abstractapi.py handles company enrichment for DOMAIN_NAME, phone validation for PHONE_NUMBER, and IP geolocation for IP events; the company endpoint emits COMPANY_NAME, LinkedIn company SOCIAL_MEDIA, GEOINFO, and raw data. modules/sfp_gleif.py searches GLEIF autocomplete results for a COMPANY_NAME, emits candidate LEI values and raw search data, retrieves each valid LEI record, and emits legal names plus legal/headquarters PHYSICAL_ADDRESS values. modules/sfp_opencorporates.py searches OpenCorporates for exact company-name matches at a configured confidence, then extracts registered addresses, previous company names, and officer names from both search results and detailed company records. modules/sfp_openstreetmap.py geocodes PHYSICAL_ADDRESS values via Nominatim to PHYSICAL_COORDINATES, skipping PO boxes and trying to strip attention/floor/suite prefixes before lookup. modules/sfp_crossref.py is the affiliate-content bridge: it fetches external linked URLs, similar domains, co-hosted sites, or darknet mention URLs, searches their content (or the external base URL) for target names, and on a match emits AFFILIATE_INTERNET_NAME plus AFFILIATE_WEB_CONTENT; those affiliate content events can then be parsed by sfp_email.py and sfp_company.py into affiliate identity findings.

The owned tests in test/…/test_sfp_accounts.py, test/…/test_sfp_email*.py, test/…/test_sfp_haveibeenpwned.py, test/…/test_sfp_hunter.py, test/…/test_sfp_names.py, test/…/test_sfp_phone.py, and test/…/test_sfp_social*.py mostly assert option/metadata shape and basic event behavior; test/…/test_sfp_social.py directly verifies a LinkedIn URL emits SOCIAL_MEDIA, while test/…/test_sfp_email.py has a skipped email-extraction behavior test. Integration coverage for this area is under test/…/test_sfp_accounts.py and test/…/test_sfp_email*.py.

Coupling

Risks

  • modules/sfp_accounts.py stores cross-thread site results in self.siteResults and writes it from worker threads guarded by a lock, but handleEvent() calls res = self.checkSites(user) after iterating for user in set(users) and therefore uses the last loop variable rather than the event username if multiple candidates were built. If username generation changes to produce multiple candidates for a USERNAME event, account lookups can run against the wrong value.
  • modules/sfp_debounce.py, modules/sfp_trumail.py, modules/sfp_nameapi.py, and modules/sfp_textmagic.py set self.results[eventData] = True but do not check it before querying. If duplicate EMAILADDR or PHONE_NUMBER events reach these modules, they can repeat external API calls and consume quota despite having temp storage.
  • modules/sfp_openstreetmap.py computes a cleaned address value but calls self.query(eventData) instead of self.query(address). If callers rely on the prefix stripping in that module, geocoding will still use the unmodified PHYSICAL_ADDRESS and may miss coordinates.
  • Several social profile parsers scrape brittle HTML (modules/sfp_myspace.py, modules/sfp_slideshare.py, modules/sfp_twitter.py, modules/sfp_callername.py, and modules/sfp_h1nobbdde.py). If a provider changes markup, the modules will silently emit fewer names, locations, vulnerability disclosures, or malicious-phone findings rather than failing loudly.