Python Link Maintenance Tools
Overview
This area covers the Python utilities that audit and prune links in the OSINT resource catalog stored in public/arf.json. The catalog is a nested JSON tree rooted at name: "OSINT Framework", where category folders carry children arrays and link resources are represented as type: "url" nodes with a url field plus metadata such as description, status, pricing, opsec, and related booleans. The maintenance scripts assume they are run from the repository root because their paths are hard-coded as public/arf.json, urls.txt, and tools/link_audit_report.json.
tools/extract_urls.py is the CI-oriented URL extraction entrypoint. Running python3 tools/extract_urls.py loads public/arf.json as JSON, recursively walks every node through children, and collects only url values that start with http. Before writing urls.txt, it filters out template URLs that are not intended to resolve literally: angle-bracket placeholders such as <username> and percent-encoded placeholder forms matched by %3C.*?%3E, plus two explicit sample/template URLs (https://www.facebook.com/photo.php?fbid=PHOTO-ID-HERE and https://api.opencnam.com/v2/phone/+19073372323). It prints skipped templates to stderr and writes one checked URL per line to urls.txt.
tools/link_checker.py is the full audit entrypoint. Running python3 tools/link_checker.py reads public/arf.json as raw text and extracts every unique http or https substring using a regular expression rather than the catalog tree walker. It then checks URLs concurrently with ThreadPoolExecutor(max_workers=20), using urllib.request.urlopen with a 30-second timeout, a browser-like OSINT Framework user agent, and certificate verification disabled in the SSL context. Results are classified as dead (status >= 400, or a non-404 error), redirected (final URL differs from requested URL), or slow (elapsed time at least 10 seconds). The script prints a human-readable audit report and writes the machine-readable report to tools/link_audit_report.json; its process exit code is the number of dead links, so any dead-link finding makes the command fail for shell/CI purposes.
tools/remove_dead_links.py is the destructive cleanup entrypoint and should only be run after reviewing tools/link_audit_report.json. Running python3 tools/remove_dead_links.py reads the audit report, selects only a narrow subset of dead URLs for removal, then rewrites public/arf.json. It deliberately skips .onion URLs, HTTP 403 bot-block cases, 5xx responses, and timeouts. It removes URLs only when the audit report shows status == 404, an error containing nodename nor servname, or an error containing Connection refused. Removal is recursive and only deletes leaf nodes: a node with no children is dropped if its url exactly matches a confirmed dead URL; folder/category nodes are preserved, with their children arrays rewritten after descendants are removed. The output JSON is formatted with two-space indentation, ",: " separators, and a trailing newline.
Safe maintenance flow:
- Run
python3 tools/link_checker.pyfrom the repo root to produce tools/link_audit_report.json and inspect the printed dead, redirected, and slow sections. - Treat the report as advisory, especially for bot-blocked hosts and transient network failures. tools/link_checker.py disables TLS verification and does not skip template URLs, so use judgment before removing anything.
- If removals are warranted, run
python3 tools/remove_dead_links.pyand review the resulting diff to public/arf.json. Because it rewrites the full JSON file, check that only intended leaf URL resources were removed and no category structure was unintentionally emptied. - Use
python3 tools/extract_urls.pywhen the desired output is a filtered urls.txt for CI/link-check consumers that should not check placeholder templates.
Routing / ownership
This document owns:
Route changes here when modifying catalog URL extraction, link audit behavior, dead-link pruning, or the public/arf.json schema assumptions those scripts depend on.
Coupling
- If public/arf.json changes away from the nested
childrentree with URL-bearing leaf objects, update both tools/extract_urls.py and tools/remove_dead_links.py; they traverse parsed JSON nodes and will silently miss or preserve data if URLs move to a different field or container shape. - If placeholder/template URL conventions in public/arf.json expand beyond angle brackets, percent-encoded angle brackets,
PHOTO-ID-HERE, or the OpenCNAM sample number, update tools/extract_urls.py; otherwise CI consumers of urls.txt will start checking literals that were meant as user-filled patterns. - If maintainers expect the audit report and CI extraction to cover the same URL set, compare tools/link_checker.py with tools/extract_urls.py: the checker regexes raw JSON text and can include template URLs and URLs embedded inside non-resource strings, while the extractor only walks
urlfields that start withhttpand filters templates. - If the JSON report format in tools/link_checker.py changes, update tools/remove_dead_links.py at the same time; the remover reads
report["dead"]entries withurl,status, anderrorkeys and does not validate alternate schemas. - If tools/remove_dead_links.py is broadened to delete more status classes, verify the link-checker classifications first; the checker treats most URL errors as dead, including bot blocks and network failures, while the remover currently narrows deletion to DNS failure, connection refused, and HTTP 404.
Risks
- tools/remove_dead_links.py rewrites public/arf.json in place. If it is run against an unreviewed or stale tools/link_audit_report.json, valid catalog entries can be removed because deletion is based on exact URL matches from the report rather than a fresh check.
- tools/link_checker.py disables certificate hostname and chain verification in
check_url. If TLS failure should count as a broken resource, this checker will not surface that class of problem and may report a link as OK when strict clients would reject it. - tools/link_checker.py and tools/extract_urls.py discover different URL sets. If maintainers assume the full audit excludes templates because CI extraction does, template literals in public/arf.json can appear as false dead links in tools/link_audit_report.json.