Introducing a new way to review your PRs.

Read the announcement

lockfale / osint-framework

OSINT Framework

View on GitHub

8 documents · Updated Nov 16, 2026

docs / python-link-maintenance-tools

Overview

This area covers the Python utilities that audit and prune links in the OSINT resource catalog stored in public/arf.json. The catalog is a nested JSON tree rooted at name: "OSINT Framework", where category folders carry children arrays and link resources are represented as type: "url" nodes with a url field plus metadata such as description, status, pricing, opsec, and related booleans. The maintenance scripts assume they are run from the repository root because their paths are hard-coded as public/arf.json, urls.txt, and tools/link_audit_report.json.

tools/extract_urls.py is the CI-oriented URL extraction entrypoint. Running python3 tools/extract_urls.py loads public/arf.json as JSON, recursively walks every node through children, and collects only url values that start with http. Before writing urls.txt, it filters out template URLs that are not intended to resolve literally: angle-bracket placeholders such as <username> and percent-encoded placeholder forms matched by %3C.*?%3E, plus two explicit sample/template URLs (https://www.facebook.com/photo.php?fbid=PHOTO-ID-HERE and https://api.opencnam.com/v2/phone/+19073372323). It prints skipped templates to stderr and writes one checked URL per line to urls.txt.

tools/link_checker.py is the full audit entrypoint. Running python3 tools/link_checker.py reads public/arf.json as raw text and extracts every unique http or https substring using a regular expression rather than the catalog tree walker. It then checks URLs concurrently with ThreadPoolExecutor(max_workers=20), using urllib.request.urlopen with a 30-second timeout, a browser-like OSINT Framework user agent, and certificate verification disabled in the SSL context. Results are classified as dead (status >= 400, or a non-404 error), redirected (final URL differs from requested URL), or slow (elapsed time at least 10 seconds). The script prints a human-readable audit report and writes the machine-readable report to tools/link_audit_report.json; its process exit code is the number of dead links, so any dead-link finding makes the command fail for shell/CI purposes.

tools/remove_dead_links.py is the destructive cleanup entrypoint and should only be run after reviewing tools/link_audit_report.json. Running python3 tools/remove_dead_links.py reads the audit report, selects only a narrow subset of dead URLs for removal, then rewrites public/arf.json. It deliberately skips .onion URLs, HTTP 403 bot-block cases, 5xx responses, and timeouts. It removes URLs only when the audit report shows status == 404, an error containing nodename nor servname, or an error containing Connection refused. Removal is recursive and only deletes leaf nodes: a node with no children is dropped if its url exactly matches a confirmed dead URL; folder/category nodes are preserved, with their children arrays rewritten after descendants are removed. The output JSON is formatted with two-space indentation, ",: " separators, and a trailing newline.

Safe maintenance flow:

  1. Run python3 tools/link_checker.py from the repo root to produce tools/link_audit_report.json and inspect the printed dead, redirected, and slow sections.
  2. Treat the report as advisory, especially for bot-blocked hosts and transient network failures. tools/link_checker.py disables TLS verification and does not skip template URLs, so use judgment before removing anything.
  3. If removals are warranted, run python3 tools/remove_dead_links.py and review the resulting diff to public/arf.json. Because it rewrites the full JSON file, check that only intended leaf URL resources were removed and no category structure was unintentionally emptied.
  4. Use python3 tools/extract_urls.py when the desired output is a filtered urls.txt for CI/link-check consumers that should not check placeholder templates.

Routing / ownership

This document owns:

Route changes here when modifying catalog URL extraction, link audit behavior, dead-link pruning, or the public/arf.json schema assumptions those scripts depend on.

Coupling

  • If public/arf.json changes away from the nested children tree with URL-bearing leaf objects, update both tools/extract_urls.py and tools/remove_dead_links.py; they traverse parsed JSON nodes and will silently miss or preserve data if URLs move to a different field or container shape.
  • If placeholder/template URL conventions in public/arf.json expand beyond angle brackets, percent-encoded angle brackets, PHOTO-ID-HERE, or the OpenCNAM sample number, update tools/extract_urls.py; otherwise CI consumers of urls.txt will start checking literals that were meant as user-filled patterns.
  • If maintainers expect the audit report and CI extraction to cover the same URL set, compare tools/link_checker.py with tools/extract_urls.py: the checker regexes raw JSON text and can include template URLs and URLs embedded inside non-resource strings, while the extractor only walks url fields that start with http and filters templates.
  • If the JSON report format in tools/link_checker.py changes, update tools/remove_dead_links.py at the same time; the remover reads report["dead"] entries with url, status, and error keys and does not validate alternate schemas.
  • If tools/remove_dead_links.py is broadened to delete more status classes, verify the link-checker classifications first; the checker treats most URL errors as dead, including bot blocks and network failures, while the remover currently narrows deletion to DNS failure, connection refused, and HTTP 404.

Risks