Introducing a new way to review your PRs.

Read the announcement

lockfale / osint-framework

OSINT Framework

View on GitHub

8 documents · Updated Nov 16, 2026

docs / node-link-recheck-flow

Overview

The Node-based recheck flow is the second stage of the scheduled/manual link-check job for catalog URLs. The CI workflow in .github/…/ci.yml first runs tools/extract_urls.py, which reads public/arf.json, skips placeholder/template URLs, and writes urls.txt. Lychee then checks that URL list and writes its JSON output to lychee/out.json without failing the job. The owned Node entrypoint, tools/…/recheck.mjs, consumes that lychee JSON and decides whether any reported errors still fail when loaded in a real browser.

tools/…/package.json defines a private ESM package with one runtime dependency, playwright@1.63.0; tools/…/package-lock.json pins both playwright and playwright-core and records their Node engine requirement (>=20). The CI job uses actions/setup-node with Node 22, runs npm ci in tools/link-recheck, then installs Chromium with npx playwright install --with-deps chromium before invoking node recheck.mjs ../../lychee/out.json ../../link-recheck-report.md.

tools/…/recheck.mjs accepts node recheck.mjs <lychee-json> [report.md]. If the output path is omitted it writes link-recheck-report.md relative to the current working directory. lycheeFailures() reads error_map, fail_map, and timeout_map from the lychee report so the script works with both newer and older lychee JSON shapes; each map is expected to contain input-file keys whose values are result arrays with url and status fields. The script deduplicates by URL in insertion order and preserves lychee's status text as the comparison reason shown in the final report.

For each URL, recheck.mjs launches Playwright Chromium and creates one browser context with a non-headless-looking user agent (HeadlessChrome is replaced with Chrome) and en-US locale. Four workers share the context (CONCURRENCY = 4). Each URL is opened with page.goto(..., waitUntil: "domcontentloaded", timeout: 45_000), while a response listener tracks the last main-frame navigation response. After the first navigation, the script waits up to 10 seconds for JavaScript challenge pages to redirect to a real page; it considers the page settled when the main document is a non-202 status below 400, or a terminal 404/410.

Validation is intentionally browser-oriented rather than equivalent to lychee's HTTP probe. classify() returns ok when the browser gets a status below 400, failing for navigation errors, 404/410, and 5xx statuses, and blocked for 401/403/429 or challenge-page content containing markers such as just a moment, checking your browser, verify you are human, cf-chl, or captcha. Other 4xx responses are treated as blocked, not dead, because the goal is to avoid failing CI for sites that are up but reject automated checks. Network errors, DNS failures, certificate failures, refused/reset connections, timeouts, and 5xx responses remain failing and cause process exit code 1.

The output is a Markdown report from renderReport(). It starts with a summary table for Still failing, Up, but blocks automated browsers, and Loaded fine, then emits URL tables per non-empty group with the browser result and original lychee reason. The report is written to the requested path and appended to $GITHUB_STEP_SUMMARY when that environment variable is present. The workflow uploads both lychee/out.json and link-recheck-report.md as the link-check-report artifact; only the Node recheck's count of still-failing URLs determines whether the CI link-check job fails.

This flow complements, rather than replaces, the Python maintenance tools. tools/extract_urls.py is the CI source of truth for which catalog URLs from public/arf.json are checked by lychee, while tools/link_checker.py is a separate urllib-based audit tool that extracts URLs directly from public/arf.json and writes tools/link_audit_report.json. The Node flow does not read public/arf.json directly; its input is the lychee JSON generated from urls.txt.

Coupling

Risks

  • tools/…/recheck.mjs assumes every result in lychee's failure maps has a stable url field. If lychee emits a different result shape, lycheeFailures() can deduplicate everything under undefined or omit failures, causing the link-check job to pass without rechecking the intended URLs.
  • tools/…/recheck.mjs shares one Playwright browser context across four concurrent workers. If a checked site mutates shared context state such as cookies, locale-dependent challenges, or per-domain rate limits, another URL checked in the same run may be classified differently than it would in a fresh browser context.

Routing and ownership