Knowledge Base updated: 2026-11-16
Knowledge Base Index
1. Orientation
This repository is the OSINT Framework static/Worker-backed site. The highest-level flow is: Cloudflare Worker serves the public app, the browser loads public/arf.json, and the UI renders the OSINT resource catalog for exploration.
Main parts:
- Cloudflare Worker runtime — request entrypoint and deployment/runtime configuration.
- Public explorer UI — HTML, JavaScript, D3 rendering, and CSS for the interactive catalog.
- OSINT catalog data — arf.json plus enrichment artifacts that describe resources and metadata.
- Maintenance tools — Python and Node utilities for extracting, checking, rechecking, pruning, and enriching catalog links.
- Project tooling and CI — package metadata, lockfiles, ignore rules, and GitHub Actions validation.
Critical state lives in files rather than an external database:
- public/arf.json is the primary OSINT resource catalog consumed by the browser UI and maintenance scripts.
- enrichment-batch4-domains.json and THE-127-domain-enrichment-batch1.md are enrichment inputs/records used around catalog metadata updates.
- package-lock.json and tools/…/package-lock.json pin Node dependency state for root tooling and the link recheck workflow.
2. Architecture diagrams
flowchart TD
subgraph Runtime
Worker[src/worker.js]
Wrangler[wrangler.jsonc]
end
subgraph PublicSite[public/]
HTML[index.html]
UI[public/js/arf.js]
D3[public/js/d3.min.js]
CSS[public/css/*.css]
Catalog[public/arf.json]
end
subgraph Maintenance
Merge[merge_enrichments.py]
PyTools[tools/*.py]
Recheck[tools/link-recheck/recheck.mjs]
Enrichment[enrichment artifacts]
end
subgraph Automation
RootPkg[package.json]
CI[.github/workflows/ci.yml]
end
Wrangler --> Worker
Worker --> HTML
Worker --> UI
Worker --> CSS
UI --> D3
UI --> Catalog
Merge --> Catalog
Enrichment --> Merge
PyTools --> Catalog
Recheck --> Catalog
RootPkg --> CI
CI --> PublicSite
CI --> Runtime
sequenceDiagram
participant Browser
participant Worker as Cloudflare Worker
participant Static as public assets
participant Catalog as public/arf.json
Browser->>Worker: Request site URL
Worker->>Static: Resolve index/assets
Static-->>Worker: HTML, JS, CSS
Worker-->>Browser: Public explorer app
Browser->>Worker: Request arf.json
Worker->>Catalog: Serve catalog data
Catalog-->>Browser: Resource graph JSON
Browser->>Browser: Render and interact with graph
flowchart LR
Batch[enrichment batch files] --> Merge[merge_enrichments.py]
Catalog[public/arf.json] --> Merge
Merge --> UpdatedCatalog[updated public/arf.json]
UpdatedCatalog --> UI[public/js/arf.js]
UpdatedCatalog --> LinkTools[tools link checkers]
3. Glossary
- OSINT Framework — the resource catalog and explorer UI maintained by this repository.
arf— the catalog/UI naming used by public/arf.json, public/…/arf.js, and related styles.- THE-127 — a domain-enrichment batch artifact represented by THE-127-domain-enrichment-batch1.md.
4. Codebase-wide invariants
- The primary catalog is a checked-in JSON file, not an external service; both the UI and maintenance tools route through public/arf.json.
- Runtime deployment is Cloudflare Worker-oriented; the Worker entrypoint is src/worker.js and its deployment configuration is wrangler.jsonc.
- Link maintenance is split across Python utilities under tools/*.py and a separate Node workflow under
tools/link-recheck/**, each with its own dependency context where applicable. - Browser presentation assets live under
public/with a simple split between public/index.html,public/js/**,public/css/**, and public/arf.json.
5. Routing table
-
any change to Worker request handling, asset serving, or Cloudflare runtime configuration -> docs/cloudflare-worker-runtime.md
-
any browser rendering, graph interaction, data-loading, or visual styling change -> docs/public-osint-explorer-ui.md
-
enrichment-batch4-domains.json -> docs/osint-resource-data.md
-
THE-127-domain-enrichment-batch1.md -> docs/osint-resource-data.md
-
any change to catalog schema, resource metadata conventions, or enrichment data shape -> docs/osint-resource-data.md
-
changes to enrichment merge behavior involving enrichment-batch4-domains.json or public/arf.json -> docs/enrichment-merge-flow.md
-
tools/extract_urls.py -> docs/python-link-maintenance-tools.md
-
tools/link_checker.py -> docs/python-link-maintenance-tools.md
-
tools/remove_dead_links.py -> docs/python-link-maintenance-tools.md
-
Python-based URL extraction, link checking, or dead-link removal changes -> docs/python-link-maintenance-tools.md
-
tools/link-recheck/**-> docs/node-link-recheck-flow.md -
Node-based link recheck workflow changes -> docs/node-link-recheck-flow.md
-
.gitignore-> docs/project-tooling-and-ci.md -
any root dependency, script, CI, or repository hygiene change -> docs/project-tooling-and-ci.md