Knowledge Base updated: 2026-11-16
Knowledge Base Index
Orientation
This repository is Semgrep: a static-analysis scanner with Python CLI surfaces and an OCaml analysis core. The highest-level flow is a scan request that starts in cli/ or src/osemgrep/, resolves rules and targets, runs the OCaml parsing/matching/analysis pipeline, then reports findings and optional fixes.
Main parts:
- Python CLI (
cli/) — user-facing command package, output, Semgrep Core invocation, dependency-aware helpers, RPC/MCP hooks. - OCaml CLI (
src/main/,src/core_cli/,src/osemgrep/) — native command entrypoints and subcommand orchestration. - OCaml core engine (
src/core_scan/,src/engine/,src/matching/,src/analyzing/,src/tainting/) — target scheduling, parsing, matching, semantic/dataflow analysis, and taint mode. - Language frontends (
languages/, parsing libs,src/ast_generic/) — per-language parsers and the generic AST consumed by the engine. - Reporting and integrations (
src/reporting/,src/fixing/,src/rpc/,src/lsp_legacy/) — output formats, autofix, RPC, editor/server integrations. - Shared libraries and build infra (
libs/,scripts/,bin/, Nix/opam/Dune files) — reusable OCaml utilities and development/package tooling.
Critical state is mostly file-based rather than service-backed: rule/config files become internal rule models, target trees become scan target sets, manifests/lockfiles feed SCA, and CLI/runtime settings or metrics are local process/file state. There is no repository-owned database, queue, or object store in this codebase.
Architecture diagrams
flowchart LR
subgraph Inputs
Rules[Rule/config files]
Targets[Source targets]
Deps[Manifests and lockfiles]
end
subgraph CLI
PyCLI[cli Python package]
OCLI[osemgrep OCaml CLI]
end
subgraph Core
Targeting[src/targeting]
Parsing[languages + src/parsing]
AST[src/ast_generic]
Scan[src/core_scan]
Match[src/engine + src/matching]
Analysis[src/analyzing + src/tainting]
SCA[src/sca + semdep]
Output[src/reporting + src/fixing]
end
subgraph Integrations
RPC[src/rpc]
LSP[src/lsp_legacy]
MCP[cli MCP/RPC hooks]
end
Shared[libs + build infra]
Rules --> PyCLI
Rules --> OCLI
Targets --> PyCLI
Targets --> OCLI
Deps --> SCA
PyCLI --> Scan
OCLI --> Scan
Scan --> Targeting
Scan --> Parsing
Parsing --> AST
AST --> Match
Match --> Analysis
SCA --> Match
Match --> Output
Analysis --> Output
RPC --> Scan
LSP --> Scan
MCP --> PyCLI
Shared --> Core
flowchart TD
Entry[CLI scan command] --> Config[Resolve rules and options]
Config --> Targets[Discover and filter targets]
Targets --> Jobs[Build language jobs]
Jobs --> Parse[Parse target files]
Parse --> Match[Run matching engine]
Match --> Taint[Taint and dataflow modes]
Match --> Results[Core match results]
Taint --> Results
Results --> Report[JSON/text output]
Results --> Fix[Autofix output]
flowchart TD
RuleFile[Rule file] --> ParseRule[src/parsing]
ParseRule --> RuleModel[src/rule]
RuleModel --> Pattern[Pattern/formula/xpattern]
Pattern --> Matcher[src/engine + src/matching]
TargetFile[Target source file] --> LangParser[languages + parsing libs]
LangParser --> GenericAST[src/ast_generic]
GenericAST --> Matcher
Matcher --> Findings[Findings with metavars]
flowchart LR
Editor[Editor/tool client] --> LSP[src/lsp_legacy]
Tool[External RPC caller] --> RPC[src/rpc]
LSP --> Helpers[scan helpers]
RPC --> CoreScan[src/core_scan]
Helpers --> CoreScan
CoreScan --> Results[Results]
Results --> LSP
Results --> RPC
Glossary
- osemgrep — OCaml-native Semgrep CLI implementation under
src/osemgrep/. - semdep — Python dependency/SCA support package under
cli/src/semdep/. - SCA — software composition/dependency analysis, centered on manifests, lockfiles, packages, and dependency rules.
- nosemgrep — inline suppression mechanism handled in reporting/output paths.
- xpattern — extended pattern abstraction used by matching engines beyond normal AST patterns.
Codebase-wide invariants
- Dune is the primary OCaml build graph: the repo has
dune-project, top-leveldune, and subsystemdunefiles such assrc/core_scan/dune,src/engine/dune, andsrc/reporting/dune. - OCaml subsystems commonly expose implementation/interface pairs where APIs are public, evidenced by pairs such as
src/core_scan/Core_scan.ml+src/core_scan/Core_scan.mli,src/targeting/Find_targets.ml+src/targeting/Find_targets.mli, andsrc/reporting/Core_json_output.ml+src/reporting/Core_json_output.mli. - The Python CLI is packaged separately under
cli/, evidenced by cli/pyproject.toml, cli/setup.py,cli/Makefile, and package code undercli/src/semgrep/andcli/src/semdep/. - Language support is organized as per-language directories under
languages/plus shared parsing/generic AST layers (libs/lib_parsing/**,libs/lib_parsing_tree_sitter/**,src/ast_generic/**).
Routing table
cli/src/semdep/**-> docs/sca-dependency-analysis.md- cli/…/dependency_aware_rule.py -> docs/sca-dependency-analysis.md
- cli/…/dependency_path.py -> docs/sca-dependency-analysis.md
- cli/…/resolve_dependency_source.py -> docs/sca-dependency-analysis.md
- cli/…/resolve_subprojects.py -> docs/sca-dependency-analysis.md
- cli/…/sca_subproject_support.py -> docs/sca-dependency-analysis.md
- cli/…/subproject.py -> docs/sca-dependency-analysis.md
- cli/…/rpc.py -> docs/rpc-lsp-integrations.md
- cli/…/rpc_call.py -> docs/rpc-lsp-integrations.md
cli/src/semgrep/mcp/**-> docs/rpc-lsp-integrations.mdcli/**-> docs/python-cli.mdsrc/main/**-> docs/osemgrep-cli.mdsrc/core_cli/**-> docs/osemgrep-cli.mdsrc/osemgrep/**-> docs/osemgrep-cli.mdsrc/core_scan/**-> docs/core-scan-pipeline.mdsrc/core/**-> docs/core-scan-pipeline.mdsrc/target/**-> docs/core-scan-pipeline.mdsrc/testing/**-> docs/core-scan-pipeline.mdsrc/rule/**-> docs/rule-config-parsing.mdsrc/parsing/**-> docs/rule-config-parsing.mdsrc/configuring/**-> docs/rule-config-parsing.mdinterfaces/**-> docs/rule-config-parsing.mdlanguages/**-> docs/language-frontends-and-ast.mdsrc/ast_generic/**-> docs/language-frontends-and-ast.mdlibs/lib_parsing/**-> docs/language-frontends-and-ast.mdlibs/lib_parsing_tree_sitter/**-> docs/language-frontends-and-ast.mdlibs/ocaml-tree-sitter-semgrep/**-> docs/language-frontends-and-ast.mdsrc/engine/**-> docs/matching-engine.mdsrc/matching/**-> docs/matching-engine.mdsrc/aliengrep/**-> docs/matching-engine.mdsrc/spacegrep/**-> docs/matching-engine.mdsrc/prefiltering/**-> docs/matching-engine.mdsrc/metachecking/**-> docs/matching-engine.mdsrc/analyzing/**-> docs/analysis-taint-dataflow.mdsrc/il/**-> docs/analysis-taint-dataflow.mdsrc/naming/**-> docs/analysis-taint-dataflow.mdsrc/typing/**-> docs/analysis-taint-dataflow.mdsrc/tainting/**-> docs/analysis-taint-dataflow.mdsrc/targeting/**-> docs/target-discovery-and-filtering.md.semgrepignore-> docs/target-discovery-and-filtering.mdsrc/reporting/**-> docs/reporting-output-autofix.mdsrc/printing/**-> docs/reporting-output-autofix.mdsrc/fixing/**-> docs/reporting-output-autofix.mdsrc/sca/**-> docs/sca-dependency-analysis.mdsrc/rpc/**-> docs/rpc-lsp-integrations.mdsrc/lsp_legacy/**-> docs/rpc-lsp-integrations.mdlibs/**-> docs/shared-libraries-and-build-infra.mdbin/**-> docs/shared-libraries-and-build-infra.mddev/**-> docs/shared-libraries-and-build-infra.mdscripts/**-> docs/shared-libraries-and-build-infra.mdopam-lockfiles/**-> docs/shared-libraries-and-build-infra.mddune-> docs/shared-libraries-and-build-infra.mddune-project-> docs/shared-libraries-and-build-infra.mdMakefile-> docs/shared-libraries-and-build-infra.mdsemgrep.opam-> docs/shared-libraries-and-build-infra.mdsemgrep.opam.template-> docs/shared-libraries-and-build-infra.md- setup.py -> docs/shared-libraries-and-build-infra.md
pysemgrep.nix-> docs/shared-libraries-and-build-infra.mdsemgrep.nix-> docs/shared-libraries-and-build-infra.mdflake.nix-> docs/shared-libraries-and-build-infra.md- flake.lock -> docs/shared-libraries-and-build-infra.md
Dockerfile-> docs/shared-libraries-and-build-infra.mdtree-sitter-config.mk-> docs/shared-libraries-and-build-infra.md- tree-sitter-config.sh -> docs/shared-libraries-and-build-infra.md
- depot.json -> docs/shared-libraries-and-build-infra.md
- any change to Semgrep scan command dispatch or CLI output behavior -> docs/python-cli.md, docs/osemgrep-cli.md
- any change to target selection, ignore handling, or skipped-file reporting -> docs/target-discovery-and-filtering.md
- any change to parsing a rule file, formula, pattern, metavariable type, or rule option -> docs/rule-config-parsing.md
- any change to adding or updating language parsing support -> docs/language-frontends-and-ast.md
- any change to match semantics, metavariable behavior, generic matching, or specialized pattern matchers -> docs/matching-engine.md
- any change to taint, dataflow, CFG/IL lowering, naming, or typing behavior -> docs/analysis-taint-dataflow.md
- any change to dependency-aware rules, lockfile/manifest interpretation, package ecosystems, or SBOM output -> docs/sca-dependency-analysis.md
- any change to JSON/text output, nosemgrep suppressions, metavariable replacement, or autofix generation -> docs/reporting-output-autofix.md
- any change to RPC, LSP, MCP, or editor/tool integration behavior -> docs/rpc-lsp-integrations.md
- any change to repository build, packaging, shared OCaml utility libraries, or development scripts -> docs/shared-libraries-and-build-infra.md