Evaluate Greptile with the CLI

Use the Greptile CLI to review historical changes without reopening a PR. The CLI reviews a local Git diff, not the PR’s state. Reviews are saved to your Greptile account; they do not post comments on the PR. CLI reviews default to Base, regardless of the repository’s tier. Add --plus or --apex to each run to evaluate another tier. Unavailable tiers fail rather than falling back. Each run uses review credits.

Before you start

Install the CLI. These examples use v3.8.0 and Bash.
Enable the repository in Greptile with an active trial or paid plan.
Run greptile login, or supply GREPTILE_API_KEY through your CI secret store. Do not put keys in scripts or command-line arguments.
For the GitHub examples below, install and authenticate the GitHub CLI.

Use a separate clone so switching commits does not disturb your work. Keep the full Git history:

git clone https://github.com/OWNER/REPO.git greptile-eval
cd greptile-eval

Review a closed PR

Replace OWNER/REPO, 123, and main with your repository, PR number, and target branch.

Closed, not merged
Merged

Fetch the PR head even if its source branch was deleted. Choose the base, then create a local branch at the PR head:

PR=123
BASE_BRANCH=main
git fetch origin "$BASE_BRANCH"
git fetch origin "refs/pull/$PR/head"
HEAD_SHA=$(git rev-parse FETCH_HEAD)
BASE_SHA=$(git merge-base "origin/$BASE_BRANCH" "$HEAD_SHA")
git switch -c "greptile-eval-$PR" "$HEAD_SHA"

This uses the target branch’s current history. For an exact historical evaluation, use a saved base commit from before the PR closed instead.

Review the merged result against the target branch immediately before the merge:

PR=123
MERGE_SHA=$(gh api "repos/OWNER/REPO/pulls/$PR" --jq '.merge_commit_sha')
git fetch origin "$MERGE_SHA"
git switch -c "greptile-eval-$PR" "$MERGE_SHA"
BASE_SHA=$(git rev-parse HEAD^1)

This works for merge commits and squash merges. It evaluates the landed change, which can differ from the original PR head if the merge resolved conflicts.For GitHub rebase merges, MERGE_SHA above points to the last rebased commit. Replace the base calculation with the PR’s commit count so the review includes all commits, not just the last one:

N=$(gh api "repos/OWNER/REPO/pulls/$PR" --jq '.commits')
BASE_SHA=$(git rev-parse "HEAD~$N")
git log --oneline "$BASE_SHA"..HEAD

Check that these are exactly the rebased PR commits. This assumes all N commits landed as one linear sequence. Otherwise, use the known parent of the first rebased commit. To evaluate the original PR instead, use its saved base and head SHAs.

Check the diff, then run the review:

git diff --stat "$BASE_SHA"...HEAD
greptile review --branch "$BASE_SHA" --json > review.json 2> review.log

Do not use today’s default branch as the base for a merged PR: it already contains the change, so the diff may be empty. --branch accepts a commit SHA and reviews merge-base(BASE, HEAD)...HEAD. The CLI also requires a named local branch, not a detached HEAD.

Only committed, non-binary changes are sent. Check stderr for warnings about sensitive files held back from review. See sensitive files before overriding exclusions.

Read the results

--json writes one review object to stdout. Progress and errors go to stderr. With jq, extract the findings:

jq '.comments[] | {path, startLine, endLine, severity, securityIssue, body}' review.json
jq '.comments | length' review.json

The object also includes summary, confidence, confidenceReasoning, and securitySummary. comments: [] means no findings. Exit code 0 means the review completed, not that it found no bugs. Treat any nonzero exit as an incomplete run; do not score it as a clean review.

Run a batch

Pin each case’s base and head to full commit SHAs. Use unique PR numbers and no header in dataset.csv:

123,BASE_COMMIT_SHA,HEAD_COMMIT_SHA
456,BASE_COMMIT_SHA,HEAD_COMMIT_SHA

Replace the placeholders with your evaluation commits. The base should be an ancestor of the head; record the merge base, not a moving branch name. For original-PR evaluations, use the original PR head rather than a merge or squash commit. Run this from a fresh evaluation clone with dataset.csv in its root. It needs jq and fetch access to each commit. Each successful case keeps its full JSON plus metadata in a JSONL file; failed cases keep a log and are skipped:

set -euo pipefail
OUT=$(mktemp -d "$PWD/results.XXXXXX")
RUN_ID=${OUT##*.}
greptile --version > "$OUT/cli-version.txt"

while IFS=, read -r pr base head || [[ -n "$pr" ]]; do
  git fetch origin "$base" "$head"
  git merge-base --is-ancestor "$base" "$head"
  git switch -c "greptile-eval-$RUN_ID-$pr" "$head"
  if greptile review --branch "$base" --json \
      > "$OUT/$pr.json" 2> "$OUT/$pr.log"; then
    jq -c --arg pr "$pr" --arg base "$base" --arg head "$head" \
      '{pr: $pr, base: $base, head: $head, review: .}' \
      "$OUT/$pr.json" >> "$OUT/reviews.jsonl"
  else
    printf 'PR %s failed; see %s/%s.log\n' "$pr" "$OUT" "$pr" >&2
  fi
done < dataset.csv

Git fetch, ancestry, or checkout errors stop the batch so it cannot review the wrong commit. For deleted GitHub branches, fetch refs/pull/PR_NUMBER/head to recover the PR head before building the dataset.

Score the evaluation

Label known bugs for each diff before reading Greptile’s output.
Match findings by behavior and location, not exact wording. Count each bug once.
Track precision (valid findings / all findings) and recall (known bugs found / known bugs).
Keep the CLI version, commit SHAs, tier, instructions, and review config with the results.

Use the same cases and settings across comparisons. Repeat runs to measure variation. Fixed SHAs fix the diff, not the service’s models or organization context; this is a fresh review of historical code, not a replay of the original PR review.

Getting Started

Code Review

Integrations

TREX

Self-Hosting

Account

How Greptile Works

Evaluate Greptile with the CLI

Before you start

Review a closed PR

Read the results

Run a batch

Score the evaluation

​Before you start

​Review a closed PR

​Read the results

​Run a batch

​Score the evaluation

Before you start

Review a closed PR

Read the results

Run a batch

Score the evaluation