> ## Documentation Index
> Fetch the complete documentation index at: https://www.greptile.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluate Greptile with the CLI

> Review closed and merged pull requests, collect JSON findings, and run repeatable evaluations.

Use the [Greptile CLI](https://www.greptile.com/docs/code-review/greptile-cli.md) to review historical changes without reopening a PR. The CLI reviews a local Git diff, not the PR's state. Reviews are saved to your Greptile account; they do not post comments on the PR.

CLI reviews default to Base, regardless of the repository's tier. Add `--plus` or `--apex` to each run to [evaluate another tier](https://www.greptile.com/docs/code-review/review-tiers.md). Unavailable tiers fail rather than falling back. Each run uses review credits.

## Before you start

* [Install the CLI](https://www.greptile.com/docs/code-review/greptile-cli.md#install). These examples use v3.8.0 and Bash.
* [Enable the repository in Greptile](https://www.greptile.com/docs/code-review/cli-onboarding.md) with an active trial or paid plan.
* Run `greptile login`, or supply `GREPTILE_API_KEY` through your CI secret store. Do not put keys in scripts or command-line arguments.
* For the GitHub examples below, install and authenticate the [GitHub CLI](https://cli.github.com/).

Use a separate clone so switching commits does not disturb your work. Keep the full Git history:

```bash theme={}
git clone https://github.com/OWNER/REPO.git greptile-eval
cd greptile-eval
```

## Review a closed PR

Replace `OWNER/REPO`, `123`, and `main` with your repository, PR number, and target branch.

> [tab] Closed, not merged
>
> Fetch the PR head even if its source branch was deleted. Choose the base, then create a local branch at the PR head:
>
> ```bash theme={}
> PR=123
> BASE_BRANCH=main
> git fetch origin "$BASE_BRANCH"
> git fetch origin "refs/pull/$PR/head"
> HEAD_SHA=$(git rev-parse FETCH_HEAD)
> BASE_SHA=$(git merge-base "origin/$BASE_BRANCH" "$HEAD_SHA")
> git switch -c "greptile-eval-$PR" "$HEAD_SHA"
> ```
>
> This uses the target branch's current history. For an exact historical evaluation, use a saved base commit from before the PR closed instead.

> [tab] Merged
>
> Review the merged result against the target branch immediately before the merge:
>
> ```bash theme={}
> PR=123
> MERGE_SHA=$(gh api "repos/OWNER/REPO/pulls/$PR" --jq '.merge_commit_sha')
> git fetch origin "$MERGE_SHA"
> git switch -c "greptile-eval-$PR" "$MERGE_SHA"
> BASE_SHA=$(git rev-parse HEAD^1)
> ```
>
> This works for merge commits and squash merges. It evaluates the landed change, which can differ from the original PR head if the merge resolved conflicts.
>
> For GitHub **rebase merges**, `MERGE_SHA` above points to the last rebased commit. Replace the base calculation with the PR's commit count so the review includes all commits, not just the last one:
>
> ```bash theme={}
> N=$(gh api "repos/OWNER/REPO/pulls/$PR" --jq '.commits')
> BASE_SHA=$(git rev-parse "HEAD~$N")
> git log --oneline "$BASE_SHA"..HEAD
> ```
>
> Check that these are exactly the rebased PR commits. This assumes all `N` commits landed as one linear sequence. Otherwise, use the known parent of the first rebased commit. To evaluate the original PR instead, use its saved base and head SHAs.

Check the diff, then run the review:

```bash theme={}
git diff --stat "$BASE_SHA"...HEAD
greptile review --branch "$BASE_SHA" --json > review.json 2> review.log
```

> [!WARNING]
>
> Do not use today's default branch as the base for a merged PR: it already contains the change, so the diff may be empty. `--branch` accepts a commit SHA and reviews `merge-base(BASE, HEAD)...HEAD`. The CLI also requires a named local branch, not a detached HEAD.

Only committed, non-binary changes are sent. Check stderr for warnings about sensitive files held back from review. See [sensitive files](https://www.greptile.com/docs/code-review/greptile-cli.md#include-a-sensitive-file) before overriding exclusions.

## Read the results

`--json` writes one review object to stdout. Progress and errors go to stderr. With [jq](https://jqlang.org/), extract the findings:

```bash theme={}
jq '.comments[] | {path, startLine, endLine, severity, securityIssue, body}' review.json
jq '.comments | length' review.json
```

The object also includes `summary`, `confidence`, `confidenceReasoning`, and `securitySummary`. `comments: []` means no findings. Exit code `0` means the review completed, **not** that it found no bugs. Treat any nonzero exit as an incomplete run; do not score it as a clean review.

## Run a batch

Pin each case's base and head to full commit SHAs. Use unique PR numbers and no header in `dataset.csv`:

```text theme={}
123,BASE_COMMIT_SHA,HEAD_COMMIT_SHA
456,BASE_COMMIT_SHA,HEAD_COMMIT_SHA
```

Replace the placeholders with your evaluation commits. The base should be an ancestor of the head; record the merge base, not a moving branch name. For original-PR evaluations, use the original PR head rather than a merge or squash commit.

Run this from a fresh evaluation clone with `dataset.csv` in its root. It needs `jq` and fetch access to each commit. Each successful case keeps its full JSON plus metadata in a JSONL file; failed cases keep a log and are skipped:

```bash theme={}
set -euo pipefail
OUT=$(mktemp -d "$PWD/results.XXXXXX")
RUN_ID=${OUT##*.}
greptile --version > "$OUT/cli-version.txt"

while IFS=, read -r pr base head || [[ -n "$pr" ]]; do
  git fetch origin "$base" "$head"
  git merge-base --is-ancestor "$base" "$head"
  git switch -c "greptile-eval-$RUN_ID-$pr" "$head"
  if greptile review --branch "$base" --json \
      > "$OUT/$pr.json" 2> "$OUT/$pr.log"; then
    jq -c --arg pr "$pr" --arg base "$base" --arg head "$head" \
      '{pr: $pr, base: $base, head: $head, review: .}' \
      "$OUT/$pr.json" >> "$OUT/reviews.jsonl"
  else
    printf 'PR %s failed; see %s/%s.log\n' "$pr" "$OUT" "$pr" >&2
  fi
done < dataset.csv
```

Git fetch, ancestry, or checkout errors stop the batch so it cannot review the wrong commit. For deleted GitHub branches, fetch `refs/pull/PR_NUMBER/head` to recover the PR head before building the dataset.

## Score the evaluation

* Label known bugs for each diff before reading Greptile's output.
* Match findings by behavior and location, not exact wording. Count each bug once.
* Track precision (valid findings / all findings) and recall (known bugs found / known bugs).
* Keep the CLI version, commit SHAs, tier, instructions, and review config with the results.

Use the same cases and settings across comparisons. Repeat runs to measure variation. Fixed SHAs fix the diff, not the service's models or organization context; this is a fresh review of historical code, not a replay of the original PR review.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.