Rise of the Overnight Agents

Daksh Gupta - co-founder, Greptile

I'm Daksh, a co-founder of Greptile. We're working on automating code validation, starting with AI agents that review and test PRs. Over the last year we've reviewed several million PRs across 65,000 organizations.

In the nearly two years that Greptile has been reviewing PRs, we have seen the composition of the PRs change dramatically. In 2024, most code was hand-written with some AI autocomplete from Cursor. In 2025, agents had started to work, and programmers were letting Claude Code edit multiple files at once.

In 2026, we observed a step function increase in the adoption of fully end-to-end coding agents, which turned tickets into PRs with no human intervention. In the extreme cases, I saw reports of enthusiastic vibecoders experimenting with polyphasic sleep; their agents code 24/7 and they wake up every now and then to give them their next task.

These anecdotes intrigued me, but I was skeptical that any real work in any real codebases could be done by completely autonomous "overnight" agents.

Being a code review company, we have data on millions of pull requests, any written by agents. I decided to study the data to answer some questions:

• Were real companies with real codebases coding this way?
• Were the end-to-end AI-generated PRs any good?
• In what ways did they fail?

First I had to figure out which PRs were end-to-end AI-generated

Harder than I expected. I figured that if an agent was the "author" of the PR in GitHub, it was likely the agent wrote nearly all the code. Unfortunately fewer than 1% of PRs in my dataset listed a bot (devin-ai-integration[bot], claude[bot]) as the author on GitHub, and I suspected the percentage of vibed PRs was quite a bit more.

Thankfully, it turns out Claude and some others add a Co-Authored-By: (Claude|Devin) as a footer to the PR description if you have them open the PR. Nearly 20% of PRs reviewed by Greptile in March had these footers.

Codex doesn't leave a footer, but the Codex app names branches like codex/<task-slug>. There were 32k codex/-prefixed PRs in March alone. Cursor's background agents (the ones that open PRs autonomously, not the IDE inline-edit feature) tag branches with cursor/.

With all of these signals stacked, I had a good floor for which PRs were likely end-to-end generated by AI.

Sure enough, fully AI-generated PRs have only recently become a large share of total PRs opened. Only 0.86% of PRs in Feb 2025 had evidence of being fully vibed. In April 2026, the share was 27.6%.

Next, I had to find a way to measure PR quality.

Method 1: Using revert rates

My first heuristic was that a PR was probably not great if it had to be reverted. GitHub's auto-revert format is revert-<orig-pr-number>-<orig-branch>, so each revert pins back to the original PR. Mar 15 – Apr 14, 2026:

Two of the four agents had lower revert rates than humans. That surprised me, so I started looking for an explanation.

One theory: maybe humans were delegating the easy work to agents and keeping the hard work for themselves. If a developer hands the simple PRs to an agent and writes the complicated ones themselves, the comparison isn't agent vs. human. It's agent on easy work vs. the same person on hard work.

To check this, I needed a proxy for task complexity. The number of files changed felt like a reasonable one. So I compared the median PR sizes for individual developers between PRs that did and didn't have evidence of being end-to-end AI generated.

Same-developer PR sizes (engineers who opened ≥3 AI PRs and ≥3 non-AI PRs in April 2026):

The opposite. The same developer's AI PRs are about 20% larger by median LOC than their non-AI PRs. Looks like reversion rates for AI-generated PRs are pretty similar to revert-rates for human-written PRs.

I was curious how revert rates correlated with PR size, because I suspected larger PRs that were AI-generated had higher revert rates.

Same developers, revert rate broken out by PR size:

Wrong again. For larger PRs, revert rates were actually higher for human-written PRs than agent-written PRs.

Method 2: Code churn

Reverts are rare (4 per 1,000). I wanted a denser signal. Churn is the obvious one: if you touch a file and a week later someone else touches it again, that's a form of rework.

I don't have the full diff per PR, but I know which files Greptile commented on. For that subsample (about 40% of merged PRs), I ask: does any file the PR touched get modified by a different author in a later PR within 7 days? Same-author follow-ups are usually iteration ("I'll ship it in four more PRs"), not rework.

Per agent, broken out by PR size, comparing each agent against the human baseline at equal complexity:

Codex sits around 5–6% rework, Claude around 8%. Human stays glued to ~10%. Cursor BG and Devin are at or above human in most buckets. The ranking is the same one we saw in revert rate.

Method 3: Greptile comments on AI-generated PRs

Reverts and churn can happen for any number of reasons, some critical and other minor. I wanted some more granularity.

Every PR in my dataset was reviewed by Greptile. Every Greptile comment is tagged:

• P0 (critical, will break in prod)
• P1 (real bug, won't necessarily page anyone)
• P2 (style / nit / wrong-idiom).

I wanted to know if agents produced more P0s than humans.

Issues per 10,000 merged LOC, April 2026:

A number of agents actually produce fewer P0s than humans. This is also true for P1s and P2s.

Method 4: How many review rounds to merge

Comments per PR is the obvious denser signal but it scales with PR size, so it's hard to read directly. An ostensibly better metric is to see how many review cycles a PR goes through before it is merged. In other words, how much did the agent-generated PR need to be revised before it could be merged.

A PR that opens, gets approved, and merges = 1 cycle. A PR that gets fixes pushed and re-reviewed = 2. And so on.

Intuitively, larger PRs (by lines of code) are revised more often than smaller ones.

Average cycles to merge by PR size (cross-population, all authors, April 2026):

Every agents' revisions per PR are fairly consistent with humans.

What are the patterns of failure?

I didn't see any evidence that agent-generated PRs were any worse than human-written PRs. However, I was curious if the patterns of failure were different.

I decided to run a keyword search across Greptile comments left on PRs generated by the various agents. Keywords like "off-by-one" or "sql injection".

Then, I computed how often each agent produced that type of error relative to the human baseline. I have included an assortment of the most interesting rows here:

A few things stood out per agent:

• Cursor BG is the only column where most categories sit above 2× the human rate. Its three highest cells are n+1 query (3.45×), "breaks existing behavior" (2.37×), and missing tests (2.37×). Off-by-one is at 2.27×.

• Codex's above-human categories cluster around configuration and breakage: env-var / config bugs (1.35×), "breaks existing" (1.34×), and "secret in logs" (1.34×). Most other categories are at or below the human rate.

• Claude's highest cells are IDOR / missing tenant check (1.75×), stale comment / wrong doc (1.69×), off-by-one (1.64×), and XSS (1.57×). Auth bypass is at 1.50×.

• Devin is at or below the human rate on every category I tested except env-var / config bugs (1.35×). Its security cells (auth bypass 0.5×, IDOR 0.69×, SQL injection 0.7×) are well below human.

A note on Devin: its low rates here don't square cleanly with its high revert rate from Method 1. The simplest explanation I can offer is that the things Devin gets wrong are not the things this keyword sweep catches - for example, "completed the wrong task" wouldn't show up in any of these categories but would still get reverted. I don't have a way to confirm this from the data.

I think I was wrong

I went in expecting agent PRs to look noticeably worse. Across all four methods, they don't. Two of the four agents had lower revert rates than humans. Every agent except Cursor BG flagged fewer P0s per 10k LOC. Review cycles all clustered within 0.4 of the human mean.

So real codebases are letting these things ship code, and the code is roughly fine. 27.6% of merged PRs in April were end-to-end AI-generated, and that number is still climbing.

The interesting part is what each agent gets wrong. Cursor BG over-indexes on n+1s. Claude on tenancy and auth. Codex on config. The bugs don't disappear, they just move. Whatever review process you've built for human PRs probably wasn't built for those.

Appendix: shortcomings of my methods

• Adverse selection. While there was no discernable difference in PR size between human and agent PRs, I can't help but feel that humans choosing to hand write a PR for a certain task rather than use an agent might say something about the riskiness of the change.
• Line-level churn. "What fraction of the lines an agent merged this month are still in the file in three months" is the gold-standard quality signal. I can't compute it because we don't store diffs.
• Cross-agent contamination. A Codex PR might have had a Claude Code review pass over it before merge; a Claude PR might have had Cursor inline-edits before the headless run. My classifier picks one agent per PR.
• Agent self-review vs Greptile review. Cursor's background agents and Codex now ship with their own pre-PR review hooks. PRs that were already cleaned up by an in-agent review look better to Greptile than PRs that weren't, and I can't see from the GitHub side whether that pre-review happened.
• The "human" code. Everyone with a keyboard in 2026 has Cursor's IDE running. It won't surprise me if the "human" PRs are nearly as AI-generated as the ones attributed to an agent.

If you have feedback on my methodology or conclusions, I'd love to hear from you. My email is daksh@greptile.com.