How Mixpanel chose a code reviewer when agents write most of the PRs

Latency. Once you finally got human eyes on a PR, oftentimes it wasn't even ready for review to begin with. Now we're skipping a couple of steps. Greptile is fast and cost efficient, so you can iterate very quickly responding to its feedback.

The point at which a human gets their eyes on a PR: the PR has clearly been thought through, a lot of edge cases are shaken out, we can go directly into the meat of it and engage with the why behind the PR instead of wasting time hunting for flaws.

We had a shootout: try a bunch of providers, try to do it ourselves, all that kind of stuff.

I'd encourage anyone who doesn't have a coding agent running reviews on PRs to try building it yourself real quick. Depending on your PR volume and codebase it might actually be good enough.

In the end, we were unhappy with the high cost and the verbosity of our internal shot at building a code review agent. A lot of the reviews from our homebrew review agent embodied the worst reviews you might get from a human: a consistent body of nitpicks striped with actual showstopper issues you really must fix before merging.

Even if we could get it to stop bothering us with nitpicks, it would still be too expensive to run on every single push.

Which led us to think about what a good code review looks like. Some of the best reviews are a staff or principal engineer coming into your PR and giving you context from across the codebase: “you shouldn't do this, because of a thing you were never aware existed.”

Greptile threaded that needle. It's very cost effective, way better than what we can build, and it was giving that codebase-wide review experience.

Mixpanel is a giant distributed columnar database with a webapp on top. Once you get lower down the stack, it gets very stateful, irreversible data corruption becomes a concern, and the stakes get really high.

There was a PR from our infra team, and the PR description asserted that the change was a standalone change with no blast radius to other services. Greptile teased out that that wasn’t true, that this was transitively changing the behavior of some downstream service.

I was able to determine that despite the clues being nowhere in the diff at all. That's one of those "a staff engineer comes into your PR and just knows something you couldn't have possibly known" moments.

We got a lot of anecdotes like this during the POC which led to developers really appreciating having Greptile reviews available.

It started as: we think we need agentic code reviews because they're nice to have, we like using code review skills when driving our own personal agents. Now it's part of our larger agentic system, TAL, that's churning out PRs automatically.

We can make the cost per task of TAL lower because we can outsource the code review step. Having the adversarial review, or the different context Greptile has, is probably better than what we could cook up inside that coding agent anyway.

Directly integrating review into our coding agent isn't worthwhile, since Greptile is literally wall-clock-time faster and more optimized cost-wise.

I totally recommend people give it a shot. We all love it here.

When we were doing the POC, the trial lapsed, and everyone came out of the woodwork: “can you turn this thing back on?” “Can we get it back?” That was when I thought, “okay. I can't really imagine our workflow without something like Greptile.”