AI Code Review That Engineers Actually Trust

Every engineering team we talk to has tried an AI code review tool. Almost none of them still have it turned on six months later. We built one internally, killed two versions of it, and the third one stuck. This is what changed.

The metric that misled us

Our first version optimized for comment volume — more findings felt like more value. It was wrong. Engineers muted the bot within the first sprint because most of the comments were technically correct and practically useless: naming nitpicks, redundant null checks that couldn't actually be reached, stylistic preferences already covered by the linter.

> Note: If your AI reviewer generates more noise than your human reviewers do, the team will route around it — permanently. Trust, once lost on a tool, rarely comes back.

We rebuilt around a single constraint: every comment has to name a specific failure mode, not a style preference. If the bot can't point to a concrete way the code breaks, it stays quiet.

What the second version got right, and still missed

Version two scoped comments to genuine correctness and security issues — race conditions, unhandled error paths, injection risks. Precision went up. Engineers started reading the comments again. But it still missed context that any senior engineer on the team would have caught immediately: this service is mid-migration, that pattern is intentional technical debt with a tracked ticket, this function is deliberately synchronous because of a downstream constraint.

// Simplified shape of the context we now attach to every review pass
type ReviewContext = {
  filePastReviews: ReviewComment[];
  linkedTickets: TicketRef[];
  ownerTeam: string;
  migrationState: "none" | "in-progress" | "complete";
};

async function reviewDiff(diff: Diff, context: ReviewContext) {
  const findings = await model.analyze(diff, { context, mode: "correctness-only" });
  return findings.filter((f) => f.confidence > CONFIDENCE_THRESHOLD);
}

Review confidence distribution before and after adding review context

Feeding it the context a human reviewer already has

The fix wasn't a bigger model — it was giving the same model access to context a human reviewer pulls from memory: recent commits to the file, open tickets referencing the code path, the owning team's known conventions, and whether the surrounding module is mid-migration. Once the model had that context, false-positive rate on our internal benchmark dropped from around one-in-four comments to under one-in-twenty.

What shipped

The version running today comments on three categories only: correctness bugs, security issues, and violations of conventions the team itself has documented. It cites the specific line and the specific failure condition every time, and it stays silent rather than guess when confidence is low.

> Pull Quote: "The best compliment an AI reviewer can get isn't 'this caught something' — it's that nobody remembers it's still running."

Six months in, it's still enabled on every repository, and two engineers have told us independently they'd notice if it went away. That's the bar we should have set from the start.