Build Daily
Tinley Park · October 21, 2026Using GPT to Judge GPT Is a Mistake, Here's Why
Unify CTO Connor Heggie argues that using the same LLM family (e.g., GPT) to both run an agent and grade its outputs creates a circular evaluation that measures agreement rather than true quality. He advocates for a cross-family, almost adversarial judge to produce meaningful evals for production agents.
The outline
Every point the video makes, in the order it makes them.
- 01
Self-evaluation bias
When an agent and its eval judge both run on GPT, the system tends to measure agreement with itself rather than actual output quality.
- 02
Cross-family judging
Connor Heggie explains that an LLM judge should come from a different model family than the agent it is grading to avoid circular reasoning.
- 03
Adversarial judge relationship
Heggie argues the judge-to-agent relationship should be almost adversarial so the evaluation genuinely challenges the agent's outputs.
- 04
LLM-as-a-judge methodology
The episode addresses the broader practice of using LLMs as evaluators and the pitfalls of naive same-model judging in agent pipelines.
- 05
Production agent evaluation
The discussion is framed within the context of building and evaluating production-grade AI agents, as covered on LangChain's Max Agency podcast.
- 06
Unify's perspective
As CTO of Unify, Connor Heggie brings a practitioner's view on how to structure eval systems that yield trustworthy signals for agent quality.
A solo builder picks the model, ships the agent, and writes the evals in the same afternoon, so the pull to grade with the same family is strong. But as the material notes, that produces mode collapse and overlapping distributions — the eval measures agreement, not quality. Swapping in a cross-family judge is a small architectural decision that keeps the eval from becoming a rubber stamp.