Build Daily
Tinley Park · October 13, 2026Three Questions Every Agent Engineer Should Be Able To Answer
Jake Broekhuizen of LangChain Labs argues that owning your own intelligence as an agent engineer comes down to three questions: where your agent is failing, how to reliably reproduce those failures, and what to change so they stop. He covers fine-tuned judges for monitoring at scale, building world specs and tasks that mimic production, and fixing failures at both the harness and model layers.
The outline
Tap a timestamp to jump there in the video.
What it means to own your own intelligence
The video opens by framing what it means for an agent engineer to truly own their own intelligence rather than depending on external model providers.
The three questions every agent engineer should answer
Jake Broekhuizen introduces the three core questions every agent engineer should answer: where is the agent failing, how can you reproduce the failure, and what must change to fix it.
Question 1: mining production traces to find failures
The first question focuses on mining production traces to identify where agents are actually failing in real-world use.
Fine-tuned judges for monitoring agents at scale
A fine-tuned judge model is presented as a cost-effective alternative to expensive frontier LLM-as-judge evaluators for monitoring agents at scale.
Question 2: reproducing failures beyond input and output evals
The second question addresses how to reproduce failures reliably, going beyond simple input-and-output evaluations to capture the full context of a failure.
World specs and tasks that mimic production
World specs and tasks are introduced as a method for building evaluation environments that closely mimic real production conditions.
Example: the world and tasks for LangChain's GTM agent
LangChain's internal GTM agent is used as a concrete example of how to define a world and associated tasks for evaluation.
Keeping tasks hard and hard to reward hack
The discussion covers strategies for keeping tasks genuinely hard and resistant to reward hacking so evaluations remain meaningful over time.
Turning production data into tasks and worlds
Production data is turned into structured tasks and worlds that can drive ongoing evaluation and improvement of the agent.
What app teams see that frontier labs can't
Application teams, who observe how real users interact with agents, can build tasks and environments that a frontier lab could never replicate.
Question 3: fixing failures at the harness and model layers
The third question focuses on fixing failures at both the harness layer (prompting, tooling, orchestration) and the model layer (fine-tuning, architecture).
Staying ahead as models and users change
The video closes with guidance on staying ahead as underlying models and user behaviors continuously evolve.
For a solo builder, the three-question framework—where it fails, how to reproduce, what to change—becomes the entire quality loop when no one else is watching. Fine-tuned judges and world specs that mimic production stop being infrastructure questions and become the only practical way to catch failures before users do. The harness-versus-model split matters most when there is no one else to hand the problem to.