Build Daily
Tinley Park · September 9, 2026The "Whack-a-Mole" Problem Every AI Team Hits
This video explains how an AI team uses user feedback and LangSmith to identify issues in production traces and convert them into evals. The process helps codify expected behavior and avoid the 'whack-a-mole' problem of fixing individual issues without establishing a broader standard.
The outline
Tap a timestamp to jump there in the video.
Getting Feedback from the Wild
The team emphasizes the importance of releasing a product to gather user feedback, which is stored in their system and LangSmith for trace review.
Codifying Learnings into Evals
The team takes learnings from individual traces and codifies them into evals to represent the desired behavior and performance bar.
The Whack-a-Mole Problem
Fixing individual issues without a broader strategy leads to a 'whack-a-mole' game, which needs to be addressed by creating specific evals for related problems.
Building Datasets in LangSmith
The team builds datasets in LangSmith, finding it highly efficient to translate specific production traces directly into eval cases.
Solo builders can use user feedback to identify specific traces and codify them into evals, creating a structured loop that prevents recurring issues. This approach helps maintain a clear bar for behavior while optimizing over time.