"Everyone says AI evaluations are important, so let's actually build one live from scratch.” Here's my new episode with @_amankhan (Arize) where we build AI evals for a customer support agent live, including: ✅ Creating the eval criteria ✅ Labeling the golden dataset ✅ Aligning LLM judges with human scores Some insights from Aman: 1. PMs must do manual labeling themselves. "I never found it useful to outsource human evals to contractors. The PM has to be in the spreadsheet to maintain good judgment." 2. Define what good/average/bad looks like on criteria like accuracy and tone upfront. This becomes your rubric for consistent evaluation across your team. 3. Make sure your LLM judges align with your human scores before you scale. Test the judges on a few dozen cases first and aim for at least 80%+ match rate. 📌 Watch now: Also available on: Spotify: Apple: Newsletter: