Haize Labs with Leonard Tang - Weaviate Podcast #121!

How do you ensure your AI systems actually do what you expect them to do? Leonard Tang takes us deep into the revolutionary world of AI evaluation with concrete techniques you can apply today. Learn how Haize Labs is transforming AI testing through "scaling judge-time compute" - stacking weaker models to effectively evaluate stronger ones. Leonard unpacks the game-changing Verdict library that outperforms frontier models by 10-20% while dramatically reducing costs. Discover practical insights on creating contrastive evaluation sets that extract maximum signal from human feedback, implementing debate-based judging systems, and building custom reward models that align with enterprise needs. The conversation reveals powerful nuggets like using randomized agent debates to achieve consensus and lightweight guardrail models that run alongside inference. Whether you're developing AI applications or simply fascinated by how we'll ensure increasingly powerful AI systems perform as expected, this episode delivers immediate value with techniques you can implement right away, philosophical perspectives on AI safety, and a glimpse into the future of evaluation that will fundamentally shape how AI evolves. Learn more about Haize Labs! - https://www.haizelabs.com/ Check out Verdict on GitHub - https://github.com/haizelabs/verdict Chapters 0:00 Weaviate Podcast #121! 0:46 Welcome Leonard! 1:16 Founding Haize Labs 8:31 UX for Evals 17:01 Scaling Judge-Time Compute with Verdict 23:26 Debate Judges 26:06 Compute Scaling 28:50 Declarative Judge Pipelines 31:13 Custom Reward Models 37:20 Reasoning in Reward Models 39:20 Mechanistic Interpretability 45:30 Guardrails in Inference Pipelines 47:35 Can we control Superintelligence? 52:08 Exciting Directions for AI