Most Evals Measure the Wrong Thing
📰 Dev.to · Aamer Mihaysi
Learn why most AI evaluations measure the wrong thing and how to focus on reliability instead of capability
Action Steps
- Read the awesome-evals list on GitHub to understand current evaluation methods
- Identify the limitations of lab evaluations that only test capability
- Design production evaluations that test reliability in messy environments
- Implement testing for ambiguous instructions and tool call failures
- Monitor and debug your AI models in real-world scenarios
Who Needs to Know This
AI engineers and developers can benefit from understanding the difference between lab and production evaluations to improve their models' reliability
Key Insight
💡 Lab evaluations measure capability, while production evaluations measure reliability
Share This
Most AI evals measure capability, not reliability. Focus on production evals to ensure your models work in messy environments #AI #reliability
Key Takeaways
Learn why most AI evaluations measure the wrong thing and how to focus on reliability instead of capability
Full Article
Title: Most Evals Measure the Wrong Thing
URL Source: https://dev.to/o96a/most-evals-measure-the-wrong-thing-c76
Published Time: 2026-07-11T14:38:38Z
Markdown Content:
[Skip to content](https://dev.to/o96a/most-evals-measure-the-wrong-thing-c76#main-content)
[](https://dev.to/)
[Powered by Algolia](https://www.algolia.com/developers/?utm_source=devto&utm_medium=referral)
[Log in](https://dev.to/enter?signup_subforem=1)[Create account](https://dev.to/enter?signup_subforem=1&state=new-user)
## DEV Community
0 Add reaction
0 Like 0 Unicorn 0 Exploding Head 0 Raised Hands 0 Fire
0 Jump to Comments 0 Save Boost
Copy link
Copied to Clipboard
[Share to X](https://twitter.com/intent/tweet?text=%22Most%20Evals%20Measure%20the%20Wrong%20Thing%22%20by%20Aamer%20Mihaysi%20%23DEVCommunity%20https%3A%2F%2Fdev.to%2Fo96a%2Fmost-evals-measure-the-wrong-thing-c76)[Share to LinkedIn](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fdev.to%2Fo96a%2Fmost-evals-measure-the-wrong-thing-c76&title=Most%20Evals%20Measure%20the%20Wrong%20Thing&summary=Reading%20through%20the%20awesome-evals%20list%20on%20GitHub%20yesterday%2C%20I%20realized%20something%20I%27ve%20been%20circling...&source=DEV%20Community)[Share to Facebook](https://www.facebook.com/sharer.php?u=https%3A%2F%2Fdev.to%2Fo96a%2Fmost-evals-measure-the-wrong-thing-c76)[Share to Mastodon](https://s2f.kytta.dev/?text=https%3A%2F%2Fdev.to%2Fo96a%2Fmost-evals-measure-the-wrong-thing-c76)
[Share Post via...](https://dev.to/o96a/most-evals-measure-the-wrong-thing-c76#)[Report Abuse](https://dev.to/report-abuse)
[](https://dev.to/o96a)
[Aamer Mihaysi](https://dev.to/o96a)
Posted on Jul 11
# Most Evals Measure the Wrong Thing
[#ai](https://dev.to/t/ai)[#opensource](https://dev.to/t/opensource)[#programming](https://dev.to/t/programming)
Reading through the awesome-evals list on GitHub yesterday, I realized something I've been circling for months: most of us are benchmarking wrong.
Not "wrong" as in bad numbers. Wrong as in measuring the wrong thing.
The list is a curated library of papers, tools, and benchmarks for evaluating AI agents — no fluff, no vendor pitches. What struck me is how many evals test whether a model _can_ do something in isolation, and how few test whether it _will_ do the right thing when the environment is messy, the instructions are ambiguous, and the tool call fails three times in a row.
That's the gap. Lab evals measure capability. Production evals measure reliability.
I've been running a small self-hosted agent stack for a few months now, and the bugs that actually bite me are never "the model couldn't answer this question." They're "the model tried to call a tool with a malformed argument because the previous step returned something unexpected." Or "the model got stuck in a loop because it didn'
URL Source: https://dev.to/o96a/most-evals-measure-the-wrong-thing-c76
Published Time: 2026-07-11T14:38:38Z
Markdown Content:
[Skip to content](https://dev.to/o96a/most-evals-measure-the-wrong-thing-c76#main-content)
[](https://dev.to/)
[Powered by Algolia](https://www.algolia.com/developers/?utm_source=devto&utm_medium=referral)
[Log in](https://dev.to/enter?signup_subforem=1)[Create account](https://dev.to/enter?signup_subforem=1&state=new-user)
## DEV Community
0 Add reaction
0 Like 0 Unicorn 0 Exploding Head 0 Raised Hands 0 Fire
0 Jump to Comments 0 Save Boost
Copy link
Copied to Clipboard
[Share to X](https://twitter.com/intent/tweet?text=%22Most%20Evals%20Measure%20the%20Wrong%20Thing%22%20by%20Aamer%20Mihaysi%20%23DEVCommunity%20https%3A%2F%2Fdev.to%2Fo96a%2Fmost-evals-measure-the-wrong-thing-c76)[Share to LinkedIn](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fdev.to%2Fo96a%2Fmost-evals-measure-the-wrong-thing-c76&title=Most%20Evals%20Measure%20the%20Wrong%20Thing&summary=Reading%20through%20the%20awesome-evals%20list%20on%20GitHub%20yesterday%2C%20I%20realized%20something%20I%27ve%20been%20circling...&source=DEV%20Community)[Share to Facebook](https://www.facebook.com/sharer.php?u=https%3A%2F%2Fdev.to%2Fo96a%2Fmost-evals-measure-the-wrong-thing-c76)[Share to Mastodon](https://s2f.kytta.dev/?text=https%3A%2F%2Fdev.to%2Fo96a%2Fmost-evals-measure-the-wrong-thing-c76)
[Share Post via...](https://dev.to/o96a/most-evals-measure-the-wrong-thing-c76#)[Report Abuse](https://dev.to/report-abuse)
[](https://dev.to/o96a)
[Aamer Mihaysi](https://dev.to/o96a)
Posted on Jul 11
# Most Evals Measure the Wrong Thing
[#ai](https://dev.to/t/ai)[#opensource](https://dev.to/t/opensource)[#programming](https://dev.to/t/programming)
Reading through the awesome-evals list on GitHub yesterday, I realized something I've been circling for months: most of us are benchmarking wrong.
Not "wrong" as in bad numbers. Wrong as in measuring the wrong thing.
The list is a curated library of papers, tools, and benchmarks for evaluating AI agents — no fluff, no vendor pitches. What struck me is how many evals test whether a model _can_ do something in isolation, and how few test whether it _will_ do the right thing when the environment is messy, the instructions are ambiguous, and the tool call fails three times in a row.
That's the gap. Lab evals measure capability. Production evals measure reliability.
I've been running a small self-hosted agent stack for a few months now, and the bugs that actually bite me are never "the model couldn't answer this question." They're "the model tried to call a tool with a malformed argument because the previous step returned something unexpected." Or "the model got stuck in a loop because it didn'
DeepCamp AI