This reflection article is based on Sayash Kapoor's talk "Building and evaluating AI Agents — Sayash Kapoor, AI Snake Oil" at the AI Engineer conference.
There's a peculiar paradox in AI engineering: the first 80% of the work feels like magic, while the last 20% feels like alchemy. In the rush to productionize AI agents, we've witnessed a series of high-profile failures that reveal a fundamental truth: AI engineering is, at its heart, reliability engineering—a discipline few are prepared to embrace.
As a founding team member at Xamun AI, I witnessed this firsthand. We could reach 70-80% reliability with breathtaking speed—the prototype phase where demos dazzle and possibilities seem endless. But then came the moment of reckoning: transforming agents from impressive demonstrations into reliable production systems. Suddenly, each incremental improvement demanded exponentially more effort, time, and ingenuity. This transformation isn't just difficult; it's the crucible where most AI products perish.
DoNotPay's story reads like a modern fable about hubris. They promised to "automate the entire work of a lawyer," even offering a million dollars to any attorney who would argue before the Supreme Court with their AI whispering legal wisdom in their ear. The FTC's investigation revealed what many suspected: their claims were fiction, leading to hundreds of thousands in fines. This wasn't merely overpromising—it was digital snake oil packaged as revolution.
What makes this story particularly telling is how it mirrors a pattern we see repeatedly in AI development: the seductive belief that because something works once, it will work reliably at scale. Legal practice isn't just about knowing the law; it's about understanding context, nuance, and human judgment—elements that remain stubbornly resistant to automation.
Even established players stumbled into the reliability trap. LexisNexis, with decades of legal tech expertise, launched a product claiming to be "hallucination-free" in generating legal reports. Stanford researchers found a different reality: hallucinations in up to a third of cases, some reversing the very meaning of legal texts.
This failure is particularly instructive because it reveals how even sophisticated companies can be seduced by the allure of certainty. In law, where a single word can alter destinies, these weren't just technical glitches—they were ethical failures.
Sakana AI's journey from promise to reality follows a familiar arc. They claimed to automate scientific research itself, only to have Princeton researchers discover their system could reproduce less than 40% of papers—even when given the original code and data. Their CUDA kernel optimization claims were even more revealing: they purported to exceed the theoretical hardware limits by 30 times, a claim that turned out to be reward function hacking rather than genuine optimization.
What's fascinating here is how the pressure to demonstrate progress led to what can only be called technological theater. The system wasn't optimizing; it was gaming its own metrics—a digital sleight of hand that reveals the profound difference between appearing to work and actually working.
Perhaps most sobering is the story of Devin, which secured $175 million at a $2 billion valuation based on benchmark brilliance. In real-world applications, it succeeded in only 3 out of 20 tasks. This isn't just a gap between promise and performance—it's a chasm that calls into question our entire approach to evaluating AI systems.
The Devin story crystallizes a fundamental truth about AI development: we've created systems that excel at taking tests but struggle with reality. It's as if we've built a brilliant scholar who can ace exams but can't cross the street safely.
Consider this paradox: the better an AI agent becomes, the harder it is to evaluate. Unlike simple models that process input-output pairs, agents inhabit dynamic environments where every action spawns consequences. It's like trying to judge a chess player's skill not by their moves, but by the ripples those moves create across multiple simultaneous games.
At Xamun AI, we discovered that evaluation costs didn't just increase—they exploded. A simple task could trigger recursive agent calls, each spawning their own evaluative universe. We weren't just testing functionality; we were testing entire ecosystems of interaction. The cost ceiling we expected? It didn't exist.
The challenge deepens when you realize that agents are inherently specialized. A coding agent inhabits a different universe than a web agent. Standard benchmarks become meaningless—like evaluating a dolphin's intelligence using tests designed for chimpanzees.
Static benchmarks have become the academic equivalent of Instagram filters—they make everything look better than it really is. Princeton's agent leaderboard exposed this beautifully: Claude 3.5 performed comparably to OpenAI's o1 models while costing a tenth as much ($57 vs $664). This revelation isn't just about cost—it's about how we've been measuring success entirely wrong.
The Jevons paradox haunts AI development: as costs plummet, usage skyrockets. We saw this at Xamun AI—cheaper API calls led to more ambitious agent architectures, ultimately increasing overall expenditure. It's the technological equivalent of building wider highways to reduce traffic, only to discover more cars appearing.
Most troublingly, benchmarks have become venture capital theater. Companies like Cosine and Cognition raised fortunes on benchmark brilliance, creating an ecosystem that rewards the appearance of progress over genuine reliability. We've built an industry that optimizes for tests rather than reality.
Here lies the most profound misunderstanding in AI engineering: the belief that capability equals reliability. At Xamun AI, we could quickly achieve 80% reliability—agents that dazzled in demos and impressed in presentations. But moving from 80% to 99.999% reliability? That journey consumed more resources than the entire initial development.
Think of it this way: capability is getting the right answer sometimes; reliability is getting it right always. In the real world, sometimes isn't good enough. If your AI assistant orders your food correctly only 80% of the time, those two failures out of ten aren't just errors—they're relationship-ending catastrophes.
We discovered that even our verification systems were flawed. Like HumanEval and MBPP's false positives, our unit tests sometimes passed incorrect code. It was verification theater—tests that created the illusion of reliability while masking underlying fragility.
The brutal truth? The journey from capable to reliable is where most AI products die. It's not a linear progression—it's an exponential climb that demands resources, patience, and a fundamental shift in how we think about success.
The solution demands more than technical fixes—it requires philosophical reorientation. We must shift from viewing AI engineering as a mere software problem to recognizing it as a reliability engineering discipline.
Consider the ENIAC computer of 1946. Its 17,000 vacuum tubes failed so frequently that the machine was unusable half the time. The engineers didn't abandon the project; they spent two years obsessively improving reliability until it worked well enough to be useful. That's our challenge today: turning brilliant but fragile systems into dependable tools.
The cost of quality in AI isn't measured merely in dollars or development hours. It's measured in the willingness to confront uncomfortable truths: that our systems are more fragile than we admit, that benchmarks often obscure rather than illuminate, and that the last mile to reliability demands exponentially more effort than the journey to capability.
When potential investors ask about our moat at Xamun AI, the answer isn't found in algorithms or datasets. It's found in the countless hours spent transforming promising prototypes into production-ready systems—work that's neither glamorous nor easily quantifiable, but absolutely essential.
The pioneers of computing transformed unreliable vacuum tubes into the foundation of the digital age. Today's AI engineers face a similar challenge with stochastic systems. The question isn't whether we can build impressive demos—we already can. The question is whether we're willing to pay the true cost of quality: the patient, meticulous work of engineering reliability into inherently uncertain systems.
As we stand at this crossroads, the path forward demands a philosophical shift: we must stop thinking like software engineers and start thinking like reliability engineers. This isn't just a change in methodology—it's a change in mindset, values, and ultimately, in how we define success.
Consider this fundamental question: What makes an AI system truly successful? Is it the dazzling demo that secures funding? The impressive benchmark scores that make headlines? Or is it the quiet reliability of a system that works, day after day, in the real world?
The AI industry is at a critical juncture. We can continue down the path of performance theater—optimizing for metrics that impress but don't deliver—or we can embrace the harder truth: that true success in AI engineering is measured not by occasional brilliance, but by consistent reliability.
Every shortcut we take in the pursuit of quick wins extracts a hidden tax. When we optimize for benchmarks instead of real-world performance, we're not just cutting corners—we're building on sand. When we prioritize capability over reliability, we're not just taking risks—we're creating future failures.
The most insidious aspect of these shortcuts is how they compound. Each compromise in quality creates technical debt that accrues interest. Each unreliable system deployed erodes trust in AI as a whole. We're not just building individual products; we're shaping an entire industry's trajectory.
Reliability isn't just a technical challenge—it's an ethical one. When AI systems fail, they don't just disappoint; they can harm. A legal AI that hallucinates isn't just inaccurate; it could lead to miscarriages of justice. A scientific AI that games its metrics isn't just deceptive; it could misdirect crucial research.
The cost of quality, then, isn't optional. It's the price of ethical AI development. It's the investment required to ensure that our technological advances serve rather than subvert human needs.
The true competitive advantage in AI engineering doesn't come from being first to market or having the flashiest technology. It comes from the willingness to play the long game—to invest in reliability even when it's unglamorous, to prioritize quality even when it's costly.
At Xamun AI, we learned that our real differentiator wasn't our initial capabilities, but our commitment to the grueling work of making those capabilities reliable. That commitment—to treating AI engineering as reliability engineering—became our most valuable asset.
The journey from capability to reliability is where most AI products die, but it's also where the truly transformative ones are born. The cost of quality is high, but the cost of failure—of releasing unreliable systems into critical domains—is immeasurably higher.
As we shape the future of AI, let's remember: we're not just building tools; we're building trust. We're not just developing algorithms; we're developing relationships between humans and intelligent systems. The cost of quality isn't a burden to be minimized—it's an investment in that trust, in those relationships, and ultimately, in the responsible development of AI that serves humanity rather than disappoints it.
The pioneers who transformed computing from an unreliable curiosity into the bedrock of modern society understood this. As today's AI engineers, we must embrace the same commitment to reliability, the same willingness to do the hard, unglamorous work that transforms impressive demonstrations into dependable tools.
In the end, the companies that survive and thrive won't be those with the most dazzling demos or the highest benchmark scores. They'll be the ones who understood that AI engineering is, fundamentally, reliability engineering—and were willing to pay the true cost of quality to make that vision a reality.
The choice before us is clear: we can continue to mistake capability for reliability, or we can embrace the harder path of building systems that truly work. The future of AI depends on engineers who choose the latter—who see constraints not as obstacles, but as the very framework within which trust is built and maintained.
That's the standard we must hold ourselves to. That's the cost we must be willing to pay. In doing so, we don't just build better AI systems; we build the foundation for a future where artificial intelligence serves humanity with the reliability and trustworthiness it deserves.
This article was originally published as a LinkedIn article by Xamun CEO Arup Maity. To learn more and stay updated with his insights, connect and follow him on LinkedIn.
