Bryan Silverthorn, Director of AGI Autonomy at Amazon, told attendees at VB Transform 2026 on Tuesday that the enterprise AI agent industry has a reliability problem, not a capability problem. Cisco data shows 85% of enterprises are piloting AI agents, but only 5% have shipped them to production. That 80-point gap, Silverthorn argued, persists because teams measure the wrong things.
Four Dimensions of Reliability
Silverthorn, who joined Amazon through its acquisition of Adept AI and now leads multimodal agent training inside the company’s AGI lab, proposed breaking reliability into four distinct dimensions: consistency, robustness, predictability, and safety. He credited the framework to research from Princeton.
“It unpacks different factors that I see tangled together in almost every eval I’ve ever seen,” Silverthorn told VentureBeat.
The distinction matters because agents routinely pass internal evaluations and then collapse in production. Silverthorn described a customer that deployed an agent for software QA involving serial number extraction from screens. The agent worked correctly for two months before it began intermittently reading wrong numbers. The root cause: the underlying vision encoder behaved differently depending on where the serial number appeared on screen, and a software change imperceptible to humans triggered the failure.
The Measurement Gap
VentureBeat’s own proprietary research, presented before Silverthorn’s session, reinforced the pattern: half of surveyed companies shipped agents that passed internal evals but failed real customers. Enterprises overwhelmingly tracked uptime while ignoring accuracy. Most defaulted to the model makers’ own evaluations with little independent testing.
Silverthorn’s prescription was that teams need to identify their specific dimensions of variability and match measurement rigor to the stakes of the application. “The models have to be better. Obviously, we’re working hard on making the models better,” he said. But better models alone will not close the gap without better measurement practices.
Agents as Interns
Inside Amazon’s AGI lab, researchers call their agents “interns.” The term carries an operational philosophy: agents are powerful but occasionally clueless, capable of strong output and spectacular failure in the same workflow.
Managing them, Silverthorn argued, requires management skills rather than software skills: asking what could go wrong, adding backups and undo capabilities, and consciously deciding what risk is acceptable. “You can ask the intern, ‘Hey, what might you do wrong here? How might you mitigate your negative outcomes?’” he said. Amazon’s lab has embraced that trade-off, accepting agents occasionally running wrong experiments in exchange for research velocity.
The Production Path
Silverthorn was candid about limits. Self-improving AI remains “a loaded term.” Computer use is a core focus of his lab, with a commercial trucking customer already using browser automation to process warranty claims across fragmented systems, but he stressed that no future agent will rely on computer use alone. Production agents will combine browser automation with MCP, APIs, and other tool integrations for end-to-end workflows.
For enterprises stuck between pilot and production, the takeaway from the session was concrete: stop measuring whether an agent can do something once and start measuring whether it can do it correctly a thousand times in a row. The companies that move past the 5% production threshold will be the ones that treat agent management as an operational discipline, not a deployment checkbox.