AGNTCon + MCPCon Japan 2026

AI systems are increasingly evaluated using benchmarks designed for individual components: Word Error Rate (WER) for speech recognition, MOS for speech synthesis, latency for infrastructure, and task success for agents. Yet users experience none of these components in isolation when using voice agents, they experience conversations.

Voice agents expose the limitations of traditional evaluation more clearly than any other AI system. An agent can achieve state-of-the-art WER, MOS, and latency metrics while still delivering a frustrating user experience.

This talk explores why established metrics are becoming insufficient for real-time AI applications and introduces a framework for evaluating voice agents holistically. We will examine concepts such as semantic understanding versus transcription accuracy, latency distributions versus averages, interruption handling, recovery from errors, and conversation-level success metrics.

Through real-world examples from production voice systems, attendees will learn how to move beyond component benchmarks and begin measuring what ultimately matters: whether an AI system successfully helps a user achieve their goal.