AGNTCon + MCPCon Japan 2026

The AI industry has moved so fast that we are still evaluating probabilistic software using the same deterministic metrics we applied to traditional code. As a Product Analyst working on AI agents at PagerDuty, I saw a critical need for a new observability standard, one that moves beyond clicks to measure true reasoning and reliability. To address this, I’ve developed and open-sourced a specialized framework designed to help teams decide, with data, when to hire, train, or fire an AI agent. In this session, I will walk through the H.I.R.E. Framework methodology and share the technical architecture of an evaluation pipeline that turns qualitative conversational data into structured, actionable product insights. I will share the open-source repository containing these metric definitions and templates, providing resources for the community to move past agent-washing and toward building verifiable, trustworthy agentic systems.

AI systems are increasingly evaluated using benchmarks designed for individual components: Word Error Rate (WER) for speech recognition, MOS for speech synthesis, latency for infrastructure, and task success for agents. Yet users experience none of these components in isolation when using voice agents, they experience conversations.

Voice agents expose the limitations of traditional evaluation more clearly than any other AI system. An agent can achieve state-of-the-art WER, MOS, and latency metrics while still delivering a frustrating user experience.

This talk explores why established metrics are becoming insufficient for real-time AI applications and introduces a framework for evaluating voice agents holistically. We will examine concepts such as semantic understanding versus transcription accuracy, latency distributions versus averages, interruption handling, recovery from errors, and conversation-level success metrics.

Through real-world examples from production voice systems, attendees will learn how to move beyond component benchmarks and begin measuring what ultimately matters: whether an AI system successfully helps a user achieve their goal.