AGNTCon + MCPCon Japan 2026

The AI industry has moved so fast that we are still evaluating probabilistic software using the same deterministic metrics we applied to traditional code. As a Product Analyst working on AI agents at PagerDuty, I saw a critical need for a new observability standard, one that moves beyond clicks to measure true reasoning and reliability. To address this, I’ve developed and open-sourced a specialized framework designed to help teams decide, with data, when to hire, train, or fire an AI agent. In this session, I will walk through the H.I.R.E. Framework methodology and share the technical architecture of an evaluation pipeline that turns qualitative conversational data into structured, actionable product insights. I will share the open-source repository containing these metric definitions and templates, providing resources for the community to move past agent-washing and toward building verifiable, trustworthy agentic systems.