AGNTCon + MCPCon Japan 2026

The hardest part of agentic engineering isn't writing the code; It’s proving that a prompt tweak or a new tool schema didn't completely break your agent's routing logic. Traditional unit testing falls short when dealing with non-deterministic agent trajectories.

This session looks at how to build an automated local evaluation and regression testing pipeline for ADK workflows. We will walk through how to programmatically mock tool responses, simulate user edge cases, and run parallel assertions against agent trajectories using open-source evaluation frameworks. Attendees will learn how to catch infinite loops, detect tool-calling degradation, and establish a baseline scoring rubric for agent accuracy before code ever hits a production branch.

___________________________
Presentation Language: English

The AI industry has moved so fast that we are still evaluating probabilistic software using the same deterministic metrics we applied to traditional code. As a Product Analyst working on AI agents at PagerDuty, I saw a critical need for a new observability standard, one that moves beyond clicks to measure true reasoning and reliability. To address this, I’ve developed and open-sourced a specialized framework designed to help teams decide, with data, when to hire, train, or fire an AI agent. In this session, I will walk through the H.I.R.E. Framework methodology and share the technical architecture of an evaluation pipeline that turns qualitative conversational data into structured, actionable product insights. I will share the open-source repository containing these metric definitions and templates, providing resources for the community to move past agent-washing and toward building verifiable, trustworthy agentic systems.

___________________________
Presentation Language: English

Captioning will be available for attendees in 50+ languages through Wordly. See instructions in each room to utilize captioning.

Teams are moving coding agents onto self-hosted open-weight models to control inference cost, and real sessions routinely span tens of turns and hundreds of thousands of tokens. But the only common signal for judging those models is the static accuracy benchmark, which says little about how a model behaves across a real, multi-turn session.

This session closes that gap with a trace-based evaluation method built on open technology: OpenTelemetry for capture and MLflow for analysis. Through a case study tracing real Claude Code sessions on a 120B-class open-weight model and a frontier model, it surfaces behaviors no leaderboard reports. For example, in the sessions we traced, the model's logged reasoning recorded a user's constraint, and the same turn violated it. An accuracy score sees only the wrong action; the trace shows the model had the rule in hand and did not follow it.

Attendees leave with a reusable observability architecture for their own agents, metrics that go beyond accuracy, and a clear view of what running a coding agent on open weights actually costs, in reliability and not only in dollars.

___________________________
Presentation Language: English

Captioning will be available for attendees in 50+ languages through Wordly. See instructions in each room to utilize captioning.