AGNTCon + MCPCon Japan 2026

Teams are moving coding agents onto self-hosted open-weight models to control inference cost, and real sessions routinely span tens of turns and hundreds of thousands of tokens. But the only common signal for judging those models is the static accuracy benchmark, which says little about how a model behaves across a real, multi-turn session.

This session closes that gap with a trace-based evaluation method built on open technology: OpenTelemetry for capture and MLflow for analysis. Through a case study tracing real Claude Code sessions on a 120B-class open-weight model and a frontier model, it surfaces behaviors no leaderboard reports. For example, in the sessions we traced, the model's logged reasoning recorded a user's constraint, and the same turn violated it. An accuracy score sees only the wrong action; the trace shows the model had the rule in hand and did not follow it.

Attendees leave with a reusable observability architecture for their own agents, metrics that go beyond accuracy, and a clear view of what running a coding agent on open weights actually costs, in reliability and not only in dollars.