When looking at ATOM, AMD has better performance per dollar than GB300 NVL72 SGLang and even TRTLLM for some parts of the range of p90 E2E Normalized Interactivity. Great work to the AMD team on these results.

The usual two requests apply to both vendors. We look forward to AMD porting these optimizations over to SGLang, and we are also looking forward to NVIDIA quickly optimizing GB300 NVL72 in the coming weeks.

What E2E Normalized Interactivity measures

This result is plotted against an experimental metric worth understanding before drawing conclusions from it. E2E Normalized Interactivity is meant to evaluate how fast a user experiences responsiveness when considering both time to first token and tokens per second. It is defined as output sequence length divided by end to end latency.

Substituting the fact that end to end latency equals TTFT plus output length times time per output token, and noting that in reality only output length minus one tokens are decoded, gives the following.

This is effectively interactivity, the reciprocal-TPOT portion, plus an additional penalty proportional to TTFT.

Where the metric misleads

Please note that this metric is experimental and is not perfect. For instance, it heavily penalizes high TTFT and does not capture all the nuances of certain optimizations such as prefill-decode disaggregation. All submissions for AgentX v1.0 optimize for both regular interactivity and TTFT separately, and we will continue working on new north star metrics that reflect all nuances of modern agentic inference.

That caveat cuts in both directions here. A configuration that trades first-token latency for streaming speed is penalized on this axis whether or not that trade suits the product, so a win measured only against E2E Normalized Interactivity should be checked against the separate interactivity and TTFT curves before it becomes a procurement argument.

These results are one slice of AgentX 1.0. The full analysis, the replay methodology, and the 70+ upstream PRs the benchmark drove are in AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?. Every point is explorable on the free dashboard.