AITER
Also known as AMD AITER, ROCm kernel library
In plain English
AITER is AMD’s tuned kernel library, the layer that decides how fast attention and matrix work actually run on Instinct chips.
Technical definition
AITER is AMD’s library of optimized kernels for Instinct accelerators, used by inference engines running on ROCm.
Engineering details
Engines express a strategy; kernels decide whether it is fast. AITER supplies tuned attention, matrix, and fused operations, and an engine dispatches to it in place of a generic path. That dispatch decision is itself tunable, and a kernel that wins on one shape can lose on another, so selection may depend on context length rather than being fixed.
Why it matters
A parallelism strategy is only real if the kernels can express it, which is why context parallelism and long-context sparse attention on AMD arrived as kernel work rather than engine work. Very large caches also expose failures short requests never reach, such as address arithmetic that overflows once a pool crosses a size boundary and silently addresses the wrong row.
How to read it in InferenceX
The library sits inside the container image a recipe pins, so an AITER improvement can move a curve with no change to the engine version or the hardware. Kernel-level gains measured on uniform shapes do not always survive agentic traces, where cache and scheduling variance can swamp them.
Source material
See the concept in real benchmarks
AMD MI355X Kimi K2.5 Inference: 7.7x Throughput, Up To 15x Interactivity in 25 Days on vLLM
vLLM PR #35850 Fixed AITER MLA Dispatch on MI355X CDNA4, Unlocking Kimi K2.5 Inference Performance at TP=8, Shipped in vLLM 0.18
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
MI355X DeepSeek-V4-Pro on SGLang: 110.5x Throughput per GPU in 26 Days
The amd/deepseek_v4 side branch shipped TileLang attention indexer, Triton sparse MLA, fused RoPE/Hadamard, FlyDSL MoE, and FP4 weights across 31 performance optimizations PRs — lifting first-light 20 tok/s/GPU at 2.4 tok/s/user into 2,256 tok/s/GPU at 9.4 tok/s/user on 8K/1K, with both throughput and interactivity climbing together