Grouped-query attention
Also known as GQA, multi-query attention, MQA
In plain English
Grouped-query attention lets several query heads share one set of key and value heads, shrinking the KV cache without giving up much quality.
Technical definition
Grouped-query attention is an attention variant where query heads are divided into groups that each share a single key and value head, reducing KV cache size and bandwidth per token.
Engineering details
Standard multi-head attention stores keys and values for every head, so cache size scales with head count. Multi-query attention collapses all heads onto one KV pair, which is maximally cheap but can hurt quality. GQA sits between the two: a model might serve 64 query heads from 8 KV heads, cutting cache size eight fold. Because decode is dominated by reading the KV cache, the saving translates directly into faster token generation.
Why it matters
GQA became the default attention layout for dense open models because it attacks the memory side of decode where serving is actually bound. It also set the stage for more aggressive KV compression schemes such as multi-head latent attention.
How to read it in InferenceX
Attention layout is fixed by each model architecture, so GQA shows up in InferenceX through model level differences in KV bytes per token, which shape achievable concurrency and interactivity on identical hardware and engine versions.
Source material
See the concept in real benchmarks
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
Agentic Benchmark for LLM Inference: Metrics and Methodology
How an agent benchmark replays long-context, multi-turn workloads to measure latency, throughput, cache behavior, and serving cost