GEMM
Also known as general matrix multiply, matrix multiplication kernel
In plain English
GEMM is the general matrix multiply operation, the single computation that consumes most of the arithmetic in training and serving neural networks.
Technical definition
GEMM is the general matrix to matrix multiply routine, the core primitive that linear layers, attention projections, and expert computations in neural networks reduce to.
Engineering details
Transformers are mostly stacks of linear transformations, so serving a model means executing enormous numbers of matrix multiplies. Tensor cores exist specifically to accelerate them, and peak TFLOP/s specifications are quoted for these dense operations. Shape determines efficiency: prefill produces large square-ish multiplies that saturate compute, while decode produces skinny ones that are bandwidth bound. MoE adds grouped GEMMs, where many small expert multiplies are batched into one efficient launch.
Why it matters
GEMM efficiency is the substrate of every performance claim in the industry. The gap between delivered and peak GEMM throughput at real serving shapes, especially the thin matrices of decode, explains much of why spec sheet ratios fail to predict benchmark rankings.
How to read it in InferenceX
Behind every InferenceX curve sits a stack of GEMM kernels from libraries such as CUTLASS, hipBLASLt, and Triton generated code, and quantized recipes ultimately stand on how well each vendor executes low precision GEMMs at the shapes its scheduler produces.
Source material
See the concept in real benchmarks
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
AMD MI355X GLM-5 Inference: Up to 40% Cheaper per Million Tokens than B200 on SGLang FP8
14 weeks after GLM-5 launched, AMD landed both MTP and non-MTP SGLang FP8 recipes on MI355X — fused MLA + FP8 KV cache via TileLang flips the single-node FP8 cost curve in AMD favor across most of the performance Pareto