CUTLASS
Also known as CUDA templates for linear algebra, CuTe
In plain English
CUTLASS is an NVIDIA template library that provides the building blocks for writing matrix multiply kernels that approach peak hardware speed.
Technical definition
CUTLASS is an open source NVIDIA library of composable C++ templates for building GEMM and related kernels that target tensor cores across GPU generations.
Engineering details
Peak matrix multiply performance demands precise choreography of tensor core instructions, shared memory movement, and asynchronous pipelines, and it changes with every architecture. CUTLASS packages that choreography as composable pieces, with its CuTe layer describing data layouts, so kernel authors assemble near peak GEMMs and fuse epilogues such as bias, activation, or quantization scaling instead of starting from scratch. Much of the high performance kernel work in serving engines builds on it directly.
Why it matters
CUTLASS is where NVIDIA teaches the ecosystem to use each new tensor core generation, including the FP4 and FP8 paths on Blackwell. The speed at which its patterns propagate into engines is a real component of how quickly new silicon reaches its advertised performance.
How to read it in InferenceX
The GEMM and attention kernels behind InferenceX results on NVIDIA hardware lean heavily on CUTLASS derived code, and version bumps of these libraries inside engine images are a recurring source of day over day curve movement the platform tracks.
Source material
See the concept in real benchmarks
SGLang 0.5.6 on B200 DeepSeek R1 FP4: Up to 1.8x at Low Concurrency
Piecewise CUDA graphs for DeepSeek V3, a unified event loop, and JIT kernels push 8k/1k throughput from 508 to 907 tok/s/GPU on the same 16 GPU B200 pool
GB300 NVL72 vs GB200 NVL72 Inference Performance & Perf per Dollar - on DeepSeek-V4-Pro 1.6T: Up to 2.83x Throughput
DSv4-Pro FP4 8K/1K, Dynamo+vLLM, disaggregated on both racks. GB300's 50% extra HBM (288 vs 192 GB/GPU) unlocks a wider prefill+decode recipe GB200 can't fit — lifting middle-of-curve perf/$ by 2.31x despite a 20% per-GPU TCO premium.