FlashAttention
Also known as flash attention kernel, fused attention
In plain English
FlashAttention computes exact attention in fast on-chip memory tiles, avoiding the huge intermediate matrix that made attention slow and memory hungry.
Technical definition
FlashAttention is an attention algorithm that tiles the computation through on-chip SRAM and rescales results incrementally, producing exact attention without materializing the full score matrix in HBM.
Engineering details
Naive attention writes a score matrix that grows with the square of sequence length to main memory and reads it back, so bandwidth rather than arithmetic sets its speed. FlashAttention fuses the whole computation into one kernel that streams blocks of keys and values through on-chip memory, using an online softmax to keep results exact. Successive versions and vendor implementations extend the idea to new hardware generations, head layouts, and inference specific decode paths.
Why it matters
This kernel family is what made long context practical, turning attention from the dominant cost of long sequences into one manageable component. It is also the canonical example of how a single well engineered kernel can shift performance across the entire industry.
How to read it in InferenceX
Every serving engine InferenceX benchmarks relies on fused attention kernels descended from this line, through libraries such as FlashInfer on NVIDIA hardware and AITER on AMD, and kernel improvements there routinely move published curves without any hardware change.
Source material
See the concept in real benchmarks
SGLang 0.5.6 on B200 DeepSeek R1 FP4: Up to 1.8x at Low Concurrency
Piecewise CUDA graphs for DeepSeek V3, a unified event loop, and JIT kernels push 8k/1k throughput from 508 to 907 tok/s/GPU on the same 16 GPU B200 pool
AMD MI355X Qwen3.5 397B-A17B Inference: Up to 19x Throughput per GPU in 3 Months on SGLang FP8
From v0.5.8 (Feb) → v0.5.10rc0 (Apr) → v0.5.12 (May), three AITER kernel landings on MI355X plus a TP=8 → TP=2/TP=4 retune push Qwen3.5 8k/1k peak from 1.3k to 6.4k tok/s/GPU and extend the curve out to 75 tok/s/user