FlashInfer
Also known as attention kernel library
In plain English
FlashInfer is a library of attention kernels that serving engines call instead of writing their own attention implementations.
Technical definition
FlashInfer is an open-source library of attention kernels and backends used by inference engines for prefill, decode, and speculative verification.
Engineering details
Attention is where most serving-specific complexity lives: paged caches, variable sequence lengths, grouped query heads, sparsity patterns, and verification of drafted tokens all reshape the kernel. A shared library lets several engines reuse one tuned implementation, and engines select a backend per shape and per hardware target.
Why it matters
Because backends are selected rather than fixed, kernel availability becomes a portability question. A feature implemented only for one vendor’s backend leaves the alternative running a generic path, which on long context is not a small compromise but the wrong kernel for the shape.
How to read it in InferenceX
The backend in use is part of the recipe, and a change to it can move a curve without any hardware or engine version change. Support for checkpointing recurrent state in these kernels is what allowed hybrid models to participate in prefix reuse at all.
Source material
See the concept in real benchmarks
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200
SGLang 0.5.6 on B200 DeepSeek R1 FP4: Up to 1.8x at Low Concurrency
Piecewise CUDA graphs for DeepSeek V3, a unified event loop, and JIT kernels push 8k/1k throughput from 508 to 907 tok/s/GPU on the same 16 GPU B200 pool
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM