Inference engine
Also known as serving engine, LLM serving framework
In plain English
The inference engine is the traffic controller behind an AI service: it keeps incoming requests moving and makes sure the GPUs do the right work at the right time.
Technical definition
An inference engine is the software runtime that turns model weights and incoming requests into generated outputs on accelerators.
Engineering details
The engine owns request scheduling, batching, KV-cache allocation, distributed execution, kernel selection, and token sampling. vLLM, SGLang, and TensorRT-LLM can run the same model on the same GPU yet produce different curves because their schedulers, kernels, and distributed strategies differ.
Why it matters
Engine version and configuration can matter as much as GPU choice. A scheduler change, a fused attention kernel, or a corrected model-specific path can move throughput several-fold without any hardware change.
How to read it in InferenceX
InferenceX records the engine and container image as part of each reproducible recipe. Historical views are therefore useful for separating software gains from silicon gains.
Source material
See the concept in real benchmarks
SGLang 0.5.6 on B200 DeepSeek R1 FP4: Up to 1.8x at Low Concurrency
Piecewise CUDA graphs for DeepSeek V3, a unified event loop, and JIT kernels push 8k/1k throughput from 508 to 907 tok/s/GPU on the same 16 GPU B200 pool
AMD MI355X Kimi K2.5 Inference: 7.7x Throughput, Up To 15x Interactivity in 25 Days on vLLM
vLLM PR #35850 Fixed AITER MLA Dispatch on MI355X CDNA4, Unlocking Kimi K2.5 Inference Performance at TP=8, Shipped in vLLM 0.18
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM