CUDA
Also known as NVIDIA CUDA
In plain English
CUDA is NVIDIA’s software toolbox for making programs run on its chips.
Technical definition
CUDA is NVIDIA’s chip computing platform, programming model, compiler toolchain, and library ecosystem.
Engineering details
LLM engines use CUDA kernels and libraries for matrix multiplication, attention, collectives, graph capture, memory management, and custom fused operations. Container, driver, CUDA, and chip architecture versions must be compatible.
Why it matters
Serving performance depends on the software above the silicon. New kernels, CUDA Graph usage, compiler specialization, and library releases can move the benchmark curve without changing the chip.
How to read it in InferenceX
InferenceX recipes pin container images and therefore a concrete CUDA stack. Historical comparisons can isolate the effect of an engine image bump on otherwise identical hardware and configuration.
Source material
See the concept in real benchmarks
SGLang 0.5.6 on B200 DeepSeek R1 FP4: Up to 1.8x at Low Concurrency
Piecewise CUDA graphs for DeepSeek V3, a unified event loop, and JIT kernels push 8k/1k throughput from 508 to 907 tok/s/GPU on the same 16 GPU B200 pool
B200 NVFP4 vs H200 FP8 on GLM-5: Up to 3.65x Better Performance per Dollar with SGLang MTP
Both SKUs run SGLang EAGLE MTP; the Blackwell generation lifts perf/$ by ~1.2x at the peak and the NVIDIA GLM-5-NVFP4 checkpoint on FlashInfer TRT-LLM sparse MLA stacks another ~2.4–3.0x on 8K/1K
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
Ultra-High Interactivity on NVIDIA GPUs? TileRT on InferenceX
Can TileRT software on NVIDIA GPUs compete with Cerebras, Groq LPU, and SambaNova? Batch size 1, disaggregated engine, high-throughput prefill engine, high-interactivity decode engine