TileRT
Also known as TileRT engine
In plain English
TileRT is an inference runtime built for very fast single-user generation, compiling a model into one resident program instead of many separate kernel launches.
Technical definition
TileRT is an inference engine that targets ultra-low-latency serving by abolishing the individual kernel as the unit of execution.
Engineering details
A conventional runtime dispatches a sequence of kernels for every decode step, and at very small batch sizes the launch and scheduling overhead between them dominates the arithmetic. A persistent engine kernel keeps the work resident on the accelerator instead, which is what makes the far-right end of the interactivity axis reachable at all.
Why it matters
The high-interactivity corner of the frontier is a different engineering problem from the high-throughput corner, and an engine tuned for one rarely wins the other. Recipes that reach hundreds of tokens per second per user matter for latency-critical products even when their aggregate throughput per chip is unremarkable.
How to read it in InferenceX
InferenceX reports TileRT as its own framework label and deliberately retains it in best-per-SKU views, because a curve that only survives where it dominates on throughput would drop the operating points TileRT exists to serve. Compare it at matched interactivity rather than on peak throughput alone.
Source material
See the concept in real benchmarks
Ultra-High Interactivity on NVIDIA GPUs? TileRT on InferenceX
Can TileRT software on NVIDIA GPUs compete with Cerebras, Groq LPU, and SambaNova? Batch size 1, disaggregated engine, high-throughput prefill engine, high-interactivity decode engine
AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?
$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200