Tokenization
Also known as tokenizer, token, byte pair encoding, BPE
In plain English
Tokenization splits text into the sub word units a model actually reads and writes, and every performance and price number is quoted in them.
Technical definition
Tokenization is the conversion between text and the discrete token IDs a model processes, using a fixed vocabulary learned with methods such as byte pair encoding.
Engineering details
A tokenizer maps common words to single tokens and rarer strings to several, with English averaging very roughly four characters per token. Vocabularies differ between model families, so identical text can produce meaningfully different token counts across models. Everything downstream is denominated in tokens: context windows, KV cache size, throughput, latency per token, and price per million tokens all count these units, not characters or words.
Why it matters
Token efficiency is a hidden price lever, since a model that needs fewer tokens for the same content is cheaper at an identical per token rate. Comparing providers or benchmarks without normalizing for tokenizer differences quietly distorts cost and speed conclusions.
How to read it in InferenceX
InferenceX metrics are token denominated, and its AgentX traces replace original text with deterministic synthetic tokens while preserving per turn token counts, so replayed sessions stress serving systems with the same token arithmetic as the source workload.
Source material
See the concept in real benchmarks
Agentic Benchmark for LLM Inference: Metrics and Methodology
How an agent benchmark replays long-context, multi-turn workloads to measure latency, throughput, cache behavior, and serving cost
InferenceMAX: Open Source Inference Benchmarking
NVIDIA GB200 NVL72, AMD MI355X, Throughput Token per GPU, Latency Tok/s/user, Perf per Dollar, Cost per Million Tokens, Tokens per Provisioned Megawatt, DeepSeek R1 670B, GPTOSS 120B, Llama3 70B