INT8
Also known as 8-bit integer quantization, W8A8
In plain English
INT8 stores numbers as 8 bit integers with a scale factor, halving memory versus 16 bit formats and doubling math rates on supporting hardware.
Technical definition
INT8 is an 8 bit integer numerical format used with per tensor or per channel scale factors to represent model weights and activations in quantized inference.
Engineering details
Integer quantization maps floating point values onto 256 evenly spaced levels via a scale, and sometimes a zero point. Uniform spacing handles outliers poorly, so techniques such as SmoothQuant migrate activation outliers into weights before quantizing both sides, in the W8A8 pattern. On older accelerator generations INT8 was the primary fast path below 16 bits, while newer chips add FP8, whose exponent gives it a wider dynamic range at the same bit width.
Why it matters
INT8 defined the first mainstream wave of LLM quantization and remains relevant on hardware without floating point 8 bit support. The INT8 versus FP8 contrast also illustrates the core quantization tradeoff between uniform precision and dynamic range.
How to read it in InferenceX
InferenceX labels every result with its precision, and its comparison families exist because format changes moved curves so much. Modern recipes on Blackwell and MI350 class hardware favor FP8 and FP4 paths, with integer formats appearing in specific weight quantized configurations.
Source material
See the concept in real benchmarks
B200 NVFP4 vs H200 INT4 on Kimi K2.5/K2.6: Up to 2.95x Better Performance per Dollar
On vLLM 8K/1K the NVFP4 path on B200 is 2.71x–2.95x cheaper per million tokens than H200 INT4 across the entire 30–90 tok/s/user serving band, and 2.45x–2.74x cheaper than B200 INT4 on the same silicon. Both factors decompose cleanly into B200's HBM bandwidth, HBM capacity, and NVFP4 tensor cores
InferenceMAX: Open Source Inference Benchmarking
NVIDIA GB200 NVL72, AMD MI355X, Throughput Token per GPU, Latency Tok/s/user, Perf per Dollar, Cost per Million Tokens, Tokens per Provisioned Megawatt, DeepSeek R1 670B, GPTOSS 120B, Llama3 70B