Total cost of ownership
In plain English
TCO covers the hardware purchase plus the cost of powering, cooling, networking, and operating it over time.
Technical definition
Total cost of ownership is an all-in estimate of the cost to provision and operate computing infrastructure over its useful life.
Engineering details
A GPU’s purchase price is only one input. TCO models can include host systems, networking, power delivery, cooling, facilities, financing, depreciation, maintenance, and expected utilization, then normalize the result to cost per GPU-hour.
Why it matters
Using TCO instead of list price makes cross-system economics more realistic, especially for rack-scale products whose networking and power infrastructure differ. The result remains a model and should be read with its assumptions.
How to read it in InferenceX
InferenceX combines SemiAnalysis AI Cloud TCO inputs with observed tok/s/GPU. This separates hourly system cost from the software and workload behavior that determines how many tokens that hour produces.
Source material
See the concept in real benchmarks
InferenceMAX: Open Source Inference Benchmarking
NVIDIA GB200 NVL72, AMD MI355X, Throughput Token per GPU, Latency Tok/s/user, Perf per Dollar, Cost per Million Tokens, Tokens per Provisioned Megawatt, DeepSeek R1 670B, GPTOSS 120B, Llama3 70B
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
GB200 NVL72 vs B200 on DeepSeek R1 670B: Up to 4.4x Throughput per GPU at 125 tok/s/user
DeepSeek R1 FP4 1k/1k. NVL72's 72-GPU NVLink scale-up fabric lets decode run wide EP up to EP=32, where B200's 8-GPU NVLink island caps out at EP=8 over RoCEv2