Total cost of ownership
In plain English
TCO covers the hardware purchase plus the cost of powering, cooling, networking, and operating it over time.
Technical definition
Total cost of ownership is an all-in estimate of the cost to provision and operate computing infrastructure over its useful life.
Engineering details
A chip’s purchase price is only one input. TCO models can include host systems, networking, power delivery, cooling, facilities, financing, depreciation, maintenance, and expected utilization, then normalize the result to cost per chip-hour.
Why it matters
Using TCO instead of list price makes cross-system economics more realistic, especially for rack-scale products whose networking and power infrastructure differ. The result remains a model and should be read with its assumptions.
How to read it in InferenceX
InferenceX combines SemiAnalysis AI Cloud TCO inputs with observed tok/s/chip. The Rubin article distinguishes owning at large hyperscaler volume, including amortized hardware, networking, facilities, power, and capital costs, from the customer price of a three-year cloud reservation. These are alternative cost bases, not costs to add together; rental pricing also reflects the provider’s commercial terms.
Source material
See the concept in real benchmarks
InferenceMAX: Open Source Inference Benchmarking
NVIDIA GB200 NVL72, AMD MI355X, Throughput Token per GPU, Latency Tok/s/user, Perf per Dollar, Cost per Million Tokens, Tokens per Provisioned Megawatt, DeepSeek R1 670B, GPTOSS 120B, Llama3 70B
InferenceX v2: NVIDIA Blackwell Vs AMD vs Hopper - Formerly InferenceMAX
GB300 NVL72, MI355X, B200, H100, Disaggregated Serving, Wide Expert Parallelism, Large Mixture of Experts, SGLang, vLLM, TRTLLM
GB200 NVL72 vs B200 on DeepSeek R1 670B: Up to 4.4x Throughput per GPU at 125 tok/s/user
DeepSeek R1 FP4 1k/1k. NVL72's 72-GPU NVLink scale-up fabric lets decode run wide EP up to EP=32, where B200's 8-GPU NVLink island caps out at EP=8 over RoCEv2
Vera Rubin NVL72 vs GB200 NVL72? Inference TCO & Architecture Analysis
Rubin LUT Based Tensor Core, Feynman, Rack Scale, Perf Per MegaWatt, Perf Per Dollar, Software Improvements, Public Rubin Software, PyTorch, vLLM, OpenAI Triton
OpenAI Jalapeño: Better Than Nvidia Blackwell
OpenAI’s self-designed ASIC compared with Rubin, Jalapeño’s TCO, throughput per MW, and spicy deets
Rubin NVL72 Agentic Inference: 67x better Performance per Dollar
Jensen Sandbagging Performance Again, 2x more Annual Profit Per GigaWatt, The More you Buy, The More you Earn, AgentX, InferenceX, Extreme Co-Design