Software

HiCache

Also known as SGLang HiCache, hierarchical KV cache

In plain English

HiCache extends SGLang’s prefix cache beyond accelerator memory, retaining reusable attention state in host memory and optionally a storage backend.

Technical definition

HiCache is SGLang’s hierarchical KV-cache system, extending radix-based prefix reuse across accelerator memory, instance-local host memory, and an optional storage tier.

Engineering details

The hierarchy tracks where cached state resides and manages loading, prefetching, and write-back. A host-memory hit still needs a transfer before device computation can reuse the state. Host memory is private to an inference instance; cross-instance reuse requires a storage backend configured for the intended sharing scope. Transfer implementations and supported model state depend on the serving stack.

Why it matters

A larger retained prefix cache can reduce repeated prefill in long multi-turn workloads, but transfer latency, bandwidth, and eviction policy determine whether a hit actually improves service time. High hit rate alone is not proof of lower end-to-end latency.

How to read it in InferenceX

The TPU article describes SGLang HiCache host-DRAM offload through TPU-Sync for Muse Spark Open at high concurrency. That measured recipe does not imply that every HiCache backend or configuration supports the same accelerator and model combinations.

See the concept in real benchmarks

All articles

The Iron(wood) Throne: Google's TPU Stakes Its Claim on Inference

TPUv7 AgentX results across Muse Spark Open, GLM, Qwen3.5, and Kimi K3, with TorchTPU, megakernels, and llm-d disaggregation.

AgentX - InferenceXv3: Does the CUDA Moat Hold Up in Agentic Inferencing?

$3 Million USD dataset open sourced, 1 Mil+ Context Length, Multiturn, Sub Agents 95%+ KVCache HitRate, GB300 NVL72, MI355X, B200