Sequence parallelism
Also known as sequence parallel, SP
In plain English
Sequence parallelism splits a single long sequence across chips, so the tokens of one request are processed by several accelerators at once.
Technical definition
Sequence parallelism is a strategy that partitions the token dimension of a sequence across devices, dividing activation memory and attention work for very long inputs.
Engineering details
Tensor parallelism splits weights, while sequence parallelism splits the sequence itself: each chip holds a slice of the tokens and the associated activations and KV state. Attention then needs communication, since queries on one chip must meet keys and values on others, which ring style attention algorithms overlap with compute. In inference the closely related context parallel approach is what makes prefill of contexts with hundreds of thousands of tokens tractable.
Why it matters
Sequence style partitioning is the answer when one request, not the batch, is too large: a single million token prefill can exceed the memory and time budget of any one chip. It converts context length from a hard wall into a scaling dimension, at the price of interconnect traffic.
How to read it in InferenceX
InferenceX records the full parallelism plan of each recipe, and long context agentic scenarios are where sequence and context partitioning choices, together with interconnect quality, visibly separate systems with similar single chip specifications.
Source material
See the concept in real benchmarks
Agentic Benchmark for LLM Inference: Metrics and Methodology
How an agent benchmark replays long-context, multi-turn workloads to measure latency, throughput, cache behavior, and serving cost
Vera Rubin NVL72 vs GB200 NVL72? Inference TCO & Architecture Analysis
Rubin LUT Based Tensor Core, Feynman, Rack Scale, Perf Per MegaWatt, Perf Per Dollar, Software Improvements, Public Rubin Software, PyTorch, vLLM, OpenAI Triton