Tensor, pipeline, and data parallelism compared
| Strategy | What is distributed | Typical reason to use it |
|---|---|---|
| Tensor parallelism (TP) | Pieces of layer tensors; GPUs cooperate on the same tokens | Fit larger layers and accelerate their computation |
| Pipeline parallelism (PP) | Groups of layers; activations move between stages | Fit the model across devices or nodes |
| Data parallel serving (DP) | Complete model replicas serving different requests | Increase aggregate request capacity |
These strategies can be combined. One replica might occupy four GPUs, and a router might distribute requests across two such replicas. The basic definitions are documented in TensorRT-LLM’s parallelism guide. The examples below assume a dense decoder-only model unless stated otherwise.
How tensor parallelism works
With TP, a large matrix operation is partitioned across GPUs. For example, different GPUs can calculate different output features of a linear layer. Another operation may partition the input features, producing partial sums that must be combined. Each GPU owns part of the computation for the same batch, rather than receiving an independent set of users. Hugging Face: tensor parallelism.
An all-reduce combines values across participating processes and makes the result available to each. An all-gather assembles their separate pieces; a reduce-scatter combines values and distributes pieces of the result. The actual collective depends on the tensor layout and engine, so “TP communicates at every layer” does not mean every implementation uses one identical operation. NVIDIA NCCL: collective operations.
Frequent exchanges make interconnect latency and bandwidth important. Fast GPU links are especially valuable when the work between synchronization points is small. Check the actual device topology: a server advertised as having eight GPUs does not establish that every pair has the same communication path.
How pipeline parallelism works and why bubbles appear
PP assigns consecutive model layers to stages. Stage one processes an input and passes its activations to stage two, which continues the forward pass. Different microbatches can occupy different stages simultaneously. A stage waiting for work creates a pipeline bubble; smaller batches and unbalanced stage execution can leave more devices idle. NVIDIA: inference optimization.
Consider an illustrative four-stage forward pipeline with equal 1 ms stages and zero communication cost. One microbatch takes 4 ms to reach the output. Four independent microbatches finish in 7 ms: the first needs four slots, and each remaining microbatch exits one slot later. That is 16 occupied device-slots out of 28 available, or about 57 percent utilization. This is scheduling arithmetic, not a GPU benchmark.
For ordinary autoregressive generation, the next token of one request depends on the previous token’s result. Splitting its layers into four stages does not turn those sequential dependencies into four independent requests. Other requests can fill otherwise idle stages, but a quiet interactive endpoint may have little work to overlap. Test both isolated-request latency and loaded throughput; a useful pipeline can improve one without improving the other.
Example layouts for four and eight GPUs
| Available GPUs | Layout | What the placement means |
|---|---|---|
| 4 on one node | TP=4, PP=1 | All four GPUs cooperate within the layers of one replica |
| 4 on one node | TP=1, PP=4 | One replica has four layer stages, one per GPU |
| 4 on one node | Two replicas, each TP=2, PP=1 | Each GPU pair holds a complete distributed replica |
| 8 across two 4-GPU nodes | TP=4, PP=2 | Each node holds one pipeline stage, tensor-sharded across its four GPUs |
For the final row, a hypothetical 64-layer model could put layers 1–32 on the first node and 33–64 on the second. Each stage uses four-way TP. This describes one eight-GPU replica: TP × PP = 8. Running two independent copies would require sixteen GPUs. Equal layer counts are a starting sketch; embeddings, output heads, and unequal layer costs can change the best stage boundary.
vLLM recommends starting with TP within a node and combining it with PP across nodes when the model needs more capacity. Its guide also describes PP as an option for uneven splits or GPUs without NVLink. Treat these as candidate layouts to benchmark on your hardware. vLLM: parallelism and scaling.
Why dividing model memory by GPU count is insufficient
Suppose a hypothetical checkpoint contains exactly 70 billion BF16 parameters. Its raw weight storage is 140 GB in decimal units. Dividing by four gives an idealized 35 GB share. That arithmetic is useful for rejecting impossible plans, but it is not a prediction that each GPU will consume 35 GB.
- Weights need a real partition plan. Some tensors can remain replicated; embeddings and output heads may create stage imbalance.
- KV cache follows the attention implementation. It may be sharded by heads or layers, or partly replicated. Do not assume doubling TP halves cache memory.
- Execution needs extra space. Activations, communication buffers, graph capture, and engine workspaces consume memory on each device.
- The busiest device sets the limit. Aggregate free memory cannot rescue a stage that runs out of its own allocation.
TensorRT-LLM documents KV-cache replication cases when the attention layout has too few heads for the TP degree. Verify your engine’s model-specific behavior and inspect allocations per device. TensorRT-LLM: attention parallelism. Use GPU memory for LLM inference for the initial budget and the KV cache guide for token-dependent storage.
Which parallelism strategy should you try first?
| Situation | First comparison | What decides |
|---|---|---|
| Model and useful cache fit on one GPU | One GPU versus additional independent replicas | Request throughput at the target latency |
| Model needs several tightly connected GPUs | Supported TP sizes with enough cache headroom | Compute saved versus collective time |
| GPUs have relatively slow peer communication | A supported PP layout versus TP | Activation transfer cost and pipeline occupancy |
| Model needs more than one node | TP within nodes plus PP between nodes | Stage balance and cross-node transfer time |
| Single-user streaming is too slow | Smallest viable layout versus wider TP | Inter-token latency at low concurrency |
Independent dense-model replicas have separate caches, so routing also affects prefix reuse and load balance. vLLM’s data parallel deployment guide explains this distinction. Increasing replicas can reduce queueing without making an individual forward pass faster.
Validate latency, capacity, and cost before scaling
- Fix the comparison. Keep model revision, precision, sampling settings, prompt distribution, and output lengths consistent. Record engine and driver versions.
- Check compatibility and placement. Confirm the architecture supports the chosen TP and PP degrees, then verify that workers landed on the intended GPUs and nodes.
- Load realistic traffic. Include short chat, long prompts, bursts, and sustained concurrency. Test both warm and cold prefix-cache behavior.
- Measure user-visible results. Capture time to first token, inter-token latency, tail latency, output tokens per second, queueing, and failures.
- Inspect the bottleneck. Compare per-device memory, stage execution time, and communication time. A successful model load only proves that startup fit.
- Price the whole replica. Divide its total infrastructure cost for a measured interval by useful tokens served during that same interval while meeting the latency target; count all participating GPUs.
Prefill vs. decode explains why prompt processing and token streaming stress a layout differently. Use LLM inference metrics to define the comparison. For implementation, continue with the vLLM course or SGLang course; the Dynamo course covers coordinating serving workers across a fleet.
Sources and further reading
Primary documentation and research behind this guide.