Capacity planning tool
LLM GPU memory and KV cache calculator
Estimate dense model weight storage and standard attention KV-cache tensors. Change the architecture, precision, context, and concurrency to see how capacity changes.
Model and workload
Formulas and worked example
Weight bytes = parameters × bits per weight ÷ 8
KV bytes = 2 × layers × KV heads × head dimension
× cached tokens × concurrent sequences × bytes per value
GiB = bytes ÷ 1,073,741,824With 8 billion parameters at 16 bits, weights occupy about 14.901 GiB. At 32 layers, 8 KV heads, head dimension 128, 4,096 cached tokens, one sequence, and two bytes per KV value, the cache needs 0.5 GiB. The combined tensor estimate is 15.401 GiB before overhead.
Use KV heads, not query heads, for grouped-query attention. Cached tokens include both the prompt and generated continuation. This estimate assumes equal cached lengths and no prefix sharing.
What this estimate leaves out
Quantization scales and metadata, runtime allocations, CUDA graphs, temporary activations, fragmentation, and non-model processes require extra memory. The additional allowance is your input, not a measured default. MoE weights depend on total resident parameters, not active parameters. Hybrid, sliding-window, and latent-attention models may need different cache formulas.
This is not a GPU compatibility check, latency prediction, or guarantee that a model will fit. Do not divide the total by GPU count without checking how weights and cache are actually sharded.