GPU VRAM Calculator for LLMs

The calculator sums four consumers of GPU memory: model weights, the KV cache, a working set, and framework overhead. Weights equal the parameter count times the bytes per weight (2 for FP16, 1 for INT8, 0.5 for INT4). For inference it adds a KV cache sized from your layers, hidden size, context length, and batch, plus a small activation buffer. It then adds a CUDA context allowance and a fragmentation margin, and reports the total in GiB.

A 7 billion parameter model in FP16 stores about 13.0 GiB of weights. With a 4,096-token context, one sequence, and a 32-layer, 4,096-hidden-size model, the calculator adds about 2.0 GiB of KV cache, 2.6 GiB of working memory, and 2.8 GiB of overhead, for about 20.4 GiB total. In INT4 the same assumptions produce about 7.5 GiB before the calculator's reserved-VRAM margin, so 8 GB can be tight and 12 GB offers more headroom.

Quantization sets how many bytes each weight occupies. FP16 and BF16 use 2 bytes per weight, INT8 uses 1 byte, and INT4 uses 0.5 bytes, so moving from FP16 to INT4 cuts the weight memory to a quarter. This calculator assumes an FP16 KV cache, so it does not shrink with weight quantization. Long contexts can still dominate memory even on a heavily quantized model.

The KV cache stores one Key and one Value vector per layer for every token in the context window so the model does not recompute attention each step. Its size is 2 times layers times hidden size times 2 bytes times context length times batch size. Because it scales linearly with context length and batch size, doubling either one doubles the cache, and very long contexts can need more memory than the weights themselves.

Full fine-tuning with mixed-precision Adam keeps far more than the weights in memory. The standard footprint is about 16 bytes per parameter: FP32 master weights, FP32 momentum, FP32 variance, plus FP16 weights and gradients. That is roughly eight times the FP16 weight size, so a 7B model can need well over 100 GB. LoRA and QLoRA freeze the base weights and train small adapters instead, which is why they fit on a single consumer card.

Treat the result as a planning estimate, not an exact figure. Real usage shifts with the runtime (llama.cpp, vLLM, Transformers), the attention kernel, paged or quantized KV caches, and the exact weight format, which the model approximates with fixed allowances. Leave headroom and validate the result with the actual runtime before choosing hardware.