Can I run it? VRAM & KV cache calculator

estimate above is arithmetic from published specs. measured below is what real builds reported.

While generating, the model stores a key and a value vector for every token, in every attention layer. That store is the KV cache, and it grows with context:

KV bytes = 2 × layers × KV heads × head_dim × context tokens × concurrent sequences × bytes per value

• 2: one key and one value.
• layers, KV heads, head_dim: from the model's config.json (num_hidden_layers, num_key_value_heads, head_dim). Grouped-query attention models have far fewer KV heads than attention heads, which is why their cache is small.
• bytes per value: FP16/BF16 = 2, FP8 = 1, llama.cpp q8_0 = 1.0625 and q4_0 = 0.5625 (34 and 18 bytes per block of 32, scales included).

Worked example, Llama 3.3 70B (80 layers, 8 KV heads, head_dim 128, FP16): 2 × 80 × 8 × 128 × 2 B = 327,680 bytes per token (≈0.33 MB). At 32,768 tokens that is 10.7 GB, as much as a 16 GB card. At q8_0 it is about half.

Why 2026 models need less: many new models only put full attention on some layers. Sliding-window layers (gpt-oss, Gemma 4, Muse, Step 5) cache just their window (128–2,048 tokens). Linear-attention / Gated DeltaNet layers (Qwen3.5–3.8, Ornith) and Mamba layers (Nemotron) keep a fixed-size state that does not grow. MLA (DeepSeek R1) caches one compressed latent per layer. The calculator reads these per-layer types from each model's config.json and counts only what grows with context.

Engines add some extra for compute buffers, and allocate KV in blocks/pages. vLLM pre-allocates all memory left after weights (up to gpu_memory_utilization, default 0.92) as KV cache, so it will look full however small your context is (vLLM source, FP8 KV cache docs).

• Sweet spot: Q4_K_M (GGUF) or AWQ / GPTQ 4-bit (vLLM). This is the size most people run.
• Have room? Q5_K_M or Q6_K, closer to the original.
• Near-lossless: Q8_0 or FP8.
• Below ~4 bits (Q3_K, Q2_K, IQ3, 3-bit EXL) we do not recommend it unless nothing else fits; the calculator warns you. Very large models tolerate it better than small ones.

Format	Bits / weight	Engines
Q8_0 (GGUF)	8.50	llama.cpp / Ollama / LM Studio
Q6_K (GGUF)	6.56	llama.cpp / Ollama / LM Studio
Q5_K_M (GGUF)	5.70	llama.cpp / Ollama / LM Studio
Q4_K_M (GGUF)	4.89	llama.cpp / Ollama / LM Studio
IQ4_XS (GGUF)	4.46	llama.cpp / Ollama / LM Studio
Q3_K_M (GGUF)	4.00	llama.cpp / Ollama / LM Studio
Q2_K (GGUF)	3.16	llama.cpp / Ollama / LM Studio
F16 (GGUF)	16.00	llama.cpp / Ollama / LM Studio
MXFP4 (gpt-oss native)	4.25	llama.cpp / Ollama / LM Studio, vLLM / SGLang
BF16 / FP16 (unquantized)	16.00	vLLM / SGLang
FP8 (W8A8)	8.00	vLLM / SGLang
AWQ 4-bit (g128, ≈)	4.25	vLLM / SGLang
GPTQ 4-bit (g128, ≈)	4.25	vLLM / SGLang
GPTQ 8-bit (g128, ≈)	8.25	vLLM / SGLang
NVFP4 (Blackwell)	4.50	vLLM / SGLang
EXL3 (ExLlamaV3)	you choose (2–8)	ExLlamaV2 / V3
EXL2 (ExLlamaV2)	you choose (2–8)	ExLlamaV2 / V3
MLX 4-bit	4.50	MLX
MLX 8-bit	8.50	MLX

Hardware matters for some formats. FP8 compute needs compute capability ≥ 8.9 (Ada, Hopper, Blackwell); on Ampere cards such as the RTX 3090, vLLM runs FP8 checkpoints as weight-only W8A16 with FP8 Marlin kernels, saving memory but not compute (vLLM FP8 docs). NVFP4 is 4-bit E2M1 values with one FP8 scale per 16 values plus a per-tensor FP32 scale (4.5 bits per weight), about 3.5× smaller than FP16 and 1.8× smaller than FP8, and is computed natively by Blackwell's 5th-gen Tensor Cores (NVIDIA). On older NVIDIA GPUs vLLM falls back to weight-only W4A16 via Marlin (vLLM ModelOpt docs). MXFP4 (gpt-oss) shares one power-of-two scale per 32 values, 4.25 bits per weight; gpt-oss keeps attention and embeddings in BF16, so its real files are larger than params × 4.25 ÷ 8.

GGUF bits-per-weight are llama.cpp's measured values (quantize README); real files vary a little by model. AWQ/GPTQ ≈ 4.25 / 8.25 are our approximation (4 or 8 bits plus group-128 scales and zero points); embeddings and lm_head often stay 16-bit, so files run slightly larger. MLX: group size 64 with a 16-bit scale and bias, so 4-bit = 4.5 bpw (MLX docs). vLLM hardware support: vLLM quantization docs.

Most calculators tell you to buy as much system RAM as VRAM, or more. If the model fits fully in VRAM, you don't need that. System RAM then only holds the OS and the inference engine, and the weights pass through it once while loading. llama.cpp memory-maps the model file, so pages are read from disk and copied to the GPUs. They don't all have to sit in RAM at once, so RAM can be smaller than the model.

RAM becomes important only when you offload: layers or MoE experts that don't fit on the GPUs run from system RAM, and then both its size and its bandwidth (channels, DDR4 vs DDR5) set your speed.

Maintainer note: our maintainer runs 8× RTX 3090 with just 32 GB of DDR5 ECC RDIMM at home (for a measured 8x 3090 example from another builder, see 8x RTX 3090 serving DeepSeek V4 Flash (LimeChain vLLM kit)), and it works really well, because the models live in the 192 GB of VRAM. Spend the money on GPUs, not RAM, unless you plan to offload.

Caveat: converting or quantizing a model yourself, and some loaders, can need much more RAM for a short time. Keep swap enabled if you do that.

Memory adds up: 4× RTX 3090 = 96 GB, 8× = 192 GB, two DGX Sparks = 256 GB. Each GPU loses roughly 1 GB to its CUDA context and buffers (our assumption), and vLLM only uses 0.92 of each GPU by default.

• Pipeline / layer split (llama.cpp default, vLLM --pipeline-parallel-size, ExLlama): each GPU holds a block of layers. Any number of GPUs, different sizes allowed, little PCIe traffic. One chat stream uses one GPU at a time, so you get capacity rather than speed.
• Tensor parallel (vLLM/SGLang --tensor-parallel-size, llama.cpp --split-mode row): every layer is split over all GPUs, which work on each token together. It is faster, but needs a lot of interconnect bandwidth, and in vLLM the attention-head count must divide by the TP size, so 2, 4 or 8 GPUs are the usual choice. For 3 or 6 GPUs, combine TP with PP or use PP only (vLLM parallelism docs).
• Multi-node: NVIDIA says up to four DGX Spark systems can work together over ConnectX networking (each Spark has 128 GB unified memory, 273 GB/s and a 200 Gb/s ConnectX-7 NIC; NVIDIA). Mac Studios can cluster over Thunderbolt. See the DGX Spark recipes and the measured multi-node builds.

RigBase has 33 measured multi-GPU / multi-node builds; the calculator lists those matching your setup.

The tokens/sec most benchmarks quote is generation (decode) speed: how fast the answer streams out. Before the first token appears the model must first read the whole prompt. That is prompt processing, or prefill. Chat prompts are short, so prefill hardly matters there. Coding assistants and agents are different. They send the open files, the repo map, tool output and the chat history on every turn, and these prompts easily reach thousands of tokens. On such prompts the wait before the first token (time to first token, TTFT) can take longer than generating the answer.

Prefill is compute-bound, while decode is mostly limited by memory bandwidth. So two rigs with similar generation speed can differ a lot in prefill. Discrete NVIDIA/AMD GPUs usually prefill much faster than unified-memory machines and CPU offload setups. Rough TTFT for a prompt: TTFT ≈ prompt tokens ÷ prefill tok/s (plus model load time; prompt caching in llama.cpp, MLX or vLLM can skip the repeated part).

RigBase now records prefill speed for 168 results on 90 builds, with the prompt length each source measured it at. Where it was measured on 512+ tokens we show an estimated TTFT for an 8k-token prompt, calculated from that measured speed. Across the current data the estimate ranges from Llama 2 7B on Single RTX 4090 (llama.cpp CUDA scoreboard): 14,771 tok/s prefill measured @ 512 tokens → ≈0.55 s to Llama 3.1 405B on Mac Studio M3 Ultra 512GB (Geerling test): 12.92 tok/s prefill measured @ 4,096 tokens → ≈634 s. Prefill usually slows down as prompts get longer, so an estimate taken from a much shorter or much longer prompt is only indicative. Where a source measured TTFT itself, we show it separately as measured. See the "Prompt processing" column on every build and model page.

Not enough VRAM? Rent before you buy. Run the model on a rented 24–96 GB cloud GPU by the hour and see the real speed before spending €1,000+. Try it on Runpod
Referral link: if you sign up with Google and add $10, Runpod gives you and RigBase bonus credit (terms). It never affects our data.