// blog

How Much GPU Do You Actually Need to Self-Host an LLM?

The first question every team asks about running AI on their own hardware is some version of “what GPU do we need?” — and the honest answer is: it depends on three things you can write down before buying anything. Here’s the actual math.

The core formula

Model weights are the dominant cost. At a given precision, memory need is roughly:

VRAM ≈ parameters × bytes-per-weight + KV cache + overhead

Bytes per weight by precision:

So a 27B parameter model at Q4 needs roughly 27 × 0.55 ≈ 15 GB just for weights.

Context isn’t free

Every token of conversation history lives in the KV cache, and its size scales with model architecture and sequence length. Practical rule of thumb: budget 1–4 GB for 32k–128k context depending on model and whether you use quantized KV cache (q8_0 KV is nearly free quality-wise and halves that).

This is where most DIY deployments go wrong: they size the GPU to the weights and then hit out-of-memory at long contexts or under concurrent users.

Concurrency multiplies everything

• Single user / internal tool: one GPU with weights + headroom is enough.
• Small team (2–10 concurrent): either a larger card with room for batched requests, or continuous batching via vLLM across a bigger pool.
• Production service: plan for multiple GPUs and measure throughput (tokens/sec) against your latency targets — this is where consulting pays for itself quickly.

Worked examples (2026 hardware)

• 24 GB card (RTX 3090/4090-class): up to ~32B models at Q4 with moderate context. Excellent single-team server.
• 48 GB (dual 24GB, or A6000-class): larger models or serious batching; comfortable production node.
• Multi-GPU workstation (72–96 GB+): frontier-class open weights, high concurrency, or fast fine-tuning.

The decision shortcut

1. Write down the biggest model class you’d accept (e.g., “27B-class is fine”).
2. Add your max context × expected concurrent users.
3. Pick hardware with ~20% headroom over the total.

If step 2 pushes past your budget, quantization, shorter contexts, or a hybrid setup (local for most traffic, cloud burst for peaks) usually closes the gap far cheaper than the next GPU tier.

Sizing this for a real workload — data sensitivity, user counts, latency targets — is exactly what our on-premises LLM deployment service covers. Get in touch for a requirements assessment.