12. Serving economics: quantisation, batching, KV cache

Self-hosting an LLM is a memory problem. Weights plus KV cache must fit in GPU VRAM.

- Weights: ~2 bytes/param at FP16. A 70B model ≈ 140 GB — hence quantisation to INT8/INT4 (AWQ, GPTQ, or FP8 on modern accelerators), roughly halving or quartering that. - KV cache: grows linearly with (batch × sequence length × layers × heads). It usually dominates at long context. PagedAttention (vLLM) allocates it in pages to stop fragmentation waste. - Continuous batching: new requests join the running batch each step rather than waiting for a full batch — the single biggest throughput win in real serving. - Speculative decoding: a small draft model proposes tokens, the big model verifies several at once, cutting latency for accepted spans.

Measure TTFT (time to first token, prefill-bound) and TPOT (time per output token, bandwidth-bound) separately — they have different fixes.

Gotcha: aggressive 4-bit quantisation can quietly degrade reasoning and long-context recall. Re-run your evals after any quantisation change.