Decoding strategies
Greedy decoding takes the argmax and is repetitive. Temperature flattens or sharpens the distribution; top-k and top-p (nucleus) restrict sampling to the plausible mass. Low temperature for extraction and code, higher for ideation.
Making inference cheap
The KV cache stores past keys and values so each new token costs one step instead of recomputing the prefix. Quantisation to int8 or 4-bit shrinks weights with modest quality loss; batching, paged attention and speculative decoding raise throughput further.
Hallucination
The model samples plausible continuations; it has no truth oracle and no memory of provenance. Grounding in retrieved documents, requiring citations, constraining output schemas and verifying with tools are the practical mitigations — not stern prompting.