The decision tree
Try prompting first, then retrieval, then fine-tuning. Fine-tuning teaches behaviour, format and domain style; it is a poor and expensive way to inject facts that change, because retraining is the only update path.
Parameter-efficient tuning
LoRA freezes the base weights and trains small low-rank adapter matrices, typically under 1% of parameters. QLoRA quantises the base to 4-bit so a large model fits on one GPU. Adapters are swappable per task and cheap to store.
Data is the project
A few thousand clean, consistent, deduplicated examples beat a hundred thousand noisy ones. Hold out an evaluation set before you start, watch for catastrophic forgetting of general ability, and version datasets like code.
from peft import LoraConfig, get_peft_model
cfg = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
target_modules=["q_proj", "v_proj"], task_type="CAUSAL_LM")
model = get_peft_model(base_model, cfg)
model.print_trainable_parameters() # ~0.3% of total