Improve retrieval before training
Run a PDCA loop to improve RAG retrieval quality before deciding on fine-tuning.
A RAG assistant answers only 61 percent of held-out policy questions correctly. The team wants to fine-tune, but error review shows many answers lacked the right source in retrieved context. PDCA: Plan a retrieval hypothesis, Do a small change, Check eval impact, Act on what the data shows The trap is treating any bad RAG answer as proof that RAG is inadequate. If retrieval did not bring the right evidence into context, fine-tuning the answer style is solving the wrong part of the system. Plan Hypothesis: long PDF chunks and missing effective-date metadata cause current-policy misses. Success metric: top-5 retrieval…
Sign up free — one personalized lesson every day, matched to your role and goals.
Already have an account? Sign in