Most RAG systems ship on vibes. But every part of the RAG pipeline — chunking, retrieval, reranking — is a knob, and you can't tune knobs you don't measure. Evaluation is what turns "it feels better" into "recall went from 0.6 to 0.8."
Measure retrieval separately from generation
Split the problem. First ask: did retrieval surface the right passage at all? (recall@k, precision@k, and whether the gold passage is in the top results.) Only then ask: did the model use it faithfully? Mixing the two hides where the failure is.
Answer faithfulness and citations
For generation, the metric that matters is faithfulness — is every claim in the answer supported by a retrieved passage? Citations make this checkable: if the model must point to its source, you can catch hallucinations automatically instead of eyeballing them.
Build a small reference set
You don't need a giant benchmark — 30–50 real questions with known-good answers catch most regressions. Run it whenever you change chunking, your embeddings, or reranking. Kit for AI returns cited passages by default, so faithfulness is easy to check. Start free and grade your own questions.