RAG—retrieval-augmented generation—has survived several turns of the AI hype cycle because the underlying problem has not gone away. A general model does not know your current product manual, your private policies, or what changed last Tuesday. RAG retrieves relevant material and gives it to the model while it composes an answer.
That makes a RAG application two systems wearing one chat box: retrieval and generation. Evaluating only the final prose hides which half failed.
Place the task before the technique #
One useful research taxonomy groups LLM tasks by the reasoning they require:
| Level | Task | Typical approach |
|---|---|---|
| 1 | Explicit facts available directly in a source | Basic RAG |
| 2 | Facts need retrieval plus reasoning | Iterative, planning, graph, or tree RAG |
| 3 | The answer needs domain rationale | Fine-tuning or structured reasoning prompts |
| 4 | The rationale itself is implicit | Offline learning, in-context learning, or fine-tuning |
Basic RAG is aimed primarily at the first level. Calling every retrieval loop an agent does not make it more capable; it merely gives the architecture diagram a promotion.
Follow the pipeline #
At ingestion time, documents are cleaned, divided into chunks, embedded, and stored with useful metadata. At query time, the user’s request is embedded, candidate chunks are retrieved and perhaps reranked, and selected context is passed to the generator.
Every stage can lose information. Bad parsing corrupts the source. Poor chunks split an answer from its heading. Retrieval can return plausible but irrelevant text. The generator can ignore good evidence or confidently embroider it. “The answer sounded professional” is not a diagnostic.
Measure retrieval separately #
Useful retrieval questions include:
- Context relevance: does the retrieved context help answer this query?
- Context recall: did the retriever find the evidence required by the reference answer?
- Accuracy: is the retrieved information itself correct?
- Diversity: are the results useful alternatives, or five near-duplicates?
- Noise robustness: can the system resist irrelevant passages?
Retrieval metrics need a query set that represents actual use. A beautiful score on synthetic questions nobody asks is a very efficient way to optimize the wrong system.
Then measure the answer #
Faithfulness asks whether claims are supported by the retrieved evidence. Answer relevance asks whether the response addresses the user’s question. Correctness compares it with known facts or a reference answer. Latency and refusal behavior matter too: when evidence is insufficient, a good system should decline rather than improvise a policy.
Evaluation can use deterministic metrics, domain experts, or an LLM judge. Frameworks such as RAGAS, ARES, and CRAG automate parts of the process; tools such as LangSmith and LangWatch help run datasets and record results. Human review remains important, both to calibrate judges and to catch failures the metrics did not imagine.
The most useful evaluation report attributes error to a stage. If the right passage never arrived, tune ingestion or retrieval. If the passage arrived and the answer contradicted it, tune generation. One aggregate score cannot tell the team which wrench to pick up.
Sources #
- Zhao et al., Retrieval-Augmented Generation and Beyond
- Es et al., RAGAS
- Saad-Falcon et al., ARES
- RAGAS context recall