Evaluate Retrieval Before You Blame the LLM
When a RAG assistant gives a wrong answer, the first reaction is usually to blame the language model. Teams try a bigger model, rewrite the prompt, or add more instructions.
Often none of it helps, because the model never saw the right information. A model cannot use a passage it was never given, and it will confidently answer from whatever it was given instead.
This blog explains why retrieval should be evaluated first, how to separate retrieval failures from generation failures, which measures to use, and how to fix what you find. The CodersArts wrote this guide for teams debugging production RAG systems, and it uses no invented figures, because every system and every question set is different.
Retrieval Evaluation at a Glance
Why Retrieval First?
Because the answer can only be as good as the context. If the right passage is missing or buried, no model can reliably recover it.
What Does Retrieval Evaluation Measure?
Whether the system finds the passages that contain the answer, how high it ranks them, and how much irrelevant material comes along.
How Does It Differ From Generation Evaluation?
Generation evaluation asks whether the answer is correct, grounded, and well written. Retrieval evaluation asks whether the model was given what it needed.
What Does It Give You?
A clear split between problems in search and problems in the model, so you fix the right thing.
The Short Answer
Build a test set, measure retrieval on its own, then measure generation given good context. Only then decide whether the language model is the problem.
Why Blaming the LLM Is Often a Mistake
The language model is the visible part of the system, so it takes the blame. The causes often sit earlier.
The Model Only Sees What Retrieval Gives It
If the passage with the answer is not in the context, the model must guess, refuse, or use general knowledge.
Wrong Context Produces Confident Wrong Answers
Near-miss passages can mislead a model into a fluent answer that is wrong.
Bigger Models Do Not Fix Missing Information
A stronger model may write a better-sounding wrong answer. It cannot supply facts that were never retrieved.
Prompt Changes Hide the Problem
Tweaking instructions can improve a few examples while the root cause stays in place.
Costs Add Up
Switching to a larger model raises cost and delay on every request, to treat a problem it did not cause.
The Two Failure Types: Retrieval and Generation
Every bad answer comes from one of a few places. Naming them helps you test each.
Retrieval Misses
The right passage was in the knowledge base but was not retrieved, or was ranked too low to reach the model.
Knowledge Base Gaps
The answer does not exist in the indexed content at all, because documents are missing, outdated, or failed to parse.
Context Noise
The right passage was retrieved but surrounded by irrelevant text that distracted the model.
Generation Errors
The right passage was in the context, and the model still misread it, ignored it, or added unsupported claims.
Question Problems
The question was ambiguous, or it needed information from several documents that the system did not combine.
Building a Test Set for Retrieval
A good test set is the foundation of every measurement.
Collect Real Questions
Use questions from real users, support logs, or subject-matter experts. Include easy, hard, and ambiguous ones.
Mark the Source Passages
For each question, record the document and passage that contain the answer. Experts should confirm these.
Cover the Variety of Your Content
Include different document types, topics, and question styles, so results are not skewed toward one area.
Include Questions With No Answer
Add questions the knowledge base cannot answer, to check how the system behaves when nothing relevant exists.
Keep It Stable and Versioned
Freeze a version of the test set so results can be compared over time, and add new cases as failures appear.
Measures That Tell You Whether Retrieval Works
Several simple measures cover most needs.
Hit Rate and Recall at K
Does at least one correct passage appear in the top results, and what share of all the relevant passages appear? This tells you whether the answer was reachable.
Precision at K
What share of the retrieved passages are relevant? Low precision means noise in the context.
Rank-Aware Measures
Measures such as mean reciprocal rank and normalized discounted cumulative gain reward placing the right passage near the top, which matters when only a few passages reach the model.
Context Relevance and Coverage
Judge whether the retrieved context contains the information needed for a complete answer, using human review or a validated automatic judge.
Failure Categories
Count misses by cause, such as chunking, missing documents, or ranking, so you know where to act.
Latency and Cost
Track how long retrieval takes and what it costs, since fixes often trade quality for speed.
How to Separate Retrieval From Generation
A few tests isolate the cause of a bad answer.
Check What Was Retrieved
For a failing question, read the retrieved passages. Is the answer there? If not, the problem is retrieval.
Give the Model the Right Passage
Provide the known correct passage directly and see whether the model answers correctly. If it does, generation is fine and retrieval is the issue.
Give the Model Extra Noise
Add irrelevant passages around the correct one. If answers degrade, the model is sensitive to noise, which makes precision important.
Check Grounding
Compare the answer with the passages. Claims that appear nowhere in the context point to generation problems.
Check the Knowledge Base
Search the documents directly for the answer. If it is not there, the fix is content, not code.
Measure Both Stages Together
Report retrieval measures and answer quality side by side so improvements in one can be linked to the other. Our guide on how to evaluate a RAG system before shipping shows one way to do this.
Common Retrieval Problems and Their Fixes
Once you know retrieval is the cause, the fixes are usually concrete.
Poor Chunking
Chunks that split ideas in half, or lump many topics together, hurt matching. Try chunk sizes and boundaries that follow headings and paragraphs.
Weak Embeddings for Your Domain
A general embedding model may miss specialist terms. Test alternatives or fine-tune on domain data.
Missing Keyword Matching
Exact terms, codes, and names can be lost in pure vector search. A hybrid of keyword and vector search often helps.
No Metadata Filters
Searching everything when the question concerns one product, region, or year adds noise. Use filters.
Ranking Problems
The right passage is retrieved but ranked low. Consider a re-ranker and measure whether it helps.
Stale or Missing Content
Outdated or unparsed documents mean the answer is absent. Fix ingestion and refresh schedules.
Query Mismatch
Short or vague questions may not match document wording. Query rewriting and expansion can help.
An HR Policy Assistant
let us look at an assistant that answers employee questions about leave and benefits.
The Complaint
Users report wrong answers about carry-over rules. The team's first idea is to move to a larger model.
The Test
They build a set of real questions with marked source passages and measure retrieval first. For carry-over questions, the correct passage is often missing from the top results.
The Diagnosis
Reading the failures shows that the policy table was split across chunks, and the question wording differs from the document's terms.
The Fix
They rechunk by section, add keyword matching, and filter by policy year. Retrieval measures improve on the test set.
The Result
With the right passage in context, the existing model answers correctly. The larger model is never needed, and the team keeps the test set to catch regressions.
Bringing Retrieval Evaluation Into Your Workflow
Make it a habit, not a one-off.
Build the Test Set Early
Create it before tuning anything, so improvements can be proven.
Run Retrieval Tests on Every Change
Chunking, embeddings, filters, and ranking changes should each be measured.
Add Failures to the Set
Every real bad answer becomes a new test case.
Monitor in Production
Sample real queries, check retrieved passages, and watch for drift as content changes.
Report Both Stages
Share retrieval and answer quality together so the whole team sees where problems originate.
Special Cases Worth Planning For
Some systems need extra attention when evaluating retrieval.
Multi-Hop Questions
Some questions need facts from several documents. Mark every required passage, and measure whether all of them are retrieved, not just one.
Tables, Images, and Scanned Files
If parsing loses table structure or text from scans, retrieval will fail no matter how good the search is. Test ingestion quality on these files directly.
Multiple Languages
Check retrieval separately for each language, and for questions asked in one language about documents written in another.
Frequently Updated Content
When documents change often, test that new versions are indexed promptly and old versions do not outrank them.
Access-Controlled Content
If users see different documents, evaluate retrieval under each permission set, so filters work and no restricted passage leaks.
Common Mistakes in RAG Evaluation
Evaluating Only the Final Answer
Without looking at retrieval, you cannot tell why an answer failed.
Using a Tiny or Unrealistic Test Set
A few handpicked questions hide real failures.
Marking the Source Passage Loosely
Vague labels make retrieval metrics meaningless.
Changing Several Things at Once
You cannot tell which change helped. Test one change at a time.
Ignoring Questions With No Answer
A system that always answers will make things up when the knowledge base is silent.
Advantages and Limitations of Evaluating Retrieval First
Where Retrieval-First Evaluation Delivers the Most Value
Advantage | Details |
Finds the real cause | Separates search problems from model problems. |
Saves money | Avoids paying for larger models to fix retrieval issues. |
Faster debugging | Targets the failing stage directly. |
Measurable progress | Metrics show whether each change helped. |
Prevents regressions | A stable test set catches new problems. |
What Are the Trade-Offs of Evaluating Retrieval First?
Limitation | Details |
Test set effort | Building labeled questions and source passages takes time and expert input. |
Labels can be incomplete | Several passages may answer a question, and the labels may miss some. |
Not the whole story | Good retrieval does not guarantee a good answer, so generation still needs testing. |
Needs upkeep | Content and questions change, so the set must be refreshed. |
How Much Does Retrieval Evaluation Cost?
The main cost is people's time: collecting questions, marking source passages, and reviewing failures. Compute costs for running tests are usually small compared with the cost of the system itself. Automatic judges can reduce review effort, though they need validation.
The return is usually high, because the evaluation prevents expensive detours such as switching models or rebuilding pipelines on a wrong guess.
A small, carefully built test set is often enough to start, and it can grow as real failures are found.
Ways to Debug a Bad RAG Answer Compared
Teams use several approaches.
Trying a Different Model
Quick to try and costly. It helps only when generation is the real cause.
Rewriting the Prompt
Cheap and often useful for style and grounding, and it cannot supply missing context.
Inspecting Retrieved Passages by Hand
Fast for single cases and gives deep insight. It does not scale.
Systematic Retrieval Evaluation
Slower to set up and the most reliable way to find patterns and prove fixes.
Which Teams Get the Most Value From Retrieval-First Evaluation?
This approach helps teams that:
Run RAG in production with real users
See wrong or unsupported answers
Have large or changing knowledge bases
Want to control model and infrastructure cost
Need evidence for stakeholders that changes work
Should You Really Evaluate Retrieval Before Blaming the LLM?
Yes, in most cases. Retrieval is the stage that decides what the model can know, and it is often where problems begin. Evaluating it first costs little, narrows the search for the cause, and prevents expensive changes that do not help.
The language model may still be the problem sometimes, and the same test set will show that too. Our RAG development team can build the evaluation and fix what it finds.
How CodersArts Helps You Find and Fix RAG Problems
CodersArts helps businesses diagnose RAG systems and improve them with evidence.
RAG Audits
We review your ingestion, chunking, indexing, and search settings and locate where answers fail.
Test Set Design
We build labeled question sets from your real use cases, with expert-confirmed source passages.
Retrieval and Generation Metrics
We measure both stages and report where the problems originate.
Fixes and Optimization
We improve chunking, hybrid search, filters, ranking, and prompts, and verify each change.
Ongoing Support and Maintenance
We monitor quality in production and catch regressions as content changes.
How Does a CodersArts RAG Evaluation Engagement Work?
Most engagements follow four stages. A discovery session covers your content, users, and failures. A baseline stage builds the test set and measures retrieval and answer quality.
A diagnosis stage sorts failures by cause. A fix stage tests changes one at a time and ships those that help. Each stage ends with measured results.
Start with a conversation.
Email contact@codersarts.com with a short description of your RAG system, and we will suggest where to begin.
Frequently Asked Questions
Why Evaluate Retrieval Before the LLM?
Because the model can only use what retrieval gives it. If the right passage is missing, no model can answer reliably.
What Is Recall at K?
The share of relevant passages that appear in the top K retrieved results. It shows whether the needed information was reachable.
How Do I Know Whether the Problem Is Retrieval or Generation?
Check whether the correct passage was retrieved. Then give the model the correct passage directly and see if it answers correctly.
How Big Should the Test Set Be?
Large enough to cover your question types and document areas. Start small and grow it as real failures appear.
Can an LLM Judge Evaluate Retrieval?
It can help judge relevance and coverage, as long as you validate it against human labels first.
What If the Answer Is Not in the Knowledge Base?
Then the fix is content or ingestion, not code. Include no-answer questions to test how the system behaves.
What Should a Business Evaluate Before Changing Models?
Evaluate retrieval quality, knowledge base coverage, chunking, ranking, and context noise first, because these often explain the failures.
What Services Does CodersArts Offer?
Beyond RAG evaluation and other AI and RAG specific delivery and partnership work, CodersArts offers a wider range of services that agencies, businesses, and individual developers regularly rely on, whether as part of a partnership or on their own.
AI and RAG Development
Custom AI and RAG development, starting from proof of concept through to full production builds, along with broader LLM and generative AI development for businesses building AI-powered products and internal tools.
Consultation
Project consultation for businesses and agencies evaluating an AI or machine learning initiative, helping assess feasibility, recommend the right technical approach, and scope a project before committing to full development.
One-on-One Mentorship
Personalized, expert-led mentorship for developers and teams looking to build hands-on AI, machine learning, or AI engineering skills, with guidance tailored to individual or team goals and current experience level.
Dedicated Team and Team Augmentation
Dedicated AI and machine learning engineering teams, or engineers who work as an extension of an existing in-house or agency team, scaling up or down based on project needs.
Ongoing Support and Maintenance
Post-launch monitoring, optimization, and maintenance for AI and machine learning systems already in production, helping ensure performance and reliability do not degrade over time.
Job Support Services
Remote job support for developers and engineers working on live AI, machine learning, or LLM projects, including pair programming, code reviews, workflow setup, debugging, and help meeting sprint deadlines under expert guidance.
Corporate and Team Training
Structured training and workshops for teams looking to build internal AI and machine learning capability, covering hands-on implementation as well as best practices for evaluation and production readiness.
White-Label and Partnership Delivery
CodersArts also partners with agencies, consultancies, and technology companies to deliver AI and machine learning development on their behalf, whether white-label, co-branded, or embedded alongside an existing team.
Whether you are a business running or planning a RAG assistant, an agency looking for a delivery partner, or a developer seeking hands-on mentorship, CodersArts offers services to support your AI journey.
Reach out at contact@codersarts.com or visit www.codersarts.com to discuss your RAG evaluation or broader AI project.
Continue Exploring AI Resources
If you found this blog helpful, explore more AI, RAG, and enterprise AI resources from CodersArts AI to see how organizations are applying these systems to real world applications.
Build a Cost-Efficient Writing Quality Checker with Tiered Model Routing and OpenAI
Build Your First A2A Agent: An Email Drafting Pipeline Using Python and OpenAI
Building an AI Interview Prep Agent with Qwen 3.7 Max and Streamlit
https://www.codersarts.com/post/building-an-ai-interview-prep-agent-with-qwen-3-7-max-and-streamlit
Academic Research Assistance and Literature Review Automation Using RAG
Clinical Decision Support Systems Using RAG: Intelligent Diagnostic Assistance for Healthcare
Financial Decision Making with RAG Powered Market Intelligence
https://www.codersarts.com/post/financial-decision-making-with-rag-powered-market-intelligence
Comments