How to Build a Self-Correcting RAG System with Next.js 14, LangChain & LangGraph
- Jun 10
- 14 min read
Introduction: When Your RAG Bot Lies to You
You've shipped a documentation assistant. Users love it — until the day a senior engineer runs a query about your internal API's retry configuration, and the bot replies with a perfectly formatted, confidently worded answer citing an exponentialBackoff option that has never existed in your codebase. No hedging. No "I'm not sure." Just fluent, authoritative fiction.
This is the defining failure mode of naive Retrieval-Augmented Generation. The retriever pulls documents that are close-but-wrong, the LLM stitches them into a plausible-sounding answer, and the system has no mechanism to notice either failure. It cannot ask itself: Were those documents actually relevant? Did my answer stay within what the sources support?
A self-correcting RAG system solves this. It uses an LLM-as-judge to grade its own retrieved documents for relevance, broadens and retries the search when they're weak, grades the generated answer for faithfulness against the sources, and refines or refuses the answer when it detects a hallucination — all while streaming its live reasoning trace to the browser so every correction is auditable.
Real-world use cases for this architecture include:
Internal and technical documentation assistants where a wrong answer has real costs
Developer docs and API Q&A bots that must cite sources, not invent them
Knowledge-base search for support teams that cannot fabricate policy
Legal, research, or medical document querying where hallucination is unacceptable
Any RAG product that needs an observable, debuggable reasoning trace
Learning agentic and corrective-RAG patterns on a real, working project
This post covers the full architecture, the recommended technology stack, the implementation phases, and the non-obvious challenges you'll hit building this system. It does not include the complete source code — that lives in the full course on Codersarts Labs.
📄 Before you dive in — grab the free PRD template that maps out this entire system: architecture, API spec, sprint plan, and system prompt. [Download the free PRD]
How It Works: The Core Concept Self-Correcting RAG
To understand why self-correcting RAG matters, you first need to understand why the naive approach fails — and it fails in two distinct ways.
Failure mode 1: Bad retrieval. When a user asks a nuanced or version-specific question, a cosine-similarity search may return documents that are topically related but not actually relevant to the query. A naive RAG system passes those documents straight to the LLM regardless. The LLM, trained to be helpful and fluent, fills the gap between "sort of relevant" and "actually correct" with confident inference — which is just a polite name for hallucination.
Failure mode 2: Unfaithful generation. Even when retrieval is perfect, the LLM can add claims the source documents don't support. It may conflate two documents, extrapolate from partial information, or simply confabulate a detail. Naive RAG has no post-generation check.
Self-correcting RAG addresses both failures with a looping state machine and an LLM-as-judge at each critical checkpoint:
Retrieve the top-K most similar documents.
Grade each document for relevance to the query. If fewer than a threshold pass, the query is too narrow.
Broaden the query (strip version numbers, framework-specific noise) and loop back to retrieve — capped at three iterations to prevent runaway costs.
Generate an answer using only the documents that passed the relevance grade. If none qualify, return an honest "no relevant documentation found" message instead of inventing an answer.
Grade the answer for faithfulness: does every claim in the answer appear in the source documents? Assign a faithfulness score and flag any hallucinated claims.
Refine the answer (capped at one pass) to remove or qualify unsupported claims. If the answer still fails after refinement, refuse and explain why.
Think of it like a careful researcher who checks whether their sources are relevant before writing, and then re-reads their draft to verify every claim is backed by a citation — rather than hitting send the moment they finish the first draft.
Here is the complete two-phase data-flow:
SETUP PIPELINE (offline — run once or when docs change)
────────────────────────────────────────────────────────
/docs (markdown files)
→ text-splitter (~800-char overlapping chunks)
→ text-embedding-3-small (OpenAI)
→ data/vectors.json [ { embedding[], pageContent, metadata } ]
RUNTIME PIPELINE (per query)
────────────────────────────────────────────────────────
Browser ──POST /api/rag──► Next.js Node Route
│
LangGraph state machine
│
┌───────────────▼────────────────┐
│ retrieve (top-K cosine sim) │
└───────────────┬────────────────┘
│
┌───────────────▼─────────────────┐
│ grade_docs (parallel LLM judge)│
└───────────┬──────────┬──────────┘
pass ◄─┘ └─► fail (iter < 3)
│ │
│ broaden_query ──► retrieve (loop)
│
┌──────▼──────────────────────────┐
│ generate (gpt-4o + sources) │
└───────────────┬─────────────────┘
│
┌───────────────▼─────────────────┐
│ grade_answer (faithfulness) │
└───────────┬──────────┬──────────┘
faithful─┘ └─► hallucinated
│ │
│ refine_answer (max 1x)
│ │
┌──────▼────────────────────────▼───┐
│ SSE stream → Browser React UI │
└───────────────────────────────────┘
Every node in the graph emits a trace event that is forwarded to the browser as a Server-Sent Event, so the user sees the correction loop unfold in real time.
System Architecture Deep Dive
Architecture Layers
The application is cleanly separated into five layers, each with a single responsibility.
Presentation layer. A React 18 / Tailwind CSS frontend. The user submits a query, and the UI opens an SSE connection to the API route. As trace events arrive — "grading document 3 of 5… relevance: 0.42, filtered out" — they render in a live, colour-coded thought trace panel. When the final answer arrives, it appears alongside a faithfulness badge and source references, with code blocks syntax-highlighted via react-markdown and Prism.
API and streaming layer. A Next.js App Router API route running on the Node.js runtime (not Edge). It receives the POST, initialises the LangGraph state machine, and sets up an EventEmitter. As graph nodes emit trace events, the route forwards them as SSE data: frames. The connection stays open until the graph reaches its terminal node, at which point the route sends a done event and closes the stream.
Orchestration layer. A LangGraph StateGraph with typed state channels (query, originalQuery, documents, filteredDocuments, generation, iterationCount, correctionCount) and reducer functions. Six nodes — retrieve, grade_docs, broaden_query, generate, grade_answer, refine_answer — are connected by typed edges and conditional routing functions. The entire looping, branching workflow is expressed declaratively.
AI judges and generation layer. Two OpenAI models play distinct roles. gpt-4o-mini acts as the grader LLM — it is cheap, fast, and sufficiently capable of relevance and faithfulness judgments. gpt-4o is used for answer generation, where quality matters more than cost. Both graders use withStructuredOutput with a Zod schema at temperature 0, ensuring every judgment returns a typed, validated object: { score: number, relevant: boolean, reasoning: string } for document grading, and { faithfulnessScore: number, hallucinationRisk: "low" | "medium" | "high", problematicClaims: string[], refinementSuggestion?: string } for answer grading.
Data layer. A local JSON array at data/vectors.json. Each entry is { embedding: number[1536], pageContent: string, metadata: { source: string, hasCode: boolean } }. Retrieval is pure cosine similarity implemented in ~15 lines of TypeScript — no vector database, no external service. This means the entire system can run locally with a single npm run dev after ingesting your docs.
Component Table
Component | Role | Technology Options |
Full-stack framework | API routes, SSE, React UI | Next.js 14 (App Router) |
Orchestration | Looping state machine with conditional edges | LangGraph 0.2.x |
LLM abstraction | Unified model/embedding interface | LangChain 0.3.x |
Generation model | Produce the final answer | OpenAI gpt-4o / gpt-4o-mini / Claude 3 |
Grading model | Relevance + faithfulness judgments | OpenAI gpt-4o-mini / Gemini Flash |
Embeddings | Convert text to vectors | text-embedding-3-small / ada-002 / Cohere |
Vector store | Retrieve top-K similar chunks | Local JSON / Pinecone / pgvector / Chroma |
Structured output | Validated, typed LLM responses | Zod + withStructuredOutput |
Streaming | Push reasoning trace to browser | Server-Sent Events (SSE) |
UI framework | Render trace + answer | React 18 + Tailwind CSS 3 |
Data Flow: Query to Streamed Answer
The user types a query in the React UI and clicks Submit.
The UI POSTs { query } to POST /api/rag and opens an SSE reader on the response stream.
The API route initialises the LangGraph StateGraph with { query, originalQuery: query, iterationCount: 0 } and calls .stream().
The retrieve node loads vectors.json, computes cosine similarity for every chunk, and returns the top 5 by score. It emits a trace event: { type: "retrieve", docsFound: 5 }.
The grade_docs node calls the grader LLM in parallel for each document. Each returns { relevant: boolean, score: number }. Relevant docs are collected into filteredDocuments. A trace event is emitted per document.
A conditional edge evaluates: if filteredDocuments.length >= minRequired, route to generate; otherwise, if iterationCount < 3, route to broaden_query; if iterationCount === 3, force-pass and route to generate.
broaden_query uses an LLM call to strip version-specific or framework-specific tokens from the query, increments iterationCount, and loops back to retrieve.
generate builds a prompt from only the relevant documents and calls gpt-4o. If no documents passed grading, it returns a hardcoded honest fallback. Emits a trace event.
grade_answer checks every claim in the generated answer against the source documents. Returns a faithfulness score, hallucination risk level, and any problematic claims.
A conditional edge routes faithful answers to the terminal done state, or to refine_answer if hallucination risk is medium/high.
refine_answer rewrites the answer to remove or qualify the flagged claims, increments correctionCount, then routes to done.
The route sends a final answer SSE event containing the text, faithfulness badge data, and ResponseMetadata, followed by a done event that closes the stream.
The React UI renders the complete thought trace and the badged, source-linked answer.
Non-Obvious Design Decisions
The query is broadened for search, but the answer always addresses the original query. When broaden_query runs, it stores the relaxed query in state.query for retrieval purposes, but state.originalQuery is never mutated. The generate node always reads originalQuery when building its prompt. This means the answer is precise and on-topic even when retrieval had to cast a wider net.
The StateGraph builder must be chained, not called as separate statements. LangGraph's TypeScript API narrows the inferred state type through each .addNode() return value. Writing each call as a separate const graph = builder.addNode(...) statement works in next dev but fails next build with a cryptic type error: 'retrieve' is not assignable to type '__start__'. All .addNode(), .addEdge(), and .addConditionalEdges() calls must be chained on the same builder expression. This is covered in detail in the full course.
Tech Stack Recommendation
Stack A — Beginner / Prototype (build in a weekend)
This stack can be running locally in a few hours and deployed to Vercel in an afternoon. It uses only one external service: the OpenAI API.
Layer | Technology | Why |
Framework | Next.js 14 (App Router) | API + UI in one project, zero config |
Language | TypeScript 5 | Type safety in LangGraph is non-negotiable |
Orchestration | LangGraph 0.2.x | First-class looping/conditional support |
Generation | OpenAI gpt-4o | Best quality/capability for answers |
Grading | OpenAI gpt-4o-mini | Cheap, fast, reliable structured output |
Embeddings | text-embedding-3-small | $0.00002/1K tokens, good quality |
Vector store | Local JSON (cosine sim) | No external service, understand the math |
Validation | Zod | Type-safe structured LLM output |
UI | React 18 + Tailwind CSS 3 | Fast to build, responsive by default |
Estimated monthly cost: Essentially $0 at low traffic. Each query costs approximately $0.003–$0.006 in OpenAI API calls (dominated by gpt-4o generation; grading with gpt-4o-mini is cheap). At 1,000 queries/month, expect ~$3–$6.
Stack B — Production-Ready (designed to scale)
Layer | Technology | Why |
Framework | Next.js 14 | Same DX, scales to edge/serverless |
Language | TypeScript 5 | Required for LangGraph type safety |
Orchestration | LangGraph 0.2.x | Same graph, different data layer |
Vector DB | Pinecone or Postgres pgvector | Millions of docs, metadata filtering |
Embeddings | text-embedding-3-large | Higher-dimensional, better retrieval |
Generation | OpenAI gpt-4o | Quality answer generation |
Grading | OpenAI gpt-4o-mini | Cost-efficient judge at scale |
Caching | Redis (Upstash) | Cache embedding lookups + answers |
Streaming | SSE → consider WebSocket at scale | Full-duplex + reconnect logic |
Observability | LangSmith + structured logging | Trace every graph run, debug graders |
Deployment | Vercel + managed DB | Instant deploys, no infra ops |
Auth | NextAuth or Clerk | Per-user namespaces, rate limiting |
Estimated monthly cost: $20–$80/month infrastructure (Pinecone Starter: $0, Upstash Redis: ~$0–$10, Vercel Pro: $20), plus OpenAI API usage proportional to query volume.
Implementation Phases
Phase 1: Local Vector Store & Ingestion Pipeline
The foundation of the entire system is the offline ingestion pipeline. You will write a scripts/ingest.ts file (run via tsx with npm run ingest) that reads every markdown file from /docs, splits each into ~800-character overlapping chunks at natural markdown and code-fence boundaries using LangChain's RecursiveCharacterTextSplitter, calls the OpenAI text-embedding-3-small API in batches, and writes the result as a JSON array to data/vectors.json.
Key decisions at this phase: chunk size and overlap (800 / 150 is the tested sweet spot for technical docs with code blocks — too small and you lose context, too large and you dilute the embedding signal), whether to store metadata like hasCode (useful for search weighting), and how to handle re-ingestion (overwrite vs. incremental).
The runtime retrieval function is equally important: for each query, embed the query text, compute cosine similarity against every stored vector, sort descending, and return the top-K results. This needs to be fast enough to run on every query, which it easily is for a few thousand documents.
The exact chunk-boundary logic and the batched embedding pattern that avoids OpenAI's rate limits are covered in detail in the full course with working, tested code.
Phase 2: LLM-as-Judge Graders with Zod Structured Output
Before you build the graph, build and test the two grader functions in isolation: the document relevance grader and the answer faithfulness grader.
Both graders follow the same pattern. You define a Zod schema for the expected output, pass it to llm.withStructuredOutput(schema), and call the resulting chain at temperature 0. The document grader takes a query and a document and returns { score: number, relevant: boolean, reasoning: string }. The faithfulness grader takes the generated answer and the source documents and returns { faithfulnessScore: number, hallucinationRisk: "low" | "medium" | "high", problematicClaims: string[], refinementSuggestion?: string }.
The critical decisions here: which model to use for grading (gpt-4o-mini is sufficient and 10× cheaper than gpt-4o), what temperature to use (0 — you want deterministic judgments, not creative ones), and how to calibrate the relevance threshold. Setting RAG_RELEVANCE_THRESHOLD = 0.75 means a document must score 0.75/1.0 to be included as context. Too high and you'll over-broaden; too low and you'll include noise.
The prompt engineering for the faithfulness grader — and the specific Zod schemas that produce reliable structured output from gpt-4o-mini — are covered in detail in the full course with working, tested code.
Phase 3: The LangGraph State Machine
With the data layer and graders working, you assemble the state machine. Define the RAGState type with LangGraph channels: query, originalQuery, documents, filteredDocuments, generation, iterationCount, correctionCount. Each channel gets a reducer — most are simple replace reducers, but iterationCount and correctionCount use add-reducers.
Then define and chain the six nodes. Each node receives the current RAGState and returns a Partial<RAGState>. The conditional edges are functions that inspect state and return a node name. The after_grade_docs edge returns "generate" if enough documents passed, or "broaden_query" if iterationCount < 3, or forces "generate" on the final iteration. The after_grade_answer edge returns "done" if faithfulnessScore >= 0.80 or "refine_answer" otherwise.
The key implementation trap: all .addNode() and .addConditionalEdges() calls must be chained on a single builder expression. Splitting them across multiple const assignments will compile locally but fail next build. The type system is doing real work here — treat it as a feature, not a bug.
The complete typed state definition, the chained builder pattern that passes next build, and the conditional edge logic that handles all four termination scenarios are covered in the full course with working, tested code.
Phase 4: SSE Streaming API and React UI
The Next.js API route wraps the LangGraph graph in an SSE response. You set up the response with Content-Type: text/event-stream headers, create an EventEmitter, pass the emitter into the graph as a config parameter, and for each node that emits a trace event, call emitter.emit("trace", data). A loop that calls graph.stream(input) yields state snapshots; each snapshot triggers the appropriate SSE frame.
The React UI opens a fetch-based SSE reader (response.body.getReader()), accumulates bytes from the stream, decodes them, and parses each data: line as a JSON event. Rendering is split into two panels: the thought trace (colour-coded by event type — retrieve is blue, grade is orange, broaden is yellow, generate is green) and the final answer (with a faithfulness badge that shows green for faithful, amber for medium risk, and a refine_answer tag if a refinement was applied).
The SSE chunk-buffering code and the React SSE consumer that correctly handles partial lines and keepalive comments are covered in the full course with working, tested code.
Phase 5: Tune, Harden, and Deploy
Once the happy path works end-to-end, you harden the system for production. This means running a suite of queries against your docs and evaluating results across three dimensions: retrieval quality (did the right docs come back?), faithfulness accuracy (did the grader correctly identify hallucinations?), and latency (end-to-end query time should be 10–15 seconds on the happy path).
Key tuning levers: RAG_TOP_K (the single biggest cost/latency lever — reducing from 10 to 5 cuts grading calls in half), RAG_RELEVANCE_THRESHOLD, and RAG_ANSWER_CONFIDENCE_THRESHOLD. For deployment to Vercel, you must commit data/vectors.json to the repository — Vercel's serverless filesystem is read-only at runtime, so the vector store must be bundled with the deployment.
The Vercel deployment checklist, the tuning methodology, and the production build hardening steps are covered in the full course with working, tested code.
Common Challenges (and How to Solve Them)
Building a self-correcting RAG system with LangGraph and Next.js involves several non-obvious failure modes that cost hours to debug. Here are the ones that will bite you.
1. next build fails with StateGraph type errors. The error reads something like Type 'retrieve' is not assignable to type '__start__'. Root cause: LangGraph's TypeScript builder narrows the graph's state type through each chained .addNode() return value. If you assign each call to a separate const, the type information is lost. Fix: chain all .addNode(), .addEdge(), and .addConditionalEdges() calls in a single expression.
2. Build and health check crash without an API key. Root cause: new ChatOpenAI({ modelName: "gpt-4o" }) at module-load time throws immediately if OPENAI_API_KEY is not set — which it isn't during next build's static page data collection. Fix: initialise LLM clients lazily, on first use, inside a function. The GET /api/rag health check returns 200 even in CI with no key.
3. The grader LLM returns free text instead of structured JSON. Root cause: without explicit constraints, even at temperature 0, some model responses don't conform to the expected schema. Fix: use llm.withStructuredOutput(zodSchema) with a strict Zod schema. LangChain will automatically retry with a corrective prompt if the first output fails validation.
4. The correction loop runs forever (or too expensively). Root cause: a self-correcting system without termination caps can loop indefinitely if the graders are calibrated too harshly. Fix: hard cap retrieval iterations at 3 (force-pass on the third), and cap answer refinement at exactly 1 pass. These caps must be checked in the conditional edges, not in the nodes themselves.
5. The browser receives garbled SSE events. Root cause: TCP and HTTP/2 framing means network chunks don't align to SSE message boundaries. A single data: line may arrive split across multiple read() calls, or multiple lines may arrive in one chunk. Fix: maintain a string buffer, split on \n, hold back any partial line (no trailing \n) until the next chunk, and skip lines that start with : (keepalive comments).
6. The system returns a confident answer when no relevant documents exist. Root cause: without an explicit check, gpt-4o will synthesise an answer from whatever is passed to it, even if all documents failed grading. Fix: check filteredDocuments.length === 0 in the generate node and return a hardcoded fallback: "I could not find relevant documentation to answer this question. Please try rephrasing or check whether the relevant docs have been ingested.".
7. The vector store is missing in production. Root cause: data/vectors.json is generated by a local script and is typically in .gitignore. Vercel's serverless runtime is read-only — you cannot run the ingestion script at deploy time. Fix: commit vectors.json to the repository (it's just JSON), or migrate the vector store to Pinecone or pgvector and run ingestion as a separate CI step.
Solving these issues took us over 30 hours of testing and iteration — the full course walks you through each fix with working code.
Ready to Build This Yourself?
Understanding an architecture and shipping a working, deployed application are two different things. The course gives you everything you need to go from zero to a production-deployed self-correcting RAG system — without hitting the walls described above.
What's included in the course:
✅ Full source code — starter scaffold AND complete final implementation
✅ 6 modules, 24 step-by-step lessons (video + written)
✅ The complete LangGraph state machine with all 6 nodes and conditional edges
✅ Local JSON vector store + TypeScript ingestion pipeline
✅ Zod-based document relevance and answer faithfulness graders
✅ SSE streaming API and React reasoning-trace UI
✅ Vercel deployment walkthrough (including the vectors.json gotcha)
✅ Threshold tuning guide and production hardening checklist
✅ Lifetime access to updates and all future lesson additions
✅ Codersarts community support
$29.99. Everything above.
Want a faster path? Book a 1:1 guided code-review session ($99.99) — a live session with a Codersarts engineer who will walk through your build, debug your implementation, and guide you on extending the graph with re-rankers, citations, or multi-query fan-out. Book the 1:1 session →
Conclusion
Naive RAG fails not because retrieval or generation are hard, but because the system has no feedback loop: it cannot observe its own mistakes. Self-correcting RAG adds two LLM-as-judge checkpoints — one on retrieval quality, one on answer faithfulness — and a looping state machine that acts on those judgments: broadening the search when documents are weak, refining or refusing the answer when claims are unsupported.
The simplest viable stack to get started is a local JSON vector store with cosine similarity search, gpt-4o-mini for grading, and gpt-4o for generation — all in a single Next.js 14 project. You can have it running locally in a weekend. The same LangGraph state machine scales to a hosted vector database when your document set grows.
If you want to ship this rather than just understand it, the Self-Correcting RAG course on Codersarts Labs has the tested code, the working deployment, and the tuned configuration — for $29.99.
Comments