How to Build a RAG System from Scratch in 2026 (Step-by-Step)

How do you build a RAG system?

You build it in eight stages: ingest your documents, chunk them, embed the chunks, store the vectors in an index, retrieve the relevant ones for a query, rerank them, generate a grounded answer, and evaluate the whole thing — then loop. The first seven get you a demo in an afternoon; the eighth, plus keeping the index fresh, is what separates a demo from production. This is application-layer AI-engineering work: RAG shows up in 26% of AI-engineer job descriptions but only 4% of AI-infrastructure ones (Dexity's live-JD scans), and "design a RAG system" is the most common system-design interview opener of 2026. Below, every stage with the decisions that actually matter.

Key facts

• A RAG pipeline has eight stages: ingest → chunk → embed → store → retrieve → rerank → generate → evaluate, split into an offline indexing phase and an online query phase.
• RAG appears in 26% of live AI-engineer JDs but just 4% of AI-infrastructure JDs (Dexity scans of 390 and 57 US postings) — it's an application-layer skill, not infra.
• Reciprocal Rank Fusion (RRF) — the standard way to merge keyword + vector results — scores each result 1 / (k + rank), with k = 60 from the original paper (Cormack, Clarke & Büttcher, SIGIR 2009).
• "Lost in the middle" is real: models use information at the start and end of the context far better than the middle (Liu et al., TACL 2024) — a reason to rerank and keep context tight, not stuff the window.
• Common practice sizes chunks at ~256–512 tokens for factoid content and ~1,024 for analytical content, with ~10–20% overlap — knobs to tune against your evals, not fixed laws.
• RAGAS evaluates RAG on four metrics: faithfulness and answer relevancy (generation), context precision and context recall (retrieval).

What is a RAG system, and why build one?

Retrieval-augmented generation gives an LLM the right context at answer time by retrieving relevant passages from your own data and putting them in the prompt. It's how you get an LLM to answer accurately about documents it was never trained on — without retraining it. The alternative approaches solve different problems:

The rule of thumb: RAG for knowledge, fine-tuning for behavior, long context for one-shot. Most production systems use RAG as the backbone and layer the others on only where they earn it.

What are the stages of a RAG pipeline?

Build it in that order, but expect to iterate: almost every quality problem traces back to retrieval, not generation.

Step 1 — How do you ingest and chunk your data?

Chunking is where most RAG quality is won or lost. Split documents into passages small enough to be precise but large enough to carry meaning, and attach metadata (source, section, date) to every chunk for filtering later.

Treat these as common practice, not fixed law — validate size and overlap against your own eval set. The bigger win is structure-aware chunking: split on headings and sections rather than a blind character count, so a chunk is a coherent idea, not a sentence sawn in half.

Step 2 — Which embedding model should you use?

The embedding model decides what "similar" means, so it caps your retrieval ceiling. As of 2026 roundups, the practical shortlist:

Prices and leaderboard positions drift — treat the MTEB leaderboard as a prior, not an oracle, and re-check current numbers before committing. In 2026 the open-vs-closed choice is usually operational (cost, latency, data residency), not pure accuracy.

Step 3 — Which vector database should you choose?

Strong consensus across 2026 roundups on a default hierarchy by scale and ops appetite:

The most common mistake is reaching for a specialized vector DB on day one. If you already run Postgres and have under ~10M vectors, pgvector gets you ~90% of the performance with none of the new infrastructure.

Step 4 — How does retrieval work (dense, sparse, and hybrid)?

Three retrieval styles, and you want the third:

• Dense (vector) — semantic similarity; great at meaning, weak at exact terms and rare tokens (names, IDs, error codes).
• Sparse (BM25 keyword) — exact-term matching; the opposite strengths.
• Hybrid — run both and fuse the results. In 2026 this is table stakes for production RAG.

The standard way to fuse two ranked lists is Reciprocal Rank Fusion: each document scores 1 / (k + rank) summed across the lists, with k = 60 from the original paper (Cormack et al., SIGIR 2009). Because RRF works on ranks, not raw scores, you don't have to normalize across systems — which is why it's the default hybrid fusion in OpenSearch, Elasticsearch, Azure AI Search, and Weaviate.

# Reciprocal Rank Fusion of several ranked lists
def rrf(ranked_lists, k=60):
    scores = {}
    for lst in ranked_lists:
        for rank, doc_id in enumerate(lst):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

Step 5 — Do you need reranking?

Usually, yes — practitioners repeatedly find reranking is the single biggest quality lift, ahead of switching to a bigger LLM. The pattern is retrieve wide, rerank narrow:

Over-retrieve to ~50 candidates, then let a cross-encoder (e.g. Cohere Rerank or an open-source sentence-transformers cross-encoder) score each against the query and keep the best handful. You spend a little latency to remove the near-misses that otherwise poison the answer.

Step 6 — How do you assemble context and generate the answer?

Put the reranked chunks into a grounded prompt that (a) instructs the model to answer only from the context, (b) requires citations, and (c) permits "I don't know." Then mind lost in the middle: models use information at the beginning and end of the context window far better than the middle (Liu et al., TACL 2024), even long-context ones. So put your best chunks at the edges and keep context tight rather than dumping fifty passages in.

Answer the question using ONLY the context below.
Cite sources as [1], [2]. If the answer isn't in the context, say you don't know.

Context:
{reranked_chunks}

Question: {query}

Step 7 — How do you evaluate a RAG system?

You cannot improve what you don't measure, and "it looks good" is not measurement. RAGAS gives four reference-free metrics that map cleanly onto the pipeline:

Build a golden set of ~50–100 question/answer pairs, re-run it after every pipeline change, and use an LLM-as-judge for continuous monitoring in production. A regression in faithfulness after a chunking tweak is exactly the signal this catches.

Step 8 — How do you take RAG to production (freshness, caching, cost)?

The demo assumes your data never changes. Production data changes constantly, so the hard problem becomes staleness. The robust pattern is a versioned, event-driven pipeline: every document update creates a new version (never overwrite embeddings in place); a change feed (CDC) re-chunks and re-embeds on each create/update/delete; retrieval filters to the latest valid version and ranks on similarity plus freshness; and high-risk queries fall back to a live lookup when a freshness SLA is breached.

Then add semantic caching (cut cost and latency on repeated queries) and monitor the metrics that actually predict RAG failure: retrieval freshness, stale-chunk rate, indexing lag (event → searchable), cache hit rate, and answer correctness after knowledge-base updates. (Deeper dive: designing RAG for frequently-changing data.)

What are the most common RAG failure modes (and fixes)?

The anti-patterns that sink interview answers and real systems: "just re-embed on a schedule," "just add a cache," overwrite embeddings in place, "increase the context window," and shipping with no retrieval metrics.

Advanced RAG techniques worth knowing

Once the base pipeline is solid, the high-leverage upgrades are query rewriting (expand/clarify the question before retrieval), metadata filtering (low-lift, high-impact — restrict by source/date/permission), and the full hybrid + rerank stack. Beyond that lie GraphRAG (retrieve over a knowledge graph), Self-RAG (the model decides when to retrieve), and agentic RAG (multi-step retrieve-reason loops) — reach for them when single-shot retrieval demonstrably isn't enough, not before.

Frequently asked questions

What chunk size should I use for RAG?

A common starting point is ~256–512 tokens for factoid/FAQ content and ~1,024 tokens for analytical content, with ~10–20% overlap — but treat these as tunable defaults and validate against your own eval set. Structure-aware chunking (by section) usually beats any fixed size.

Which vector database should I use for a first RAG project?

If you already run Postgres and have under ~10M vectors, use pgvector — it needs no new infrastructure and delivers most of the performance. Reach for Pinecone (managed) or Qdrant/Weaviate/Milvus (self-hosted, larger scale) when you outgrow it.

Do I really need a reranker?

Usually yes. Adding a cross-encoder reranker on top of hybrid retrieval is repeatedly reported as the single biggest quality improvement — often more than switching to a larger LLM.

RAG or fine-tuning?

RAG adds knowledge that changes; fine-tuning teaches style, format, or a skill. They solve different problems and are often combined — RAG for the facts, fine-tuning for the behavior.

How do I keep my RAG data fresh?

Use a versioned, event-driven (CDC) pipeline that re-embeds on document changes, filter retrieval to the latest valid version, rank on freshness as well as similarity, and fall back to a live lookup when a freshness SLA is breached — never just re-embed on a fixed schedule.

Is building a RAG system an infrastructure job?

No — it's application-layer AI-engineering work. RAG appears in 26% of AI-engineer JDs but only 4% of AI-infrastructure JDs in Dexity's scans.

Related reading

• Designing production RAG for data that changes — the freshness/staleness problem in depth, and why it's the #1 interview opener.
• The AI Engineer career path — where RAG sits in the role, and the rest of the skill map.
• The 2026 AI infrastructure stack — the serving/infra layer underneath application-layer RAG.

Build a production RAG system, not just a demo

The gap between a RAG demo and a RAG system is retrieval quality, evaluation, and freshness — the exact things this guide is about. Dexity's Ship Production Code with AI course has you build and evaluate a real retrieval system end-to-end, so you walk out with a project that survives contact with changing data — the thing every AI-engineer interview and job actually tests.

Sources: pipeline, chunking, vector-DB, and embedding practices reflect 2026 practitioner consensus (Redis, Meilisearch, Glukhov, vector-DB and embedding roundups). Verified references: Reciprocal Rank Fusion — Cormack, Clarke & Büttcher, SIGIR 2009; Lost in the Middle — Liu et al., TACL 2024; RAGAS metrics documentation. RAG/JD figures from Dexity's analysis of live US AI-engineer and AI-infrastructure job postings (2026); directional, US-only. Model prices and leaderboard positions move — confirm current figures. · Dexity.com