Start free

Zero to FDE / Blog / What is RAG? Retrieval-augmented generation, explained with working code

Blog

What is RAG? Retrieval-augmented generation, explained with working code

By Venkata Vishnu Vardhan Raju Chakravarthula · · 8 min read

Retrieval-augmented generation (RAG) means: when a question comes in, first find the most relevant passages in your own documents, then give those passages to the language model together with the question. The answer is grounded in your documents, and it can say which document it came from.

Why RAG exists

A language model only knows what was in its training data. It knows nothing about your company’s policies, your product manual or last week’s tickets. You have three options:

OptionWhat it doesThe catch
Paste every document into the promptWorks for a few pagesToo long and too expensive once you have hundreds of documents
Fine-tune the modelChanges the model’s behaviour and styleSlow and costly to update, and it cannot cite where an answer came from
RAGFetches only the few relevant passages per questionAnswers are only as good as the retrieval

For “answer questions from these documents”, RAG is usually the right first choice: documents can change daily without retraining anything, and every answer can point to its source.

The five steps

  1. Chunk. Split documents into passages small enough to be specific, often a few hundred words, sometimes with a little overlap.
  2. Embed. Turn each chunk into a vector with an embedding model. Texts with similar meaning end up with vectors that point in similar directions.
  3. Store. Keep the vectors, along with each chunk’s text and source, in a vector index or database.
  4. Retrieve. Embed the question the same way and find the chunks whose vectors are closest to it (cosine similarity).
  5. Generate. Put those chunks in the prompt with their source names and ask the model to answer only from them, citing the source.

A working example

This uses the open-source sentence-transformers library for embeddings and NumPy for the search (pip install sentence-transformers numpy). The final prompt can go to any LLM API.

import numpy as np
from sentence_transformers import SentenceTransformer

docs = {
    "refund-policy.md": "You can get a full refund within 7 days of payment. Email support with your payment ID.",
    "access.md": "Access is digital and unlocks as soon as the payment is confirmed.",
    "devices.md": "One account can be signed in on up to 3 devices at a time.",
}

# 1. Chunk. These documents are short, so each one is a single chunk.
chunks = list(docs.items())

# 2. Embed every chunk once, ahead of time, and keep the vectors.
model = SentenceTransformer("all-MiniLM-L6-v2")
vectors = model.encode([text for _, text in chunks], normalize_embeddings=True)

def retrieve(question, k=2):
    # 3. Embed the question the same way.
    q = model.encode([question], normalize_embeddings=True)[0]
    # 4. Score every chunk. With normalised vectors, a dot product is cosine similarity.
    scores = vectors @ q
    best = np.argsort(scores)[::-1][:k]
    return [(chunks[i][0], chunks[i][1], float(scores[i])) for i in best]

question = "Can I get my money back?"
hits = retrieve(question)

# 5. Put the retrieved chunks in the prompt, with their names, and ask for citations.
context = "\n\n".join(f"[{name}] {text}" for name, text, _ in hits)
prompt = (
    "Answer using only the sources below. Cite the source name in square brackets. "
    "If the answer is not in the sources, say you don't know.\n\n"
    f"Sources:\n{context}\n\nQuestion: {question}"
)
print(prompt)  # send this to any LLM API

Notice the question never says “refund”. Keyword search would match nothing; embeddings match on meaning, so the refund policy should rank first. Run it and print hits to see the scores.

Where RAG usually breaks, and the fix

  • Chunks of the wrong size. Too big and the right sentence is buried; too small and the context needed to answer is cut off. Test a few sizes on real questions.
  • Exact terms get missed. Embeddings are good at meaning and weaker at exact strings such as product codes, error IDs and names. Combining keyword search (such as BM25) with embedding search, called hybrid search, covers both.
  • The right chunk is retrieved but ranked low. A reranker re-scores the top few dozen results more carefully before you pick the final ones.
  • Nobody measures it. Write down real questions with the chunk that should answer each, and track how often that chunk appears in the top results (recall at k). Change one thing at a time and re-measure.

RAG in real client work

In a real deployment, the documents are messy: scanned PDFs, tables, duplicates and outdated versions. Some users must not see some documents, so retrieval has to respect permissions. And people only trust the answers if every claim links back to a source. That is the everyday work of a Forward Deployed Engineer.

Zero to FDE spends Week 3 building RAG from first principles: a document Q&A tool over a real PDF corpus that cites the exact source chunk it used. Start with Week 1, free, or see the full roadmap.

Zero to FDE is a 90-day path from an AIML degree to the work Forward Deployed Engineers do: backend basics, LLMs, RAG, agents, evaluation and deployment, with a shipped deliverable every week. Week 1 is free and complete, no card needed.

Start Week 1 free

More from the blog