These are my notes on RAG. I wanted to follow the data flow and connect three parts of the process: how a document enters a knowledge base, how a question finds relevant material, and how that material supports an answer.
RAG is often drawn as “a retriever plus a large language model.” Following that main path brings up many practical questions: how to parse and chunk documents, what embeddings and vector databases each do, why queries need rewriting, how hybrid retrieval and reranking work together, and how to judge whether the final answer and its citations are reliable.
The article first establishes the complete workflow, then examines each stage before returning to the system as a whole through evaluation and open-source implementations. If you are working through these mechanisms too, I hope you will read along and share your understanding and practical experience.
1 Why RAG Is Needed: Answering from External Knowledge
Knowledge sources: what the model knows and what the task needs
A model can answer questions using the knowledge in its parameters, but the material a task needs may be absent: internal organizational documents, continually updated product instructions, data a user is authorized to access, or rules that need to be checked individually against the original text.
Even when a model generates a plausible answer, we still need to know what supports it. For tasks that require external material, a system can retrieve relevant content first, then give it to the model together with the question. RAG, or Retrieval-Augmented Generation, brings retrieved material into the generation process.
The original RAG paper by Lewis et al. discusses combining parametric knowledge with external, non-parametric knowledge. RAG systems in practice can use different retrievers, indexes, and generation interfaces. Implementing retrieval augmentation does not require adopting every detail of the paper’s training and decoding methods.
External knowledge also gains a maintenance path of its own. When material changes, the corresponding content and indexes can be reprocessed so that later retrieval uses the updated material. An ordinary retrieval operation does not automatically write documents into model parameters. The model still answers from evidence in its current input.
When to use it: working with long context and fine-tuning
If there are only a few short documents, placing them directly into context may be enough. When material is extensive, changes frequently, or has different visibility for different users, retrieving first helps control how much content each request must process.
Long context expands how much material can be included in the input; retrieval selects which material is worth including. The two can be combined. Even with a sufficiently large window, relevance, duplication, cost, and the model’s use of evidence still matter. Lost in the Middle studies how the position of relevant information affects answers, suggesting that we should evaluate how context is actually used rather than considering window size alone.
Fine-tuning adjusts model parameters through training and can improve task behavior, expression, or specific capabilities. RAG supplies external material for answers and makes it easier to associate them with sources. A system can use both a fine-tuned model and a retrieval workflow; the choice depends on what the problem requires.
RAG provides a mechanism for answering from external knowledge. Reliability still depends on the stages that follow. Sources can be outdated, retrieval can miss conditions, and generation can misuse evidence. These issues need attention throughout the workflow.
2 The Overall Workflow: Connecting Knowledge Ingestion and Online Question Answering
Two pipelines: preparing material and using it
RAG can first be understood as two pipelines.
The knowledge ingestion pipeline processes source material: reading files or data sources, parsing structure, cleaning noise, creating chunks, generating the required representations, and writing them to searchable storage. The online question-answering pipeline processes the user’s question: preparing retrieval requests, obtaining candidates, reranking when needed, assembling context, and generating an answer with citations.

Original workflow diagram: the index connects the ingestion and question-answering pipelines. Query rewriting, hybrid retrieval, and reranking are selected according to the task. Generation needs both the original question and the assembled evidence.
“Offline ingestion” means that this preparation does not have to be repeated for every question. It can run as a batch job or incrementally when material is added or changed; it is not limited to the period before a system goes live.
Aymeric Roucher’s Advanced RAG Cookbook demonstrates a workflow from chunking through retrieval to generation using a real document collection. When reading such implementations, first identify each step’s inputs and outputs, then consider whether its optional optimizations suit your own material.
Intermediate objects: documents, chunks, candidates, and evidence
The objects flowing through these pipelines also change along the way.
Original documents carry layout and source information. Parsing turns them into structures that programs can process, and chunking creates units suitable for retrieval. Retrieval returns candidate content with scores or rankings. Only after filtering and context assembly does that content become the evidence the model actually reads.
The associations between these objects need to be preserved. A chunk should lead back to its document and original location. A candidate should provide access to its text or corresponding reference. Source identifiers in an answer need to map to the evidence actually used. Keeping only a set of vectors makes it difficult to display source text, update indexes, and check citations later.
Access boundaries also need to be enforced throughout the data flow. Record permission information during ingestion, constrain retrieval using an identity verified on the server, and preserve the same boundary when expanding parent chunks or reading source text. Azure AI Search’s document-level access control guide provides one implementation, but the permission mechanisms supported by different storage systems need to be checked individually.
3 Knowledge Ingestion: Parsing, Chunking, and Vector Indexing
Document parsing: preserving the relationships within content
Information in PDFs, web pages, Markdown, and spreadsheets often depends on structure for its meaning. Table cells need their column headers, a footnote may qualify an entire rule, and a heading establishes the topic of the content below it.
Parsing first needs to turn these relationships into a representation that later programs can use. Cleaning can remove repeated headers, navigation, and meaningless whitespace, but should preserve units, negation, conditions, and locations in the source text. If OCR or reading order is wrong, an embedding cannot recover the original relationships that have already been lost simply through semantic representation.
Document parsing can use tools such as MinerU. It can convert documents such as PDFs into Markdown or JSON, combining layout analysis and content recognition to handle text, tables, and formulas and provide input for subsequent cleaning and chunking.
DoclingDocument is a structured document representation. It organizes body text, headings, tables, and hierarchical relationships while associating them with source locations. The official diagram below shows how document content corresponds to the structured representation.

Source: The Docling Contributors, DoclingDocument, MIT license. The original image is unchanged. The structure on the left corresponds to the source document on the right, showing headings, paragraphs, and parent-child relationships.
After parsing, it is worth inspecting representative intermediate results: are headings still associated with their content, have tables lost columns, and can footnotes still be located? A successful import establishes only that processing finished; it does not establish that the material is ready for retrieval.
Chunking strategies: balancing retrieval units and reading units
Chunking divides longer content into retrieval units. If chunks are too small, definitions may become separated from their conditions. If they are too large, relevant content can be mixed with unrelated paragraphs and consume more context when retrieved.
Strategies can be combined. What matters is being clear about the problem each one addresses:
| Strategy | How it organizes content | Trade-offs to consider |
|---|---|---|
| Structural chunking | Splits along boundaries such as headings, paragraphs, and tables | Helps preserve document organization, but section lengths may not suit model input |
| Recursive chunking | Starts with larger boundaries, then progressively splits content that is too long | Requires suitable separators and a way to measure length; tables or code may still be split apart |
| Overlapping windows | Retains some repeated content between adjacent chunks | Reduces fragmentation at boundaries but increases storage, duplicate candidates, and context consumption |
| Semantic chunking | Finds breakpoints using similarity between representations of adjacent sentences | Depends on sentence segmentation, the model, and thresholds; it cannot guarantee that business conditions remain complete |
| Parent-child relationships | Uses smaller nodes to locate material, then follows relationships to retrieve more complete content | Retrieval units and reading units can differ; expansion still needs permission and budget constraints |
Docling’s chunking guide shows how content can be subdivided and merged on top of document structure according to tokenizer limits. LlamaIndex Node Parsers provide approaches including sentence windows, semantic splitting, and hierarchical nodes. Forming chunks from document headings and adding context through parent-child relationships after retrieval are two designs that can be combined.
Length also needs to be calculated from the actual input. Character counts are not token counts, particularly for Chinese. If titles or other context are added alongside body text in the embedding input, they count toward the model’s input limit too. No fixed chunk size or overlap suits every corpus; evaluate them against the questions the system will need to answer.
Metadata: making content locatable, filterable, and updatable
Along with its text, a chunk usually needs a stable identifier, a source location, its document association, and an access scope. The following illustrative structure explains these relationships; it is not a fixed interface from any particular framework:
{
"chunk_id": "manual/setup/02",
"document_id": "manual",
"parent_id": "manual/setup",
"text": "The body text of this chunk...",
"source": {
"uri": "docs/manual",
"section": "Installation and configuration",
"page": 12
},
"access_scope": ["team-a"]
}These fields serve different stages. Document identifiers help locate chunks that need updating. Sections and page numbers help readers return to the source. Parent relationships support expanding the reading scope, while access scope is used by the actual permission enforcement mechanism.
Deduplication also needs to consider these relationships. Two documents with identical content may have different sources or permissions. Discarding one without preserving its associations may remove the basis for later citations or access decisions.
Embeddings and vector databases: representation, storage, and querying
An embedding model encodes text as vectors, and a retrieval system uses relationships between those vectors to find candidates. This provides a representation; similarity should not be interpreted directly as factual reliability or the probability that an answer is correct.
Queries and documents need matching encoding methods within a compatible retrieval space. This could involve different input modes of the same model or jointly trained query and document encoders. Identical vector dimensions alone do not establish that the representations can be compared.
A vector database or database extension stores vectors, body text or retrievable references, and metadata, and implements indexing, filtering, and querying. The interaction between exact search, approximate indexes, and filters affects retrieval quality and latency. Approximate retrieval also needs to be checked for sufficient candidates after filtering; performance without filters is not enough.
| Option | Organization worth examining | Boundary within RAG |
|---|---|---|
| pgvector | Stores vectors in PostgreSQL alongside relational data and SQL queries | Indexes and query plans still need maintenance; the extension does not parse documents or generate answers |
| Qdrant | Vectors, payload filtering, and multistage queries | The service supports organizing data and candidates; the application still needs to enforce business permissions and assess evidence |
| Milvus | Deployment options including local and server-based forms | Deployment affects operations and available capabilities; vector counts alone cannot establish capacity |
Start with existing data systems, query patterns, filtering needs, and operational conditions before comparing implementations. Databases, embedding models, and generation models have different responsibilities. Replacing one does not automatically fix problems in the others.
Updates and migration: keeping the index aligned with the source
When material changes, the system should be able to locate affected old chunks, parse, split, or encode the content again, and replace or invalidate outdated content. Deleting a document also requires handling its index entries so that withdrawn material does not continue contributing to answers.
Changing the embedding approach usually requires re-encoding the existing corpus. Qdrant’s migration guide demonstrates gradual migration using a new collection or another set of vectors. Whatever the organization, queries must use compatible representations. Replacing only the query encoder while continuing to compare against old document vectors requires confirmed compatibility.
The identifier relationships between source documents, parsed results, and indexes make updates and migration possible. They also let later investigation return to specific material rather than merely count how many vectors remain in the database.
4 Queries and Retrieval: Rewriting, Hybrid Retrieval, and Reranking
Query preparation: expressing a question as a suitable retrieval request
A user’s natural-language question may not be ready for direct retrieval. In a conversation, “what about that limit?” needs its reference resolved. A question combining several concepts may also need different retrieval expressions.
Query rewriting can use known context to produce a more explicit request, but the original question should be preserved. Rewriting changes how to search; it should not introduce an unverified answer or unknown conditions. Access identity and permission scope should also come from the server rather than the rewriting model. Query Rewriting in Retrieval-Augmented LLMs discusses the role of improving query expressions before retrieval.
Multi-query retrieval generates a small number of alternative expressions for the same information need, or searches several aspects separately when that is necessary, then deduplicates and fuses the candidates. More queries bring additional retrieval cost and noise. Their value depends on whether they find evidence that was previously missing.
Hybrid retrieval: complementary candidates from terms and vectors
Term-based retrieval methods such as BM25 use signals including query terms, term frequency, and document length. Dense retrieval compares vector representations of the query and content. The former may be better at retaining error codes, parameter names, and specialized terminology, while the latter can help connect different ways of expressing the same idea.
Actual results still depend on the tokenizer, model, and corpus. Term-based retrieval does not guarantee character-for-character matching of every string, and vector retrieval does not guarantee that semantically similar content applies to the current question. Hybrid retrieval brings together candidates found through different signals, increasing the opportunity to obtain relevant material.
Every retrieval path should respect the same access constraints and known scope. If the question already establishes the target or document scope, filters can be applied at query time. Unknown conditions should instead be clarified or differences preserved, rather than guessing a condition and retrieving only material that matches it.
Fusion and reranking: using ranks or reading the content again
Raw scores from different retrievers often use different scales. RRF, or Reciprocal Rank Fusion, sums contributions calculated from each result list’s ranks:
Here, is the -th result list, and the sum runs over lists containing candidate . The fusion constant is , and is the candidate’s rank in that list. The constant and the number of candidates ultimately retained are different parameters. Rank origins and implementation details also need to be consistent. RRF does not require term-based scores and vector similarities to be interpreted as the same kind of probability. Original RRF paper
Fusion changes the candidate order without rereading each passage. Cross-encoder reranking feeds the question and a candidate into a model together, using their interaction to calculate relevance and select more suitable material. Since processing each pair has a cost, systems usually retrieve candidates first and rerank a smaller set.

Source: Sentence Transformers, Retrieve & Re-Rank, Apache-2.0 license. The original image is unchanged. The diagram first retrieves candidates with a bi-encoder, then reranks them with a cross-encoder.
This also shows the boundary between the two mechanisms: RRF merges existing rankings, while a cross-encoder rescores candidates based on their text. They can be combined or selected according to the task. Material that never entered the candidate set will not appear simply because reranking was added. Qdrant’s multistage queries provide concrete interfaces for candidate prefetching, fusion, and subsequent processing.
Some methods add information earlier, during ingestion. Anthropic’s Contextual Retrieval adds context to chunks before building their retrieval representations, combines term-based and vector retrieval, and can incorporate reranking. This happens at a different stage from rewriting queries during question answering. Whether it is useful depends on what has been lost: the chunk’s context or a suitable retrieval expression of the question.
5 Evidence and Generation: From Candidate Content to Checkable Citations
Evidence assembly: giving the model enough material to support its conclusions
Relevant candidates are not necessarily sufficient to answer a question. Several overlapping chunks may repeat one fact while a condition that limits the conclusion lies in another paragraph.
Evidence assembly needs to remove duplication, select content covering key conditions, add parent chunks or neighboring material when necessary, and arrange the reading order within the context budget. The original question, task instructions, and output also need space. All retrieved content cannot simply be concatenated unchanged.
Compression and summaries must likewise preserve qualifications that determine the conclusion. If a rule applies only to a particular type of object, removing that scope before giving it to the model may produce an inapplicable answer even when generation remains faithful. When sources conflict, the evidence package should preserve their respective origins and conditions so that the model can explain the differences.
Citation relationships: from source locations to specific claims
An application can assign evidence identifiers to the material actually used and maintain a mapping back to the source text. The model sees passages and their identifiers, connects specific claims to the corresponding evidence in its answer, and lets readers follow the mapping back to the document and location.

Original relationship diagram: connections show which claim a piece of evidence supports, rather than execution order. A claim may require several pieces of evidence. When support is missing, the system should search further, seek clarification, or leave the gap explicit.
At least three things need to be distinguished: whether a link exists, whether it points to material actually used, and whether that material truly supports the corresponding claim. Appending a list of links to the end of an answer cannot replace the last judgment.
The ALCE study by Gao et al. evaluates answer correctness separately from citation quality, including whether claims have sufficient citation support and whether citations are irrelevant. Citations help readers check an answer, but they do not automatically turn generated content into fact.
Generation boundaries: explain only what the material supports
The generation stage gives the model both the question and the evidence, asking it to preserve key conditions, corresponding sources, and points that remain unconfirmed. If the evidence is insufficient, the system can retrieve more, ask the user for clarification, or provide an answer with an explicit scope. Behavior absent from the material should not be presented as a definite rule based only on common practice.
Faithfulness to context also needs to be separated from correctness about the real world: a model can accurately repeat an erroneous or outdated source. Source provenance, applicability, and content conflicts still need to be assessed during retrieval and evidence processing.
External material may also contain operational instructions unrelated to the task. Entering context as retrieved content should not promote them to host instructions. Citations and source locations help preserve the boundaries of the material; the system still enforces actual permissions and execution rules.
6 Evaluation and Optimization: Tracing Answer Errors Through the Data Flow
Evaluation benchmarks: establishing the questions and evidence first
Evaluating RAG requires a set of questions that can be traced back to source text. Each example can record the question, whether it is answerable, the expected key facts, and the documents and evidence spans that support those facts.
Annotations should first be associated with source passages or locations, then mapped to chunks. Binding the correct evidence directly to a particular chunk ID can change the meaning of labels whenever chunking changes, making fair comparisons difficult.
The question set should cover actual information needs, including exact terms, alternative wording, multiple pieces of evidence, conflicting sources, and insufficient material. Synthetic questions can extend coverage, but real questions and human-reviewed examples are still needed to avoid evaluating only questions the system generates itself and finds easy to answer.
Layered metrics: relevance, correctness, faithfulness, and citations
LlamaIndex’s evaluation guide separates retrieval evaluation from answer evaluation. This division helps identify which stage a change affects:
| Observation point | Main question | Metrics or checks |
|---|---|---|
| Source text to index | Did all required material enter the index intact? | Inspect parsing and chunking for conditions, structure, and source locations |
| Candidate retrieval | Does the correct material appear among the top k candidates? | Recall@k and Hit@k, with relevant material and evidence coverage defined |
| Candidate ranking | Do relevant results appear near the top? | MRR and, when needed, ranking metrics such as NDCG |
| Reranking and evidence assembly | Do the correct candidates reach the final context? | Inspect the actual evidence package after reranking, deduplication, scope expansion, and budget trimming |
| Answer correctness | Are the key conclusions correct and applicable? | Compare against expected facts, source text, and the question’s conditions |
| Faithfulness | Does the supplied context support the claims in the answer? | Break the answer into claims and check the evidence; this is not independent fact verification |
| Citation quality | Do key claims have sufficient, relevant citations? | Check the extent of support and identify irrelevant citations |
| Cost and latency | What does the quality improvement cost? | Record ingestion and update costs separately from online retrieval, reranking, and generation |
Recall@k measures how much of the annotated relevant material is covered by the top k results. Hit@k checks only whether at least one relevant item is retrieved, then averages across questions. MRR takes the reciprocal rank of the first relevant result for each question, assigns 0 when none is found, and averages the values. Its focus on the first relevant result means that it cannot, on its own, establish whether all the necessary evidence has been retrieved.
When comparing metrics, keep fixed whether the relevant objects are documents or chunks, as well as k and the matching rules. How the recall denominator is defined and how many pieces of material a question requires also affect the meaning of a score. The Retrieval Evaluation example helps explain metric inputs, but application-specific questions still need their own evidence annotations.
RAGAS separates dimensions such as faithfulness and uses models to assist evaluation. Judging support from the supplied context cannot replace checking external factual correctness. Different evaluation dimensions also require different reference answers and annotations. Model judges are useful for assisting bulk screening, while important examples should still be checked against source text and human benchmarks.
Diagnosing failures: inspect intermediate results before deciding what to change
When an answer omits a condition, follow the data flow backward.
If the condition exists in the source but not in the parsed result, address parsing first. If parsing preserved it but chunking separated it from the relevant content, inspect splitting and associations. If the correct content is indexed but absent from the candidates, examine queries, filters, representations, and retrieval. If candidates contain it but the evidence package does not, focus on reranking, deduplication, and budgets. If the evidence package is complete but the answer omits a qualification, inspect generation and citation handling.
This connects problems to specific intermediate artifacts rather than starting every investigation by replacing the model or framework. Compare approaches using the same question set and a change with a clear purpose, recording quality, latency, and cost before deciding whether to keep it.
7 Implementation Observations: From Pipeline Frameworks to Graph Indexes
LlamaIndex: connecting nodes, retrieval, and response synthesis
LlamaIndex processes documents into nodes, retrieves them, optionally applies postprocessing, and passes them to a response synthesizer. Following these objects reveals where the chunking in Chapter 3, retrieval and reranking in Chapter 4, and evidence synthesis in Chapter 5 are implemented. VectorStoreIndex
Its Ingestion Pipeline can combine transformations, metadata processing, and embeddings. With a document store (docstore) configured, it can also use document identifiers and content hashes to detect changes, then update the corresponding vector store. Stable IDs are only the foundation. Document management must actually be enabled and connected to handle index changes on that basis.
Nodes can retain parent-child relationships, and retrievers and postprocessors can use them to fetch or merge context. The framework provides ways to compose these parts, but the specific chunking, candidate scope, and synthesis approach still need evaluation against the material and questions.
Haystack: organizing both pipelines with explicit components
Haystack Pipelines connect data flows through components and their inputs and outputs. An indexing pipeline can organize conversion, preprocessing, embeddings, and document writes; a query pipeline connects query processing, retrieval, prompt construction, and generation.
This organization makes each step’s inputs and outputs easier to inspect independently and supports branches and combinations. When quality problems arise, developers can examine what the retrieval component returned or which material ultimately reached the generation component before deciding where to make changes.
Evaluation also distinguishes component evaluation from end-to-end evaluation. Successful component execution establishes only that the agreed operation completed. Whether the whole pipeline covers the evidence and generates correct citations still requires a benchmark like the one discussed in Chapter 6.
GraphRAG: organizing material through entity relationships and community summaries
Microsoft GraphRAG adds another way to organize material beyond passage retrieval: it extracts entities and relationships from text, builds a graph, and organizes communities and summaries. For questions that require connections across sources or synthesis over the entire corpus, these intermediate representations can supply context different from locally similar passages.

Source: Darren Edge et al., Figure 1 of the original GraphRAG paper, CC BY 4.0. The complete figure area was extracted from the original paper, with its contents unchanged. It shows how graph indexes and community summaries contribute to global answers.
The query overview distinguishes local queries that organize context around entities and relationships from global queries that synthesize community reports. Global queries form candidate answers from community summaries, then aggregate content relevant to the question. This illustrates the approach of organizing a corpus first and then answering questions about it as a whole.
These structures also add costs for graph construction, summarization, and querying, and extraction and summarization can themselves introduce errors. When a task needs precise locations of clauses in the source, basic retrieval and citation chains remain important. Relationships in the graph cannot replace source evidence.
The official repository positions it as a research demonstration and notes that the project is primarily in maintenance mode. It is useful for understanding a method and a way of organizing information. Suitability for a real system still depends on question types, maintenance needs, and available resources.
When reading these implementations, keep tracing the associations between documents, nodes, candidates, and citations: how material becomes searchable objects, how a question selects material, and which evidence the generator ultimately reads. When an answer is wrong, the same data flow provides a path back to the stage that needs adjustment.

