A comprehensive framework for evaluation of RAG Systems - Part 1

1. Introduction to RAG

An LLM based application can suffer from significant challenges of limited context, hallucinations (generating fake, fabricated information), and lack of ability to update itself. The most popular solution to deal with these issues has turned out to be the Retrieval Augmented Generation (RAG) architecture. In this blog, I have focused on how we can holistically and diligently evaluate these RAG applications. But first, a quick overview of the LLM application workflows with and without RAG to understand the value of RAG. Throughout this blog, I will use the example of a banking chatbot to discuss the core concepts.

1.1 How is the RAG workflow different from a traditional LLM-based workflow?

LLM Chatbot

Let’s consider the example of a customer service chatbot of a bank enabled by an LLM. Querying this chatbot invokes the following sequence of steps at a high-level:

1. Query generation – A user enters a query (e.g., How can I buy the product XYZ? What are the steps for an international wire transfer?) in the chatbot UI.
2. Query processing – Here, the query is tokenized (text decomposed into smaller meaningful units called tokens), applicable constraints (e.g., ensure not to give any investment advice or tax guidance) or information filters (e.g., do not respond if critical input fields are missing) are applied, and overall prompt is formatted via addition of system prompts with context to the prompt template (e.g., specifying the role of the chatbot “a helpful assistant”, key instructions to generate the responses “if you don’t know, then respond with “I don’t know””). This step is crucial to prepare the input for effective processing by the LLM.
3. LLM processing – The LLM accesses its internal knowledge representations to generate the query response (through neural pathway activation, weight adjustments across layers, and pattern recognition through the transformer architecture – More on this in another write-up). The LLMs work with their pre-trained knowledge base defined till a particular cut-off date.
4. Post-processing of response – The raw output is formatted, cleaned up, standardized and prepared for user consumption.
5. Response delivery – The formatted response is delivered to the end user. There are more nuances around error handling, session management, context retention that are not covered in this high-level overview but merit discussion when looking at this in more detail from an implementation perspective. While well-equipped to handle basic queries about the bank’s policies and processes, this workflow risks providing outdated information (since responses are based solely on training date limited to a prior cut-off date).

RAG-powered LLM Chatbot

Continuing with the same example of a customer service chatbot of a bank, let’s see how this workflow changes when the chatbot is additionally powered by RAG design.

1. Query generation – A user enters a query (e.g., How can I buy the product XYZ? What are the steps for an international wire transfer?) in the chatbot UI.
2. Embedding generation – An embedding model is used to convert the user query into a vector representation, the same one used to vectorize the knowledge base (external to the LLM such as bank’s product and policy documents) so that the reference documents are also stored using the same vector representation.
3. Vector search, document ranking, and document filtering – Semantic similarity matching is done to find documents most contextually relevant for the user query. This is followed by document ranking based on relevance score (by using re-ranking algorithms such as cross-encoder). Finally, document filtering (e.g., contextual filtering, metadata filtering, content-based) is performed to refine and curate the knowledge base retrieved for LLMs.
4. Retrieval (Single-hop or multi-hop) – Based on the query requirements, the retrieval might need only a single extraction of information (single-hop retrieval) to answer a direct, straightforward question or might involve multiple extractions (single-hop retrieval) from multiple sources for reasoning-based questions that need multiple pieces of evidence. This step returns multiple chunks of information based on the retrieval.
5. Context assembly – Relevant portions from retrieved documents are combined and put into a coherent context making this information more relevant for the LLM. Poor assembly can dilute the impact of high-quality outputs of effective retrievals.
6. Prompt construction – Retrieved context chunks, query, and response format instructions are combined to create the overall prompt crucial for effective processing by the LLM.
7. LLM processing – The well-structured prompt (query + top-most relevant documents + response format instructions) is now fed into the LLM that generates a contextually aware response integrating retrieved facts with its pretrained knowledge. Since the LLM is conditioned to generate responses based on this database only, this ensures that all responses are grounded in factual information only.
8. Post-processing of response – The raw output is formatted, cleaned up, standardized, source citations are added (to provide grounding evidence), and prepared for user consumption.
9. Response delivery – The formatted response is delivered to the end user. This RAG-powered chatbot deals with the above-mentioned challenges of an LLM-only chatbot as follows:
   • RAG removes the limitation of limited context by retrieving additional sources of information and enhancing the input prompt for LLMs.
   • Via advanced context processing techniques of re-ranking, and filtering), RAG improves factual correctness and accuracy of the chatbot by generating responses grounded in actual evidence preventing hallucinations.
   • RAG allows dynamic updates by supporting real-time data retrievals and updates.
   While RAG is powerful wrt introducing tangible advantages in LLM applications, as Anthropic suggests sometimes a simple solution such as using a longer prompt which includes the entire knowledge base (~20k tokens from ~500 pages) might work the best for Claude (Anthropic’s LLMs) models.

1.2 What are the challenges in RAG evaluation?

There are several key factors that make RAG evaluation challenging:

1. Complex queries requiring advanced information synthesis: User queries requiring inter-linking of sources to develop logical and reasoning-driven answers necessitate multi-hop retrievals. The complexities of synthesizing information from multiple retrieved chunks in RAG make it difficult to ensure synthesis quality while avoiding loss of critical context. E.g., when answering a customer’s query about their eligibility for credit limit increase (CLI) on their credit card, the system must correctly synthesize information from a customer’s credit profile, account history, repayment behaviour, credit bureau reports, and bank’s product-specific regulations around internal risk assessment policies, income thresholds and permissible credit limits.
2. Multi-modular architecture: The dual modules of retrieval and generation introduce high potential for error compounding, and propagation across the pipeline. Poor quality retrieval can lead to errors in information synthesis and therefore, generation. This means retrieval quality directly impacts generation quality. E.g., failure of the retrieval module to fetch the latest, most up-to-date risk assessment policies would lead to inaccurate response and thus, invalid communication with the customer.
3. Trade-off between context preservation versus relevance: There is a classic trade-off between context window size and relevance of retrieved content while maintaining practical performance. A longer context window implies larger retrieved documents preserving vital context but leads to longer latency and redundancy. A shorter context window reduces redundancy and improves speed / performance but risks loss of critical context. This means there is a need to evaluate multiple chunking strategies to find the most optimal one for a particular use case-domain combination. E.g., when answering a customer’s query about their eligibility for credit limit increase (CLI) on their credit card, a chatbot needs to find the optimal balance between: a. extracting long-term repayment history of 12 months which provides comprehensive creditworthiness assessment but increases latency, versus b. extracting short-term repayment history of 3 months which enables fast response delivery but can potentially miss richer patterns about customer behavior
4. Need for dynamic knowledge updates: Due to regular updates of knowledge base, RAG system behavior would also drift / evolve / change, making historical validations / evaluations no longer valid and requiring more frequent validation experiments. This need for temporal consistency underlines the need for regular RAG evaluation. E.g., since product-specific terms and conditions can change over time, the chatbot must be able to leverage this updated information via timely retrievals in its responses.
5. Non-determinism of LLM outputs: Since the LLM outputs are non-deterministic (valid but different responses for the same query across different iterations), RAG systems need advanced reference-based metrics for evaluation instead of traditional reference-based metrics. E.g., when explaining terms and conditions of a product, the system can rephrase that explanation in different ways making their comparisons difficult even if all of them are truthful and valid.
6. Ground Truth Challenges: While the RAG generated answers explicitly cite sources ensuring grounding, human expert knowledge is still needed for determining correctness of the system. Integrating human expertise is essential but creating these high-quality benchmarks is costly and time-consuming. E.g., responding accurately to customer queries about complex products such as derivatives could require expert inputs and validations from product experts with deep expertise of product features, regulatory landscape, and value proposition for the customer.
7. Domain Specificity: For understanding the overall system performance, the requirements for domain-specific accuracy metrics, specialized evaluation criteria. These must be further nuanced or revised with domain expertise of humans. E.g., while informing the customer about the international wire transfer process, the system should check for exact process as well as compliance with regulations related to fund transfers across borders, applicable fee components, and any other disclosure requirements.
8. Edge case testing: It might be hard to test the RAG systems for rare, edge cases – scenarios that involve intersecting conditions that make them both rare and complex to handle. There might be limited availability of datasets with these kinds of edge scenarios making reliability testing of the system harder. Careful and calibrated synthetic data generation could help here. E.g., handling queries about releasing the hold / freeze on an account while the bank has flagged fraudulent transactions – less frequent scenarios but critical to handle correctly.
9. Evaluation Metric Limitations: Traditional metrics (e.g., BLEU, ROUGE that are too strict focused on exact word matching) don’t capture all quality aspects such as semantic faithfulness necessitating need for an array of complementary metrics around various dimensions of retrieval quality, generation quality, answer accuracy, and consistency. Since several of these metrics need to be balanced against each other, optimizing them as a whole is always effort-intensive and challenging. E.g., Evaluation of responses to user queries about investment products requires looking out for not just accuracy but also regulatory compliance, risk disclosure requirements, and appropriateness of recommendations.
10. Cost and Resource Constraints: Continuous and comprehensive RAG evaluation is expensive with regard to time and resources. This means efficient and standardized evaluation methodologies (with critical feedback loops integrating human expertise) are critical to ensure scalable and effective testing achieving depth (detailed quality assessment) and coverage (all dimensions are evaluated). E.g., evaluating a chatbot’s performance across multiple categories of queries (product, risk, advisory, compliance) in face of evolving financial industry landscape requires ongoing and nuanced efforts to understand response quality implying significant investment in infrastructure and testing methodology.

2. Understanding RAG Assessement Framework

2.1 What makes RAG evaluation fundamentally different from traditional LLM evaluation?

Evaluation of traditional LLM systems is focused on quality of generation. However, due to various above-mentioned challenges associated with RAG evaluation, evaluation of RAG systems requires a multi-layered approach to ensure its reliability, trustworthiness, fairness and accuracy. Due to these differences in architecture and knowledge systems of the two options, RAG evaluation is fundamentally different from traditional LLM evaluation in the following ways:

These core differences require developing and leveraging evaluation frameworks and metrics specifically designed for RAG systems.

2.2 What is a holistic RAG evaluation framework? What are the core components that need to be evaluated in a RAG system?

A holistic RAG assessment requires evaluating its core components of retrieval, generation and the overall system with respect to the user experience it delivers. Evaluation of each category requires careful consideration and analysis along with human expert inputs. The evaluation begins with analysing retrieval performance, where context relevance is the primary metric for measuring information retrieval effectiveness. The assessment then requires assessing generation quality composed of answer relevance, faithfulness, correctness, and citation accuracy to determine the accuracy and reliability of the generated responses. Finally, we evaluate the user experience to determine the overall system performance.
(A) Retrieval Evaluation:

1. Contextual Relevance: measures how well the system retrieves precisely targeted information that addresses the query, optimizing computational efficiency by minimizing extraneous content.

(B) Generative Evaluation:

1. Answer Relevance evaluates whether the generated response directly addresses the query’s requirements, with penalties for incomplete or superfluous content.
2. Faithfulness assesses how accurately the generated response reflects the retrieved context, ensuring all statements can be traced back to the source materials and identifying potential hallucinations and validating factual correctness.
3. Correctness compares the generated response against human experts / human annotators - validated reference answers (“golden passages”) to validate factual accuracy of the answers.
4. Citation Quality evaluates the accuracy and appropriateness of source citations in generated outputs, calculated by averaging citation precision and recall scores across all referenced citations.

(C) User Experience Evaluation: • In addition to the above metrics, RAG systems require additional evaluations such as assessing robustness to noise in documents, latency, and response diversity.
• Metrics related to computational and time efficiency of retrieval aren’t relevant for evaluation of traditional LLM applications because they operate on a fixed knowledge base that’s baked into their model weights during training. Querying a traditional LLM will trigger the simple, straightforward workflow sketched above which includes processing the input prompt, accessing the LLM’s internal knowledge representation through parameter activations, and generating the response. This is relatively uniform in terms of computational requirements.
• In contrast, as seen in the workflows above, RAG systems have additional components of vector database search, document chunking and embedding, similarity computation, and context window management.
• Thus, RAG performance time depends on multiple factors such as size of document collection, # of relevant documents, document length, # of multiple retrievals needed, and vector store performance. This is why RAG evaluation must consider these additional metrics that determine user experience. There are a few options for evaluators to pursue evaluation of each dimension.
(1) Lexical evaluators rely on exact word matching techniques and statistical calculations, employing metrics like keyword frequency and Mean Reciprocal Rank (MRR) to assess how well the document text aligns with queries at a surface level. However, this approach does not considers deeper meaning of the text.
(2) Semantic evaluators are oriented with meaning-based analysis, measuring the conceptual alignment between queries and documents by analysing contextual relationships and thematic connections. This helps capture the query intent and the documents allowing for relevance-based evaluation rather than plain keyword matching.
(3) LLM as a judge evaluators use LLMs to assess content quality by determining coherence, accuracy, and relevance through their broad language understanding capabilities.

But before we deep-dive into these metrics, it is important to understand their fundamental building blocks especially about the confusion matrix for a RAG.

RAG Evaluation Confusion Matrix

Examples of each category of the RAG Evaluation Confusion Matrix:

Image Source: Benchmarking of Retrieval Augmented Generation: A Comprehensive Systematic Literature Review on Evaluation Dimensions, Evaluation Metrics and Datasets

A comprehensive RAG evaluation framework to be sufficient needs automated metrics for basic quality checks, expert review for critical cases and enough samples, continuous monitoring in production, and regular updates to evaluation criteria as the domain and landscape of the use case evolves.

Image Source: Benchmarking of Retrieval Augmented Generation: A Comprehensive Systematic Literature Review on Evaluation Dimensions, Evaluation Metrics and Datasets

APPENDIX

Image Source: Benchmarking of Retrieval Augmented Generation: A Comprehensive Systematic Literature Review on Evaluation Dimensions, Evaluation Metrics and Datasets

References

• https://www.scitepress.org/Papers/2024/130657/130657.pdf
• https://arxiv.org/pdf/2309.01431
• https://arxiv.org/html/2405.07437v2#S2
• https://www.anthropic.com/news/contextual-retrieval
• https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/