EPIC: RAG Evaluations MVP

[jalling97]

RAG Evaluations MVP

Description

LFAI needs a framework for evaluations in order to:

• Validate the efficacy of RAG at all stages
• Make model recommendations for various scenarios
• Establish a baseline for future improvements
• Sanity check that base model functionality is implemented properly

The MVP for this evaluations framework will be limited in scope to the following evaluation scenarios:

• Chat - regular chat-based interactions with the llm without added context
• Chat with your docs - RAG-enabled chat interactions that require retrieved context

This framework can be expanded upon as LFAI develops, but this epic covers an MVP that accomplishes the above goals.

Completion Criteria

• Decision of what toolset should be used to handle evaluations (Deepeval)
  • ADR: RAG Evaluation Toolset #196
• Identification of 1 or more existing evaluation datasets for rapid prototyping
  • (spike) Identify Open Source Datasets for RAG Evaluations #715
• Creation of a custom dataset that is mission-focused while remaining openly available for more targeted evaluations
  • (spike) Create a Custom RAG Evaluations Dataset #716
• Determine a list of open-source models to evaluate on
  • (spike) Determine open source LLMs to evaluate on #717
• Establishment of which evaluation metrics and custom heuristics should be used
  • (spike) Establish Metrics for RAG Evaluations #718
• Decision of what evaluator llm should be used for llm-as-a-judge evaluations
  • (spike) Select an Evaluator LLM for LLM-as-judge Evals #719
• Decision on how evals will be run (workflow? ad-hoc python script? notebook?) and where data will be stored
  • (spike) Evals Execution Methodology #720
• Determine what the "Unicorn model card" should look like (i.e how the evaluation results are delivered/conveyed)
  • (spike) Evals "Model Card" #721
• ADR for LFAI Evals Framework
  • ADR: RAG Evaluations Framework #823
• Add Needle in a Haystack Evaluations
  • (feat) Add Needle in a Haystack Evaluation Infrastructure #944
• Add LLM-as-judge Evaluations
  • feat: add LLM-as-judge evaluations #957

[jalling97]

RAG Evals MVP Epic is now finished