testing-examples/ragas/ragas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fe90efd1",
   "metadata": {},
   "source": [
    "# RAG evaluation with RAGAS\n",
    "\n",
    "Ragas is a popular framework that helps you evaluate your Retrieval Augmented Generation (RAG) pipelines.\n",
    "\n",
    "This notebook shows how you can integrate their excellent RAG metrics in LangSmith to evaluate your RAG app.\n",
    "\n",
    "For this example, we will grade a simple RAG application based on the following metrics. Most metrics use an LLM as a judge in some capacity:\n",
    "\n",
    "**Labeled generated metrics:**\n",
    "- `answer_correctness` - Is the response correct, based on the `ground_truth` response.\n",
    "\n",
    "**Reference-free generator metrics:**\n",
    "- `faithfulness` - Proportion of claims in the response that are grounded in the retrieved context ([entailment](https://en.wikipedia.org/wiki/Textual_entailment)-based).\n",
    "\n",
    "**Reference-free retriever metrics:**\n",
    "- `context_relevancy` - Proportion of retrieved sentences that are \"relevant\" to the user question.\n",
    "\n",
    "**Labeled retriever metrics:**\n",
    "- `context_recall` - Proportion of the _ground truth_ answer that can be attributed to the docs.\n",
    "- `context_precision` - Are the relevant docs (according to the _ground truth_) ranked higher? Works by scoring each document as useful in deducing the ground truth, computing precision @ K using that score for each K, then averaging over the total number of useful docs.\n",
    "\n",
    "In reality, you likely won't need to apply all of these metrics at the same time, but each metric can shed a bit of light on the different aspects of your retriever and generator setup."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f3a99a6-6e1f-45ac-a090-b25a75c02a7c",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "Install recent versions of the required dependencies, and configure\n",
    "your environment with the appropriate LANGSMITH and OpenAI keys."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d9d733a0-398f-43fc-a217-7146a082d870",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture --no-stderr\n",
    "%pip install -U langsmith ragas numpy openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6cb3ce85-ceb5-4b01-9205-b6c079469818",
   "metadata": {},
   "outputs": [],
   "source": [
    "import getpass\n",
    "import os\n",
    "\n",
    "os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
    "os.environ[\"LANGCHAIN_API_KEY\"] = getpass.getpass(\"LANGCHAIN_API_KEY\")\n",
    "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OPENAI_API_KEY\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28504e01",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "\n",
    "Clone the [BaseCamp Q&A](https://smith.langchain.com/public/56fe54cd-b7d7-4d3b-aaa0-88d7a2d30931/d) dataset to your organization to get stated. This was generated by synthetically generating questions over  scraped documents from the [37signals](https://basecamp.com/handbook) handbook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "eb4306d5-78a0-42e4-b5b3-df7b5cacf81b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import langsmith\n",
    "\n",
    "client = langsmith.Client()\n",
    "dataset_url = (\n",
    "    \"https://smith.langchain.com/public/56fe54cd-b7d7-4d3b-aaa0-88d7a2d30931/d\"\n",
    ")\n",
    "dataset_name = \"BaseCamp Q&A\"\n",
    "client.clone_public_dataset(dataset_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e18681df",
   "metadata": {},
   "source": [
    "### Define your pipeline\n",
    "\n",
    "First, download the source docs. We've saved the raw markdown files in a zipfile to make this easier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5effaff1-3451-440a-bdeb-702fbc8682be",
   "metadata": {},
   "outputs": [],
   "source": [
    "import io\n",
    "import os\n",
    "import zipfile\n",
    "\n",
    "import requests\n",
    "\n",
    "# Fetch the source documents\n",
    "url = \"https://storage.googleapis.com/benchmarks-artifacts/basecamp-data/basecamp-data.zip\"\n",
    "\n",
    "response = requests.get(url)\n",
    "\n",
    "\n",
    "with io.BytesIO(response.content) as zipped_file:\n",
    "    with zipfile.ZipFile(zipped_file, \"r\") as zip_ref:\n",
    "        zip_ref.extractall()\n",
    "\n",
    "data_dir = os.path.join(os.getcwd(), \"data\")\n",
    "docs = []\n",
    "for filename in os.listdir(data_dir):\n",
    "    if filename.endswith(\".md\"):\n",
    "        with open(os.path.join(data_dir, filename), \"r\") as file:\n",
    "            docs.append({\"file\": filename, \"content\": file.read()})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08f6a0ce-19ae-40de-bd97-68037567d8cf",
   "metadata": {},
   "source": [
    "Next, create the retriever. For our purposes, a simple in-memory vectorstore retriever will suffice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b3e4fb09-e412-49a5-b4b4-133ee0f1c2d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import List\n",
    "\n",
    "import numpy as np\n",
    "import openai\n",
    "from langsmith import traceable\n",
    "\n",
    "\n",
    "class VectorStoreRetriever:\n",
    "    def __init__(self, docs: list, vectors: list, oai_client):\n",
    "        self._arr = np.array(vectors)\n",
    "        self._docs = docs\n",
    "        self._client = oai_client\n",
    "\n",
    "    @classmethod\n",
    "    async def from_docs(cls, docs, oai_client):\n",
    "        embeddings = await oai_client.embeddings.create(\n",
    "            model=\"text-embedding-3-small\", input=[doc[\"content\"] for doc in docs]\n",
    "        )\n",
    "        vectors = [emb.embedding for emb in embeddings.data]\n",
    "        return cls(docs, vectors, oai_client)\n",
    "\n",
    "    @traceable\n",
    "    async def query(self, query: str, k: int = 5) -> List[dict]:\n",
    "        embed = await self._client.embeddings.create(\n",
    "            model=\"text-embedding-3-small\", input=[query]\n",
    "        )\n",
    "        # \"@\" is just a matrix multiplication in python\n",
    "        scores = np.array(embed.data[0].embedding) @ self._arr.T\n",
    "        top_k_idx = np.argpartition(scores, -k)[-k:]\n",
    "        top_k_idx_sorted = top_k_idx[np.argsort(-scores[top_k_idx])]\n",
    "        return [\n",
    "            {**self._docs[idx], \"similarity\": scores[idx]} for idx in top_k_idx_sorted\n",
    "        ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e23a3852-32b1-48f4-95f3-af71d559a6f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langsmith import traceable\n",
    "from langsmith.wrappers import wrap_openai\n",
    "\n",
    "\n",
    "class NaiveRagBot:\n",
    "    def __init__(self, retriever, model: str = \"gpt-4-turbo-preview\"):\n",
    "        self._retriever = retriever\n",
    "        # Wrapping the client instruments the LLM\n",
    "        # and is completely optional\n",
    "        self._client = wrap_openai(openai.AsyncClient())\n",
    "        self._model = model\n",
    "\n",
    "    @traceable\n",
    "    async def get_answer(self, question: str):\n",
    "        similar = await self._retriever.query(question)\n",
    "        response = await self._client.chat.completions.create(\n",
    "            model=self._model,\n",
    "            messages=[\n",
    "                {\n",
    "                    \"role\": \"system\",\n",
    "                    \"content\": \"You are a helpful AI assistant.\"\n",
    "                    \" Use the following docs to help answer the user's question.\\n\\n\"\n",
    "                    f\"## Docs\\n\\n{similar}\",\n",
    "                },\n",
    "                {\"role\": \"user\", \"content\": question},\n",
    "            ],\n",
    "        )\n",
    "\n",
    "        # The RAGAS evaluators expect the \"answer\" and \"contexts\"\n",
    "        # keys to work properly. If your pipeline does not return these values,\n",
    "        # you should wrap in a function that provides them.\n",
    "        return {\n",
    "            \"answer\": response.choices[0].message.content,\n",
    "            \"contexts\": [str(doc) for doc in similar],\n",
    "        }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "27d3a9f7-2ca2-495e-8cbc-5d276bcb9f8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "retriever = await VectorStoreRetriever.from_docs(docs, openai.AsyncClient())\n",
    "rag_bot = NaiveRagBot(retriever)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "30ee69f5-d0f2-4683-ad1e-e3a71c2aa18e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'According to the provided documents, at 37signals, employees are entitled to various forms of time off, including:\\n\\n1. **Paid Time Off (Vacation Time)'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response = await rag_bot.get_answer(\"How much time off do we get?\")\n",
    "response[\"answer\"][:150]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92529864-e251-4b93-b968-b885c3d2f058",
   "metadata": {},
   "source": [
    "## Evaluate.\n",
    "\n",
    "Ragas provides you with a different metrics that you can use to evaluate each component of your RAG pipeline. You can see the entire list in the [docs](https://docs.ragas.io/en/latest/concepts/metrics/index.html). We will select a few useful ones below.\n",
    "\n",
    "To use a RAGAS metric, simply wrap as an `EvaluatorChain`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "cca237aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.smith import RunEvalConfig\n",
    "from ragas.integrations.langchain import EvaluatorChain\n",
    "from ragas.metrics import (\n",
    "    answer_correctness,\n",
    "    answer_relevancy,\n",
    "    context_precision,\n",
    "    context_recall,\n",
    "    context_relevancy,\n",
    "    faithfulness,\n",
    ")\n",
    "\n",
    "# Wrap the RAGAS metrics to use in LangChain\n",
    "evaluators = [\n",
    "    EvaluatorChain(metric)\n",
    "    for metric in [\n",
    "        answer_correctness,\n",
    "        answer_relevancy,\n",
    "        context_precision,\n",
    "        context_recall,\n",
    "        faithfulness,\n",
    "    ]\n",
    "]\n",
    "eval_config = RunEvalConfig(custom_evaluators=evaluators)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d43007aa-3e80-4039-b4e5-cff030bb89c2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View the evaluation results for project 'back-bibliography-2' at:\n",
      "https://smith.langchain.com/o/ebbaf2eb-769b-4505-aca2-d11de10372a4/datasets/e468392e-e369-4066-99d4-dd05e186e992/compare?selectedSessions=bf005eaa-498d-47f7-a752-4bd260000c23\n",
      "\n",
      "View all tests for Dataset BaseCamp Q&A at:\n",
      "https://smith.langchain.com/o/ebbaf2eb-769b-4505-aca2-d11de10372a4/datasets/e468392e-e369-4066-99d4-dd05e186e992\n",
      "[->                                                ] 1/21"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/wfh/code/lc/langchain/libs/core/langchain_core/_api/deprecation.py:117: LangChainDeprecationWarning: The function `__call__` was deprecated in LangChain 0.1.0 and will be removed in 0.2.0. Use invoke instead.\n",
      "  warn_deprecated(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[---------------->                                 ] 7/21"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Invalid JSON response. Expected dictionary with key 'question'\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[----------------------->                          ] 10/21"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Invalid JSON response. Expected dictionary with key 'question'\n",
      "Invalid response format. Expected a list of dictionaries with keys 'verdict'\n",
      "Invalid JSON response. Expected dictionary with key 'question'\n",
      "Invalid JSON response. Expected dictionary with key 'Attributed'\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[---------------------------->                     ] 12/21"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Invalid JSON response. Expected dictionary with key 'question'\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[------------------------------------------>       ] 18/21"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Invalid JSON response. Expected dictionary with key 'question'\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-------------------------------------------->     ] 19/21"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Invalid JSON response. Expected dictionary with key 'Attributed'\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[------------------------------------------------->] 21/21"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Invalid JSON response. Expected dictionary with key 'question'\n"
     ]
    }
   ],
   "source": [
    "results = await client.arun_on_dataset(\n",
    "    dataset_name=dataset_name,\n",
    "    llm_or_chain_factory=rag_bot.get_answer,\n",
    "    evaluation=eval_config,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68974f4c-195c-4fa5-9d80-5f95cc482c77",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}