testing-examples/export-test-to-csv/export-test-to-csv.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "83bff230-4b24-47cb-9be8-4afc15a3e257",
   "metadata": {},
   "source": [
    "# Exporting a Test to CSV\n",
    "\n",
    "If you want to export test/experiment results to a CSV for later analysis or reporting, you can use the client's beta `get_test_results` utility.\n",
    "\n",
    "The basic code snippet is as follows:\n",
    "\n",
    "```python\n",
    "import langsmith\n",
    "\n",
    "client = langsmith.Client()\n",
    "\n",
    "# Project here is the test / experiment name \n",
    "df = client.get_test_results(project_name=\"My Project\")\n",
    "df.to_csv(\"results.csv\")\n",
    "```\n",
    "\n",
    "For more control over the structure and content of the fields, check out the [Downloading Feedback and Examples](../download-feedback-and-examples/download_example.ipynb) notebook.\n",
    "\n",
    "We will review a quick example below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8e618c6-2015-42a3-a388-12711909aaa3",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -U langsmith langchain pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "951edc89-68f3-473f-aa4f-ff446a604c32",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import uuid\n",
    "\n",
    "# Adjust if self-hosted\n",
    "# os.environ[\"LANGCHAIN_ENDPOINT\"] = \"https://api.smith.langchain.com\"\n",
    "# os.environ[\"LANGCHAIN_API_KEY\"] = \"YOUR API KEY\"\n",
    "# os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1e79b8f-3dae-4efe-9000-9dac4d7db61e",
   "metadata": {},
   "source": [
    "## Example Dataset\n",
    "\n",
    "Our toy example will be testing whether the chain can compute the `n`'th fibonnaci number. You can skip this section if you already have a test you wish to export."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5c078a64-e65c-44d4-842f-0c2b0654a485",
   "metadata": {},
   "outputs": [],
   "source": [
    "def fibonacci(n):\n",
    "    a, b = 0, 1\n",
    "    for _ in range(n):\n",
    "        a, b = b, a + b\n",
    "    return a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "f2331fcb-d142-4ce5-8292-7fd180d5a607",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View the evaluation results for project 'My test data' at:\n",
      "https://smith.langchain.com/o/ebbaf2eb-769b-4505-aca2-d11de10372a4/datasets/b9ecc8d0-1e0c-4094-999c-cfdca9472c7b/compare?selectedSessions=11155a34-0a71-49fb-a445-5fa62867e467\n",
      "\n",
      "View all tests for Dataset My Dataset - 855dcc at:\n",
      "https://smith.langchain.com/o/ebbaf2eb-769b-4505-aca2-d11de10372a4/datasets/b9ecc8d0-1e0c-4094-999c-cfdca9472c7b\n",
      "[------------------------------------------------->] 10/10"
     ]
    }
   ],
   "source": [
    "import openai\n",
    "from langchain.smith import RunEvalConfig\n",
    "from langsmith import Client, traceable\n",
    "from langsmith.wrappers import wrap_openai\n",
    "\n",
    "client = Client()\n",
    "openai_client = wrap_openai(openai.Client())\n",
    "\n",
    "# Dataset\n",
    "\n",
    "test_name = \"My test data\"\n",
    "dataset_name = f\"My Dataset - {uuid.uuid4().hex[:6]}\"\n",
    "ds = client.create_dataset(dataset_name=dataset_name)\n",
    "client.create_examples(\n",
    "    inputs=[{\"n\": i} for i in range(10)],\n",
    "    outputs=[{\"expected\": fibonacci(i)} for i in range(10)],\n",
    "    dataset_id=ds.id,\n",
    ")\n",
    "\n",
    "\n",
    "# Evaluator\n",
    "def exact_match(run, example):\n",
    "    score = run.outputs[\"output\"] == example.outputs[\"expected\"]\n",
    "    return {\"score\": score}\n",
    "\n",
    "\n",
    "eval_config = RunEvalConfig(evaluators=[exact_match])\n",
    "\n",
    "# Model/chain we're testing\n",
    "\n",
    "\n",
    "@traceable\n",
    "def llm_fibonacci(n: int):\n",
    "    completion = openai_client.chat.completions.create(\n",
    "        model=\"gpt-3.5-turbo\",\n",
    "        messages=[\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": f\"Compute the {n}'th fibonacci number. \"\n",
    "                \"Think step by step, the n print the output. Finally, return the number at the end on a newline.\"\n",
    "                \" The final line MUST be parseable with python's int() function.\",\n",
    "            }\n",
    "        ],\n",
    "    )\n",
    "    result = completion.choices[0].message.content\n",
    "    return {\"output\": int(result.split()[-1].strip())}\n",
    "\n",
    "\n",
    "# Evaluate\n",
    "test_results = client.run_on_dataset(\n",
    "    dataset_name=dataset_name,\n",
    "    project_name=test_name,\n",
    "    llm_or_chain_factory=llm_fibonacci,\n",
    "    evaluation=eval_config,\n",
    ")\n",
    "# You could directly output as a csv using\n",
    "# test_results.to_dataframe().to_csv(...)\n",
    "# But we assume you want to export post-facto"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d63c938e-b9eb-4518-94f9-4bafce6d7e83",
   "metadata": {},
   "source": [
    "## Export"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "81a4f4b0-909d-4d56-9edd-9ada6871e695",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = client.get_test_results(project_name=test_name)\n",
    "df.to_csv(\"results.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "6ae9e31e-3a9d-4a4d-a034-bd4cfafc42da",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "reference.expected,input.n,outputs.output,feedback.exact_match,execution_time,error,id\n",
      "13,7,7,0.0,1.426419,,11193006-9afb-4896-8f01-22575a505e74\n",
      "1,1,55,0.0,5.490039,,d88763a0-d07d-42b3-aaf9-47fc0fd39968\n",
      "21,8,21,1.0,1.262267,,019a26a6-709b-4db9-9017-18c4373502ff\n",
      "1,2,1,1.0,2.711957,,930f4bb2-3c05-4e29-9c23-96d718877ea6\n",
      "34,9,9,0.0,1.861771,,a134dc53-9a68-4528-abf9-fcb5da99af98\n",
      "3,4,3,1.0,3.174816,,3e6551a3-b293-4c3e-ab6c-cc6c598625b2\n",
      "5,5,5,1.0,1.634533,,ca084860-155f-483a-9a21-5b0d36b30998\n",
      "2,3,2,1.0,2.088853,,8e56a900-4364-4f8c-b296-5bd216b52241\n",
      "0,0,0,1.0,1.809274,,86e44d51-1253-441b-a672-2b5dda04a14f\n",
      "8,6,5,0.0,4.030962,,1e9147fe-6e33-411a-9f04-98083fb6a489\n"
     ]
    }
   ],
   "source": [
    "!cat results.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d67acf82",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Congrats! You've exported a flat table of your test results. The beta `get_test_results` utility lets you easily export your Langsmith test results to a CSV file. This can be handy if you wan to:\n",
    "\n",
    "- Perform custom analysis and interpretation of the evaluation metrics\n",
    "- Create visualizations using tools like matplotlib or plotly to better understand performance\n",
    "- Share the test results with partners, leaders, or other stakeholders\n",
    "- Include the exported data in research papers or reports\n",
    "\n",
    "\n",
    "We plan on addition additional first-class support for this type of reporting in the UI, but there's no need to wait, since the API/client let you do it today."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "992907f9",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}