testing-examples/movie-demo/prompt_iteration.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f7c15551-fc23-4b3e-9b6a-226b68544447",
   "metadata": {},
   "source": [
    "# Prompt Iteration Walkthrough\n",
    "\n",
    "This notebook performs a quick walkthrough of LangSmith's evaluation flow, introducing:\n",
    "\n",
    "1. Datasets & Evaluation\n",
    "2. Summary evaluators (for aggregate statistics)\n",
    "3. Prompt Versioning in the hub.\n",
    "4. Using the LLM proxy.\n",
    "\n",
    "Importantly, the pipelines in this walkthrough **do not depend on the LangChain open source libraries**. LangChain is only used for in part 3 to show how to use one of its many off-the-shelf evaluators or an din part 4 to connect to the prompt Hub.\n",
    "\n",
    "## Setup\n",
    "\n",
    "First, we'll do some setup. Create a LangSmith API Key by navigating to the settings page in LangSmith, then set the following environment variables.\n",
    "\n",
    "```bash\n",
    "OPENAI_API_KEY=<YOUR OPENAI API KEY>\n",
    "LANGCHAIN_TRACING_V2=true\n",
    "LANGCHAIN_PROJECT=<YOUR PROJECT NAME>\n",
    "LANGCHAIN_API_KEY=<YOUR LANGSMITH API KEY>\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f18c847a-a7c1-4db5-a5f1-4f845e1a78a9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langsmith import Client\n",
    "\n",
    "client = Client()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb69a1af-87c4-4544-9be9-76d42c537655",
   "metadata": {},
   "source": [
    "## Pt. 1 -- Toxic Queries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "f4f68071-dd1d-40fb-ab80-90febaee2064",
   "metadata": {},
   "outputs": [],
   "source": [
    "toxic_examples = [\n",
    "    (\"Shut up, idiot\", \"Toxic\"),\n",
    "    (\"You're a wonderful person\", \"Not toxic\"),\n",
    "    (\"This is the worst thing ever\", \"Toxic\"),\n",
    "    (\"I had a great day today\", \"Not toxic\"),\n",
    "    (\"Nobody likes you\", \"Toxic\"),\n",
    "    (\"This movie is a masterpiece\", \"Not toxic\"),\n",
    "    (\"Go away and never come back\", \"Toxic\"),\n",
    "    (\"Thank you for your help\", \"Not toxic\"),\n",
    "    (\"This is so dumb\", \"Toxic\"),\n",
    "    (\"I appreciate your efforts\", \"Not toxic\"),\n",
    "    (\"This is a waste of time\", \"Toxic\"),\n",
    "    (\"This movie blows\", \"Toxic\"),\n",
    "    (\"This is unacceptable. I want to speak to the manager.\", \"Toxic\"),\n",
    "]\n",
    "\n",
    "toxic_dataset_name = \"Toxic Queries\"\n",
    "if not client.has_dataset(dataset_name=toxic_dataset_name):\n",
    "    toxic_dataset = client.create_dataset(dataset_name=toxic_dataset_name)\n",
    "    inputs, outputs = zip(\n",
    "        *[({\"text\": text}, {\"label\": label}) for text, label in toxic_examples]\n",
    "    )\n",
    "    client.create_examples(inputs=inputs, outputs=outputs, dataset_id=toxic_dataset.id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49ed6bdc-5046-4685-827c-0abb7fddf940",
   "metadata": {},
   "source": [
    "#### Our Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "3de889d9-4e13-4f29-acab-01e4e44692a4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langsmith import traceable, wrappers\n",
    "from openai import Client\n",
    "\n",
    "openai = wrappers.wrap_openai(Client())\n",
    "\n",
    "\n",
    "@traceable\n",
    "def label_query(text):\n",
    "    messages = [\n",
    "        {\n",
    "            \"role\": \"system\",\n",
    "            \"content\": \"Given the following user query, assess whether it contains toxic content. Please provide a simple 'Toxic' or 'Not toxic' response based on your assessment.\",\n",
    "        },\n",
    "        {\"role\": \"user\", \"content\": text},\n",
    "    ]\n",
    "    result = openai.chat.completions.create(\n",
    "        messages=messages, model=\"gpt-3.5-turbo\", temperature=0\n",
    "    )\n",
    "    return result.choices[0].message.content\n",
    "\n",
    "\n",
    "@traceable\n",
    "def label_query_alternate_prompt(text):\n",
    "    messages = [\n",
    "        {\n",
    "            \"role\": \"system\",\n",
    "            \"content\": \"Please review the user query below and determine if it contains any form of toxic behavior, such as insults, threats, or highly negative comments. Respond with 'Toxic' if it does, and 'Not toxic' if it doesn't.\",\n",
    "        },\n",
    "        {\"role\": \"user\", \"content\": text},\n",
    "    ]\n",
    "    result = openai.chat.completions.create(\n",
    "        messages=messages, model=\"gpt-3.5-turbo\", temperature=0\n",
    "    )\n",
    "    return result.choices[0].message.content"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0cfd0e8-12a2-4958-bef5-7eb6f2072595",
   "metadata": {},
   "source": [
    "#### Evaluate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "de86a367",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/gf/6rnp_mbx5914kx7qmmh7xzmw0000gn/T/ipykernel_37884/109358324.py:22: UserWarning: Function evaluate is in beta.\n",
      "  results_1 = evaluate(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View the evaluation results for experiment: 'Toxic Queries:683e040' at:\n",
      "https://smith.langchain.com/o/ebbaf2eb-769b-4505-aca2-d11de10372a4/datasets/dc6cc406-d05e-4e57-9c47-5868523f5a98/compare?selectedSessions=4fa33927-71dd-4edc-88bd-bac87df4d346\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2ab5e6984562489ab1e2e8611514db6d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from langsmith.evaluation import evaluate\n",
    "\n",
    "\n",
    "# Row-level evaluator\n",
    "def correct_label(run, example) -> dict:\n",
    "    score = run.outputs.get(\"output\") == example.outputs.get(\"label\")\n",
    "    return {\"score\": int(score)}\n",
    "\n",
    "\n",
    "# Summary (experiment-level) evaluator\n",
    "def summary_eval(runs, examples):\n",
    "    correct = 0\n",
    "    for i, run in enumerate(runs):\n",
    "        if run.outputs[\"output\"] == examples[i].outputs[\"label\"]:\n",
    "            correct += 1\n",
    "    if correct / len(runs) > 0.5:\n",
    "        return {\"key\": \"pass\", \"score\": True}\n",
    "    else:\n",
    "        return {\"key\": \"pass\", \"score\": False}\n",
    "\n",
    "\n",
    "results_1 = evaluate(\n",
    "    lambda inputs: label_query(inputs[\"text\"]),\n",
    "    data=toxic_dataset_name,\n",
    "    evaluators=[correct_label],\n",
    "    summary_evaluators=[summary_eval],\n",
    "    experiment_prefix=\"Toxic Queries\",\n",
    "    metadata={\n",
    "        \"prompt_version\": \"1\",\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "547ebbd3",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/gf/6rnp_mbx5914kx7qmmh7xzmw0000gn/T/ipykernel_37884/2093667517.py:1: UserWarning: Function evaluate is in beta.\n",
      "  results_2 = evaluate(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View the evaluation results for experiment: 'Toxic Queries:fb8f77e' at:\n",
      "https://smith.langchain.com/o/ebbaf2eb-769b-4505-aca2-d11de10372a4/datasets/dc6cc406-d05e-4e57-9c47-5868523f5a98/compare?selectedSessions=4588ca56-f3d5-4384-8a9c-680510d60142\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e7daeb5cff754ea0a428d80d8586c887",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "results_2 = evaluate(\n",
    "    lambda inputs: label_query_alternate_prompt(inputs[\"text\"]),\n",
    "    data=toxic_dataset_name,\n",
    "    evaluators=[correct_label],\n",
    "    summary_evaluators=[summary_eval],\n",
    "    experiment_prefix=\"Toxic Queries\",\n",
    "    metadata={\n",
    "        \"prompt_version\": \"2\",\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f29e897-ad02-4647-b096-45f3000465f5",
   "metadata": {},
   "source": [
    "### Aside: Using the LangSmith Hub for Prompt Management"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "5371b583-b2e3-4300-a303-5c74a1ed22d4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/gf/6rnp_mbx5914kx7qmmh7xzmw0000gn/T/ipykernel_37884/1549686522.py:20: UserWarning: Function evaluate is in beta.\n",
      "  results = evaluate(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View the evaluation results for experiment: 'Toxic Queries prompt @8d80588e:b495152' at:\n",
      "https://smith.langchain.com/o/ebbaf2eb-769b-4505-aca2-d11de10372a4/datasets/dc6cc406-d05e-4e57-9c47-5868523f5a98/compare?selectedSessions=7b346b83-e094-414d-860b-8c4c0fb23714\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "db884eee8a8746a6b2fc52930622e640",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from langchain import hub\n",
    "from langchain_openai.chat_models.base import _convert_message_to_dict\n",
    "\n",
    "HUB_COMMIT_HASH = \"8d80588e\"\n",
    "obj = hub.pull(f\"langchain-ai/movie-demo:{HUB_COMMIT_HASH}\")\n",
    "hub_messages = [\n",
    "    _convert_message_to_dict(message.format()) for message in obj.messages[:1]\n",
    "]\n",
    "\n",
    "\n",
    "@traceable\n",
    "def label_query_hub(text):\n",
    "    messages = hub_messages + [{\"role\": \"user\", \"content\": text}]\n",
    "    result = openai.chat.completions.create(\n",
    "        messages=messages, model=\"gpt-3.5-turbo\", temperature=0\n",
    "    )\n",
    "    return result.choices[0].message.content\n",
    "\n",
    "\n",
    "results = evaluate(\n",
    "    lambda inputs: label_query_hub(inputs[\"text\"]),\n",
    "    data=toxic_dataset_name,\n",
    "    evaluators=[correct_label],\n",
    "    summary_evaluators=[summary_eval],\n",
    "    experiment_prefix=f\"Toxic Queries prompt @{HUB_COMMIT_HASH}\",\n",
    "    metadata={\n",
    "        \"prompt_version\": HUB_COMMIT_HASH,\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f158db3e-eb72-4d0d-bb91-bbe305449d56",
   "metadata": {},
   "source": [
    "## Pt. 2 -- Multi-Turn Queries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "4a9e84a3-2d7a-4811-80e0-0bf4d5299f13",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define multi-turn examples\n",
    "multi_turn_examples = [\n",
    "    (\n",
    "        [\n",
    "            \"Recommend some family-friendly movies for tonight\",\n",
    "            \"Do any of these have an educational theme?\",\n",
    "            \"Which one has the highest ratings?\",\n",
    "        ],\n",
    "        [\n",
    "            \"Some family-friendly movies available are 'The Lion King', 'Finding Nemo', and 'The Incredibles'\",\n",
    "            \"'The Lion King' and 'Finding Nemo' have educational themes about the circle of life and the importance of family\",\n",
    "            \"'The Incredibles' has the highest ratings among them with a 94% on Rotten Tomatoes\",\n",
    "        ],\n",
    "    ),\n",
    "    (\n",
    "        [\n",
    "            \"What are the top sci-fi movies on your service?\",\n",
    "            \"Any recent ones?\",\n",
    "            \"Can you suggest one that involves time travel?\",\n",
    "        ],\n",
    "        [\n",
    "            \"Top sci-fi movies include 'Blade Runner 2049', 'Interstellar', and 'The Martian'\",\n",
    "            \"A recent hit is 'Tenet', released in 2020\",\n",
    "            \"'Interstellar' involves complex time travel themes and is highly recommended\",\n",
    "        ],\n",
    "    ),\n",
    "    (\n",
    "        [\n",
    "            \"I'm looking for movies directed by Christopher Nolan\",\n",
    "            \"Which one would you recommend for a movie night?\",\n",
    "            \"What's the plot of 'Inception'?\",\n",
    "        ],\n",
    "        [\n",
    "            \"Christopher Nolan movies available include 'Inception', 'Dunkirk', and 'Interstellar'\",\n",
    "            \"'Inception' is a great pick for a movie night, offering a mix of action, drama, and mind-bending storytelling\",\n",
    "            \"'Inception' is about a thief who steals corporate secrets through dream-sharing technology and is given the inverse task of planting an idea into the mind of a CEO\",\n",
    "        ],\n",
    "    ),\n",
    "    (\n",
    "        [\n",
    "            \"Show me some popular romantic comedies\",\n",
    "            \"Any classics in the list?\",\n",
    "            \"Tell me more about 'When Harry Met Sally'\",\n",
    "        ],\n",
    "        [\n",
    "            \"Popular romantic comedies include 'Crazy Rich Asians', 'The Big Sick', and 'When Harry Met Sally'\",\n",
    "            \"'When Harry Met Sally' is considered a classic in the romantic comedy genre\",\n",
    "            \"'When Harry Met Sally' explores the question of whether men and women can just be friends, through the story of its titular characters over the years\",\n",
    "        ],\n",
    "    ),\n",
    "    (\n",
    "        [\n",
    "            \"Do you have documentaries on nature?\",\n",
    "            \"Which one focuses on marine life?\",\n",
    "            \"How long is 'Blue Planet II'?\",\n",
    "        ],\n",
    "        [\n",
    "            \"Yes, we have 'Planet Earth II', 'Blue Planet II', and 'Our Planet'\",\n",
    "            \"'Blue Planet II' focuses extensively on marine life, exploring the deep ocean, coral reefs, and the open sea\",\n",
    "            \"'Blue Planet II' is approximately 7 hours long, spread across 7 episodes\",\n",
    "        ],\n",
    "    ),\n",
    "]\n",
    "\n",
    "multi_turn_dataset_name = \"Multi-Turn Queries\"\n",
    "if not client.has_dataset(dataset_name=multi_turn_dataset_name):\n",
    "    multi_turn_dataset = client.create_dataset(dataset_name=multi_turn_dataset_name)\n",
    "    multi_turn_inputs, multi_turn_outputs = zip(\n",
    "        *[\n",
    "            ({\"queries\": queries}, {\"answers\": answers})\n",
    "            for queries, answers in multi_turn_examples\n",
    "        ]\n",
    "    )\n",
    "    client.create_examples(\n",
    "        inputs=multi_turn_inputs,\n",
    "        outputs=multi_turn_outputs,\n",
    "        dataset_id=multi_turn_dataset.id,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ddf19d0-5cd4-4ba9-be2c-f042d7247be7",
   "metadata": {},
   "source": [
    "#### RAG Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "e62fc6fc-7911-44ed-a2d9-bed05d71a459",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "tools = [\n",
    "    {\n",
    "        \"type\": \"function\",\n",
    "        \"function\": {\n",
    "            \"name\": \"retrieve_movies\",\n",
    "            \"description\": \"Retrieve a list of relevant movies and their metadata from a movie database.\",\n",
    "            \"parameters\": {\n",
    "                \"type\": \"object\",\n",
    "                \"properties\": {\n",
    "                    \"query\": {\n",
    "                        \"type\": \"string\",\n",
    "                        \"description\": \"The query used to retrieve movies from the movie database, for example 'Christopher Nolan films'\",\n",
    "                    },\n",
    "                },\n",
    "                \"required\": [\"query\"],\n",
    "            },\n",
    "        },\n",
    "    },\n",
    "]\n",
    "\n",
    "system_prompt = \"\"\"\n",
    "Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.\n",
    "Note that if the question does not require additional search and can be answered using the chat history, simply respond with the answer.\n",
    "Don't make up content that's not supplied in chat history.\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "@traceable\n",
    "def generate_movie_search(chat_history, query):\n",
    "    messages = (\n",
    "        [\n",
    "            {\"role\": \"system\", \"content\": system_prompt},\n",
    "        ]\n",
    "        + chat_history\n",
    "        + [{\"role\": \"user\", \"content\": query}]\n",
    "    )\n",
    "    result = openai.chat.completions.create(\n",
    "        messages=messages, model=\"gpt-3.5-turbo-0613\", tools=tools\n",
    "    )\n",
    "    return result.choices[0].message\n",
    "\n",
    "\n",
    "def _convert_docs(results):\n",
    "    return [\n",
    "        {\n",
    "            \"page_content\": r,\n",
    "            \"type\": \"Document\",\n",
    "        }\n",
    "        for r in results\n",
    "    ]\n",
    "\n",
    "\n",
    "@traceable(run_type=\"retriever\")\n",
    "def retrieve_movies(query):\n",
    "    # Foo retriever. In production, this would search an actual database\n",
    "    if \"family-friendly\" in query.lower():\n",
    "        return _convert_docs([\"Lion King\", \"Finding Nemo\", \"The Incredibles\"])\n",
    "    elif \"sci-fi\" in query.lower():\n",
    "        return _convert_docs([\"Blade Runner 2049\", \"Interstellar\", \"The Martian\"])\n",
    "    elif \"nature\" in query.lower():\n",
    "        return _convert_docs([\"Planet Earth II\", \"Blue Planet II\", \"Our Planet\"])\n",
    "    elif \"christopher nolan\" in query.lower():\n",
    "        return _convert_docs([\"Inception\", \"Dunkirk\", \"Interstellar\"])\n",
    "    else:\n",
    "        return _convert_docs(\n",
    "            [\"Crazy Rich Asians\", \"The Big Sick\", \"When Harry Met Sally\"]\n",
    "        )\n",
    "\n",
    "\n",
    "@traceable\n",
    "def execute_function_call(message):\n",
    "    if message.tool_calls[0].function.name == \"retrieve_movies\":\n",
    "        query = json.loads(message.tool_calls[0].function.arguments)[\"query\"]\n",
    "        results = retrieve_movies(query)\n",
    "    else:\n",
    "        results = (\n",
    "            f\"Error: function {message.tool_calls[0].function.name} does not exist\"\n",
    "        )\n",
    "    return results\n",
    "\n",
    "\n",
    "@traceable\n",
    "def generate_answer(question, context):\n",
    "    messages = [\n",
    "        {\n",
    "            \"role\": \"system\",\n",
    "            \"content\": f\"Answer the user's question based only on the content below:\\n\\n{context}\",\n",
    "        },\n",
    "        {\"role\": \"user\", \"content\": question},\n",
    "    ]\n",
    "    result = openai.chat.completions.create(\n",
    "        messages=messages, model=\"gpt-3.5-turbo\", temperature=0\n",
    "    )\n",
    "    return result.choices[0].message.content\n",
    "\n",
    "\n",
    "@traceable\n",
    "def rag_pipeline(chat_history, question):\n",
    "    message = generate_movie_search(chat_history, question)\n",
    "    if message.tool_calls is None:\n",
    "        return message.content\n",
    "    else:\n",
    "        docs = execute_function_call(message)\n",
    "        context = \"\\n\".join([doc[\"page_content\"] for doc in docs])\n",
    "        return generate_answer(question, context)\n",
    "\n",
    "\n",
    "@traceable\n",
    "def run_multi_turn(queries):\n",
    "    turns = queries\n",
    "    chat_history, outputs = [], []\n",
    "    for turn in turns:\n",
    "        output = rag_pipeline(chat_history, turn)\n",
    "        chat_history.append({\"role\": \"user\", \"content\": turn})\n",
    "        chat_history.append({\"role\": \"assistant\", \"content\": output})\n",
    "        outputs.append(output)\n",
    "    return outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d72a9b22-677b-4346-a497-a0bb03530e55",
   "metadata": {},
   "source": [
    "#### Evaluate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "3b74b731",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/gf/6rnp_mbx5914kx7qmmh7xzmw0000gn/T/ipykernel_37884/3314201279.py:9: UserWarning: Function evaluate is in beta.\n",
      "  results = evaluate(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View the evaluation results for experiment: 'Multi-turn eval:c9eee6b' at:\n",
      "https://smith.langchain.com/o/ebbaf2eb-769b-4505-aca2-d11de10372a4/datasets/3d3915cf-cc7b-4bf9-a583-32ff5e60069d/compare?selectedSessions=50f97dc4-5902-40d1-a071-2e77d06227de\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "35b24bed651043488b21b257508203df",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def brevity(run, example) -> dict:\n",
    "    convo = run.outputs.get(\"output\")\n",
    "    for turn in convo:\n",
    "        if len(turn) > 200:\n",
    "            return {\"score\": 0}\n",
    "    return {\"score\": 1}\n",
    "\n",
    "\n",
    "results = evaluate(\n",
    "    lambda inputs: run_multi_turn(inputs[\"queries\"]),\n",
    "    data=multi_turn_dataset_name,\n",
    "    evaluators=[brevity],\n",
    "    experiment_prefix=f\"Multi-turn eval\",\n",
    "    metadata={\n",
    "        \"model\": \"gpt-3.5-turbo\",\n",
    "        \"prompt_version\": \"003\",\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99e88a33-7aed-4912-99fc-202758ad02bf",
   "metadata": {},
   "source": [
    "## Pt. 3 -- Structured Inputs\n",
    "\n",
    "\n",
    "#### Create dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "3108df92-459e-4a0d-96fb-b435c88229e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "structured_input_examples = [\n",
    "    (\n",
    "        {\n",
    "            \"user_preferences\": [\"Sci-Fi\", \"Action\"],\n",
    "            \"watch_history\": [\"The Matrix\", \"Inception\"],\n",
    "            \"search_query\": \"What to watch next?\",\n",
    "        },\n",
    "        \"Based on your love for Sci-Fi and Action movies, and considering you've recently watched 'The Matrix' and 'Inception', you might enjoy 'Blade Runner 2049' for its deep narrative and stunning visuals.\",\n",
    "        # Example adding notes + metadata\n",
    "        {\"note\": \"This is a free-form note\"},\n",
    "    ),\n",
    "    (\n",
    "        {\n",
    "            \"user_preferences\": [\"Drama\", \"Historical\"],\n",
    "            \"watch_history\": [\"The Crown\", \"Downton Abbey\"],\n",
    "            \"search_query\": \"Looking for a movie with a strong storyline\",\n",
    "        },\n",
    "        \"Given your interest in Drama and Historical themes, and your watch history, 'The King's Speech' offers a compelling storyline with remarkable performances.\",\n",
    "        {\"note\": \"This is another free_form note.\", \"cohort_number\": 3},\n",
    "    ),\n",
    "    (\n",
    "        {\n",
    "            \"user_preferences\": [\"Comedy\", \"Romance\"],\n",
    "            \"watch_history\": [\"Friends\", \"The Big Bang Theory\"],\n",
    "            \"search_query\": \"Need a light-hearted movie\",\n",
    "        },\n",
    "        \"Considering your preference for Comedy and Romance, along with enjoying shows like 'Friends', you'd likely enjoy 'Crazy Rich Asians' for its humor and heartwarming romance.\",\n",
    "    ),\n",
    "    (\n",
    "        {\n",
    "            \"user_preferences\": [\"Thriller\", \"Mystery\"],\n",
    "            \"watch_history\": [\"Sherlock\", \"Mindhunter\"],\n",
    "            \"search_query\": \"Suggest a suspenseful movie\",\n",
    "        },\n",
    "        \"With your taste leaning towards Thriller and Mystery, and considering you've watched 'Sherlock' and 'Mindhunter', 'Gone Girl' would be an excellent choice for its suspense and plot twists.\",\n",
    "    ),\n",
    "    (\n",
    "        {\n",
    "            \"user_preferences\": [\"Documentary\", \"Nature\"],\n",
    "            \"watch_history\": [\"Planet Earth\", \"Blue Planet II\"],\n",
    "            \"search_query\": \"Want to watch something about wildlife\",\n",
    "        },\n",
    "        \"Your interest in Documentaries and Nature, along with watching 'Planet Earth' and 'Blue Planet II', suggests you would enjoy 'The Serengeti Rules', which beautifully captures wildlife and ecosystems.\",\n",
    "    ),\n",
    "    (\n",
    "        {\n",
    "            \"user_preferences\": [\"Fantasy\", \"Adventure\"],\n",
    "            \"watch_history\": [\"Harry Potter series\", \"The Hobbit\"],\n",
    "            \"search_query\": \"Fantasy movies for the weekend?\",\n",
    "        },\n",
    "        \"Given your love for Fantasy and Adventure, having watched the 'Harry Potter series' and 'The Hobbit', 'The Witcher' series would be a fantastic choice for your weekend binge.\",\n",
    "    ),\n",
    "    (\n",
    "        {\n",
    "            \"user_preferences\": [\"Animation\", \"Family\"],\n",
    "            \"watch_history\": [\"Finding Nemo\", \"Toy Story\"],\n",
    "            \"search_query\": \"Animated movies that are fun for all ages?\",\n",
    "        },\n",
    "        \"With a preference for Animation and Family-friendly content, and given your history with 'Finding Nemo' and 'Toy Story', 'Coco' is highly recommended for its fun story and universal appeal.\",\n",
    "    ),\n",
    "    (\n",
    "        {\n",
    "            \"user_preferences\": [\"Horror\", \"Supernatural\"],\n",
    "            \"watch_history\": [\"The Haunting of Hill House\", \"Stranger Things\"],\n",
    "            \"search_query\": \"Scary movies that aren’t too gory?\",\n",
    "        },\n",
    "        \"As a fan of Horror and Supernatural genres, and having enjoyed 'The Haunting of Hill House' and 'Stranger Things', 'A Quiet Place' offers suspense without relying on gore.\",\n",
    "    ),\n",
    "    (\n",
    "        {\n",
    "            \"user_preferences\": [\"Musical\", \"Drama\"],\n",
    "            \"watch_history\": [\"La La Land\", \"The Greatest Showman\"],\n",
    "            \"search_query\": \"Musicals with a strong emotional core?\",\n",
    "        },\n",
    "        \"Your enjoyment of Musicals and Drama, seen in 'La La Land' and 'The Greatest Showman', means you might find 'Les Misérables' to be a powerful experience with its deep emotional resonance.\",\n",
    "    ),\n",
    "    (\n",
    "        {\n",
    "            \"user_preferences\": [\"Crime\", \"Legal Drama\"],\n",
    "            \"watch_history\": [\"Breaking Bad\", \"Better Call Saul\"],\n",
    "            \"search_query\": \"Engaging legal dramas?\",\n",
    "        },\n",
    "        \"Considering your interest in Crime and Legal Drama, with 'Breaking Bad' and 'Better Call Saul' in your watch history, 'The Trial of the Chicago 7' is recommended for its engaging narrative and historical significance.\",\n",
    "    ),\n",
    "]\n",
    "\n",
    "structured_input_dataset_name = \"Structured Inputs\"\n",
    "if not client.has_dataset(dataset_name=structured_input_dataset_name):\n",
    "    structured_input_dataset = client.create_dataset(\n",
    "        dataset_name=structured_input_dataset_name\n",
    "    )\n",
    "    for input_tuple in structured_input_examples:\n",
    "        metadata = None\n",
    "        if len(input_tuple) == 3:\n",
    "            inputs, answer, metadata = input_tuple\n",
    "        else:\n",
    "            inputs, answer = input_tuple\n",
    "        client.create_example(\n",
    "            inputs=inputs,\n",
    "            outputs={\"answer\": answer},\n",
    "            dataset_id=structured_input_dataset.id,\n",
    "            metadata=metadata,\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "24d7864d-ecd3-4b7d-aabd-deccf45d2360",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "system_prompt_template = \"\"\"Respond to the user's search query given what you know about them.\n",
    "\n",
    "You know they just watched: {watch_history}\n",
    "\n",
    "You know they have explicited stated preferences for: {user_preferences}\"\"\"\n",
    "\n",
    "\n",
    "@traceable\n",
    "def generate_recommendation(search_query, watch_history, user_preferences):\n",
    "    system_prompt = system_prompt_template.format(\n",
    "        watch_history=watch_history, user_preferences=user_preferences\n",
    "    )\n",
    "    messages = [\n",
    "        {\"role\": \"system\", \"content\": system_prompt},\n",
    "    ] + [{\"role\": \"user\", \"content\": search_query}]\n",
    "    result = openai.chat.completions.create(\n",
    "        messages=messages,\n",
    "        model=\"gpt-3.5-turbo\",\n",
    "    )\n",
    "    return result.choices[0].message.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "688fb930",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/gf/6rnp_mbx5914kx7qmmh7xzmw0000gn/T/ipykernel_37884/2506121668.py:19: UserWarning: Function evaluate is in beta.\n",
      "  result = evaluate(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View the evaluation results for experiment: 'Recommendations:cb1089c' at:\n",
      "https://smith.langchain.com/o/ebbaf2eb-769b-4505-aca2-d11de10372a4/datasets/18b47447-9a35-4906-bacf-35548a0e8e57/compare?selectedSessions=168393b7-dbb0-4760-8814-af29e6708e40\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4f165192e9554a5794f33d9536737183",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from langsmith.evaluation import LangChainStringEvaluator\n",
    "\n",
    "\n",
    "# The evaluator expects a single input/prediction/reference.\n",
    "# Our dataset has multiple inputs.\n",
    "# You can configure the data the wrapped evaluator sees.\n",
    "def prepare_eval_inputs(run, example):\n",
    "    return {\n",
    "        \"input\": example.inputs[\"search_query\"],\n",
    "        \"prediction\": run.outputs[\"output\"],\n",
    "        \"reference\": example.outputs[\"answer\"],\n",
    "    }\n",
    "\n",
    "\n",
    "correctness_evaluator = LangChainStringEvaluator(\n",
    "    \"cot_qa\", prepare_data=prepare_eval_inputs\n",
    ")\n",
    "structured_input_dataset_name = \"Structured Inputs\"\n",
    "result = evaluate(\n",
    "    lambda inputs: generate_recommendation(**inputs),\n",
    "    data=structured_input_dataset_name,\n",
    "    evaluators=[correctness_evaluator],\n",
    "    experiment_prefix=\"Recommendations\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "623bb70a",
   "metadata": {},
   "source": [
    "## Pt. 4 -- Dataset Versioning & Metadata\n",
    "\n",
    "Every time an example is created, updated, or deleted, a new dataset version is saved and can be\n",
    "retrieved by querying the examples `as_of` that modified time.\n",
    "\n",
    "You can save \"semantic\" versions of the dataset by tagging specific times with names.\n",
    "\n",
    "A tag can be assigned to at most 1 version at a time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "aa1d4408",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import datetime\n",
    "\n",
    "examples = list(client.list_examples(dataset_name=toxic_dataset_name))\n",
    "initial_time = max([e.modified_at for e in examples])\n",
    "len(examples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "d2554c19",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "example = client.create_example(\n",
    "    inputs={\"text\": \"hi there\"},\n",
    "    outputs={\"label\": \"Not toxic\"},\n",
    "    metadata={\"recent\": True},\n",
    "    dataset_name=toxic_dataset_name,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "eeb2f1c4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(\n",
    "    list(\n",
    "        client.list_examples(\n",
    "            dataset_name=toxic_dataset_name,\n",
    "            as_of=datetime.datetime.now(tz=datetime.timezone.utc),\n",
    "        )\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "88b9e929",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check for the time at which we first ran\n",
    "len(\n",
    "    list(\n",
    "        client.list_examples(\n",
    "            dataset_name=toxic_dataset_name,\n",
    "            as_of=initial_time,\n",
    "        )\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "443605c6-9136-4e9a-a66e-cb36fd470610",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# You can tag a specific dataset version with a semantic name, like \"prod\"\n",
    "client.update_dataset_tag(\n",
    "    dataset_name=toxic_dataset_name, as_of=initial_time, tag=\"prod\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "337aa46d-b975-4b25-b029-931c0690cc79",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You can then query the dataset for that version\n",
    "len(\n",
    "    list(\n",
    "        client.list_examples(\n",
    "            dataset_name=toxic_dataset_name,\n",
    "            as_of=\"prod\",\n",
    "        )\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "e9c4b724-4f10-497e-82b6-bc113ec098b6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "examples_modified=[] examples_added=[UUID('0a602ee7-e438-4ee6-a616-e7807cf2a373')] examples_removed=[]\n"
     ]
    }
   ],
   "source": [
    "from_version = \"prod\"\n",
    "to_version = \"latest\"\n",
    "diff = client.diff_dataset_versions(\n",
    "    dataset_name=toxic_dataset_name,\n",
    "    from_version=from_version,\n",
    "    to_version=to_version,\n",
    ")\n",
    "print(diff)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "ecad6113-f6c1-4aad-a4a7-551de71b9a37",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/gf/6rnp_mbx5914kx7qmmh7xzmw0000gn/T/ipykernel_37884/1734478713.py:3: UserWarning: Function evaluate is in beta.\n",
      "  result = evaluate(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View the evaluation results for experiment: 'dataset versioning example:856cee9' at:\n",
      "https://smith.langchain.com/o/ebbaf2eb-769b-4505-aca2-d11de10372a4/datasets/dc6cc406-d05e-4e57-9c47-5868523f5a98/compare?selectedSessions=b11e9800-ae10-47b7-ae67-54b8f21b8da4\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f9d3e51c5e374c06ba4c17089246aa78",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# You can then use tags to continue to evaluate on the same version of a dataset\n",
    "# Only updating your testing flow once you are ready to commit to a new version\n",
    "result = evaluate(\n",
    "    lambda inputs: label_query(**inputs),\n",
    "    data=client.list_examples(dataset_name=toxic_dataset_name, as_of=\"prod\"),\n",
    "    evaluators=[correct_label],\n",
    "    summary_evaluators=[summary_eval],\n",
    "    experiment_prefix=\"dataset versioning example\",\n",
    "    metadata={\n",
    "        \"prompt_version\": \"001\",\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1275901",
   "metadata": {},
   "source": [
    "# Pt. 5 -- Proxy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "0506d0e5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langsmith import traceable, wrappers\n",
    "from openai import OpenAI\n",
    "\n",
    "openai = wrappers.wrap_openai(\n",
    "    OpenAI(\n",
    "        base_url=\"http://localhost:8080/proxy/openai\",\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "454ef8c4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "system_prompt = \"\"\"Generate a three paragraph description of a movie about this topic: {topic}. Do not specify a title.\"\"\"\n",
    "\n",
    "\n",
    "@traceable\n",
    "def generate_movie(topic):\n",
    "    messages = [\n",
    "        {\"role\": \"user\", \"content\": system_prompt.format(topic=topic)},\n",
    "    ]\n",
    "    result = openai.chat.completions.create(messages=messages, model=\"gpt-4\")\n",
    "    return result.choices[0].message.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "f68bbbf0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "@traceable\n",
    "def generate_title(description):\n",
    "    messages = [\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": f\"Generate a title for the following movie description:\\n\\n{description}.\",\n",
    "        },\n",
    "    ]\n",
    "    result = openai.chat.completions.create(messages=messages, model=\"gpt-4\")\n",
    "    return result.choices[0].message.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "b2734044",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "@traceable\n",
    "def pipeline(topic):\n",
    "    description = generate_movie(topic)\n",
    "    title = generate_title(description)\n",
    "    return {\"description\": description, \"title\": title}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "25038102",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "movie_creation_examples = [\"soccer\", \"a pop star\", \"action movie in venice\"]\n",
    "\n",
    "movie_creation_dataset_name = \"Movie Creation\"\n",
    "if not client.has_dataset(dataset_name=movie_creation_dataset_name):\n",
    "    movie_dataset = client.create_dataset(dataset_name=movie_creation_dataset_name)\n",
    "    for topic in movie_creation_examples:\n",
    "        client.create_example(inputs={\"topic\": topic}, dataset_id=movie_dataset.id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "f25215f2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View the evaluation results for project 'cold cache' at:\n",
      "https://smith.langchain.com/o/8d28a774-8361-496d-a5d4-dd582a8d1b10/datasets/26db3352-e870-48b3-910b-e3cd003b0ab4/compare?selectedSessions=d8292996-db53-4ea8-92fd-6b0a3b26ec30\n",
      "\n",
      "View all tests for Dataset Movie Creation at:\n",
      "https://smith.langchain.com/o/8d28a774-8361-496d-a5d4-dd582a8d1b10/datasets/26db3352-e870-48b3-910b-e3cd003b0ab4\n",
      "[------------------------------------------------->] 3/3"
     ]
    }
   ],
   "source": [
    "result = evaluate(\n",
    "    lambda inputs: pipeline(**inputs),\n",
    "    data=movie_creation_dataset_name,\n",
    "    experiment_prefix=\"cold cache\",\n",
    "    metadata={\n",
    "        \"prompt_version\": \"1\",\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "ad7ae07e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "@traceable\n",
    "def generate_title(description):\n",
    "    messages = [\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": f\"Generate a title in SPANISH for the following movie description:\\n\\n{description}.\",\n",
    "        },\n",
    "    ]\n",
    "    result = openai.chat.completions.create(messages=messages, model=\"gpt-4\")\n",
    "    return result.choices[0].message.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "09b5eeb8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "@traceable\n",
    "def pipeline(topic):\n",
    "    description = generate_movie(topic)\n",
    "    title = generate_title(description)\n",
    "    return {\"description\": description, \"title\": title}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "27846f9b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View the evaluation results for project 'warm cache' at:\n",
      "https://smith.langchain.com/o/8d28a774-8361-496d-a5d4-dd582a8d1b10/datasets/26db3352-e870-48b3-910b-e3cd003b0ab4/compare?selectedSessions=e5b69871-cd7f-433b-a29f-4a75bf7b4094\n",
      "\n",
      "View all tests for Dataset Movie Creation at:\n",
      "https://smith.langchain.com/o/8d28a774-8361-496d-a5d4-dd582a8d1b10/datasets/26db3352-e870-48b3-910b-e3cd003b0ab4\n",
      "[------------------------------------------------->] 3/3"
     ]
    }
   ],
   "source": [
    "result = evaluate(\n",
    "    lambda inputs: pipeline(inputs[\"topic\"]),\n",
    "    data=movie_creation_dataset_name,\n",
    "    experiment_prefix=\"warm cache\",\n",
    "    metadata={\n",
    "        \"prompt_version\": \"2\",\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cbfdaa42-0329-4bb5-92a8-dba19d6810ba",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}