fine-tuning-examples/lilac/lilac.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Curate fine-tuning data with Lilac\n",
    "[![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langsmith-cookbook/blob/main/fine-tuning-examples/lilac/lilac.ipynb)\n",
    "\n",
    "Lilac is an open-source product that helps you analyze, structure, and clean unstructured data with AI. You can use it to enrich datasets of LangChain runs to create better fine-tuning datasets.\n",
    "\n",
    "In this walkthrough, we will use Lilac on a dataset of LangSmith runs to check for PII and remove approximate duplicates before fine-tuning. The overall workflow looks something like the following:\n",
    "\n",
    "![Workflow](./img/workflow.png)\n",
    "\n",
    "The basic workflow is as follows:\n",
    "\n",
    "- Create a LangSmith dataset of runs data.\n",
    "- Load LangSmith dataset into Lilac.\n",
    "- Filter and curate dataset using signals and concepts.\n",
    "- Export the dataset for fine-tuning.\n",
    "\n",
    "We will explain each of these steps in more detail below, but first, install some prerequisite packages.\n",
    "\n",
    "## Setup\n",
    "\n",
    "In addition to Lilac and LangSmith, this walkthrough requires a couple of additional packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %pip install -U \"lilac[pii]\" langdetect openai langchain --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import uuid\n",
    "import os\n",
    "\n",
    "# os.environ[\"LANGCHAIN_API_KEY\"] = \"<YOUR-API-KEY>\"\n",
    "unique_id = uuid.uuid4().hex[:8]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1: Create LangSmith dataset\n",
    "\n",
    "We've included an example dataset in this repository that you can use to complete this walkthrough.\n",
    "\n",
    "This dataset was made by querying prompt and LLM runs from an example deployment of [chat langchain](https://github.com/langchain-ai/chat-langchain). \n",
    "\n",
    "For more information on how to query runs in LangSmith, check out the [docs](https://docs.smith.langchain.com/tracing/faq/querying_traces) or explore some of the other recipes in this cookbook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langsmith import Client\n",
    "\n",
    "client = Client()\n",
    "dataset_name = f\"langsmith-prompt-runs-{unique_id}\"\n",
    "ds = client.create_dataset(dataset_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from concurrent.futures import ThreadPoolExecutor\n",
    "\n",
    "\n",
    "def create_example(line: str):\n",
    "    d = json.loads(line)\n",
    "    client.create_example(inputs=d[\"inputs\"], outputs=d[\"outputs\"], dataset_id=ds.id)\n",
    "\n",
    "\n",
    "with open(\"rag.jsonl\", \"r\", encoding=\"utf-8\") as f:\n",
    "    with ThreadPoolExecutor(max_workers=10) as executor:\n",
    "        executor.map(create_example, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can create the dataset. Lilac works best on flat dataset structures, so we will flatten (and stringify) some of the attributes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Import into Lilac\n",
    "\n",
    "Next, we can import the LangSmith dataset into Lilac. Select the dataset name you created above, \n",
    "and run the code below. Once you've run the code, you can view the the results in Lilac's UI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from IPython.display import display\n",
    "import lilac as ll"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Reading from source langsmith...: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 400/400 [00:00<00:00, 54422.01it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset \"langsmith-prompt-runs-d19f7f5b\" written to ./langsmith-finetune/datasets/local/langsmith-prompt-runs-d19f7f5b\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:     Started server process [18508]\n",
      "INFO:     Waiting for application startup.\n"
     ]
    }
   ],
   "source": [
    "ll.set_project_dir(\"./langsmith-finetune\")\n",
    "\n",
    "data_source = ll.sources.langsmith.LangSmithSource(\n",
    "    dataset_name=dataset_name,\n",
    ")\n",
    "\n",
    "config = ll.DatasetConfig(\n",
    "    namespace=\"local\",\n",
    "    name=dataset_name,\n",
    "    source=data_source,\n",
    ")\n",
    "\n",
    "dataset = ll.create_dataset(config)\n",
    "ll.start_server()\n",
    "# await ll.stop_server()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3: Enrich Dataset\n",
    "\n",
    "Now that we have our dataset in Lilac, we can run Lilac’s signals, concepts and labels to help organize and filter the dataset. Our goal is to select distinct examples demonstrating good language model generations for a variety of input types. You can explore and annotate the dataset in the app by navigating to the URL printed out by the local server above. I'd encourage you to try out their off-the-shelf \"concepts\" or try training your own.\n",
    "\n",
    "For the sake of this walkthrough, we will focus on using the Python API. You can follow along with the code below.\n",
    "\n",
    "#### Applying 'signals'\n",
    "\n",
    "Signals in Lilac refer to any function that is applied over a field. We will use a couple off-the-shelf \"signals\" to perform the following:\n",
    "\n",
    "- PII detection: we don't want to leak private data\n",
    "- Near duplicate detection: we want each training example to be informative\n",
    "\n",
    "These are useful for filtering bad examples from our dataset before fine-tuning a model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Computing pii on local/langsmith-prompt-runs-d19f7f5b:('question',): 100%|████████████████████████████████████████████████████████████████████████████████████| 400/400 [00:00<00:00, 909.97it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing signal \"pii\" on local/langsmith-prompt-runs-d19f7f5b:('question',) took 0.441s.\n",
      "Wrote signal output to ./langsmith-finetune/datasets/local/langsmith-prompt-runs-d19f7f5b/question/pii\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Computing pii on local/langsmith-prompt-runs-d19f7f5b:('output',): 100%|██████████████████████████████████████████████████████████████████████████████████████| 400/400 [00:00<00:00, 421.69it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing signal \"pii\" on local/langsmith-prompt-runs-d19f7f5b:('output',) took 0.950s.\n",
      "Wrote signal output to ./langsmith-finetune/datasets/local/langsmith-prompt-runs-d19f7f5b/output/pii\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Computing near_dup on local/langsmith-prompt-runs-d19f7f5b:('question',):   0%|                                                                                          | 0/400 [00:00<?, ?it/s]\n",
      "Fingerprinting...: 400it [00:00, 17717.11it/s]\n",
      "\n",
      "Computing hash collisions...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 865.16it/s]\u001b[A\n",
      "\n",
      "Clustering...: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 13/13 [00:00<00:00, 23133.62it/s]\u001b[A\n",
      "Computing near_dup on local/langsmith-prompt-runs-d19f7f5b:('question',): 100%|██████████████████████████████████████████████████████████████████████████████| 400/400 [00:00<00:00, 6097.39it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing signal \"near_dup\" on local/langsmith-prompt-runs-d19f7f5b:('question',) took 0.067s.\n",
      "Wrote signal output to ./langsmith-finetune/datasets/local/langsmith-prompt-runs-d19f7f5b/question/near_dup\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Computing near_dup on local/langsmith-prompt-runs-d19f7f5b:('output',):   0%|                                                                                            | 0/400 [00:00<?, ?it/s]\n",
      "Fingerprinting...: 391it [00:00, 4102.89it/s]\n",
      "\n",
      "Computing hash collisions...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 782.52it/s]\u001b[A\n",
      "\n",
      "Clustering...: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 13/13 [00:00<00:00, 53248.00it/s]\u001b[A\n",
      "Computing near_dup on local/langsmith-prompt-runs-d19f7f5b:('output',): 100%|████████████████████████████████████████████████████████████████████████████████| 400/400 [00:00<00:00, 2896.51it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing signal \"near_dup\" on local/langsmith-prompt-runs-d19f7f5b:('output',) took 0.139s.\n",
      "Wrote signal output to ./langsmith-finetune/datasets/local/langsmith-prompt-runs-d19f7f5b/output/near_dup\n"
     ]
    }
   ],
   "source": [
    "dataset.compute_signal(ll.PIISignal(), \"question\")\n",
    "dataset.compute_signal(ll.PIISignal(), \"output\")\n",
    "\n",
    "# Apply min-hash LSH (https://en.wikipedia.org/wiki/MinHash) to detect approximate n-gram duplicates\n",
    "dataset.compute_signal(ll.NearDuplicateSignal(), \"question\")\n",
    "dataset.compute_signal(ll.NearDuplicateSignal(), \"output\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding labels\n",
    "\n",
    "Labeling is best done in-app, but you can also programmatically [label rows using the python SDK](https://docs.lilacml.com/datasets/dataset_labels.html). Below is an example that labels all rows not tagged as English as `not_english`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Computing lang_detection on local/langsmith-prompt-runs-d19f7f5b:('question',): 100%|█████████████████████████████████████████████████████████████████████████| 400/400 [00:00<00:00, 820.34it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing signal \"lang_detection\" on local/langsmith-prompt-runs-d19f7f5b:('question',) took 0.494s.\n",
      "Wrote signal output to ./langsmith-finetune/datasets/local/langsmith-prompt-runs-d19f7f5b/question/lang_detection\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Computing lang_detection on local/langsmith-prompt-runs-d19f7f5b:('output',): 100%|███████████████████████████████████████████████████████████████████████████| 400/400 [00:00<00:00, 502.81it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing signal \"lang_detection\" on local/langsmith-prompt-runs-d19f7f5b:('output',) took 0.797s.\n",
      "Wrote signal output to ./langsmith-finetune/datasets/local/langsmith-prompt-runs-d19f7f5b/output/lang_detection\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "dataset.compute_signal(ll.LangDetectionSignal(), \"question\")\n",
    "dataset.compute_signal(ll.LangDetectionSignal(), \"output\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can check the current schema by running the following. Select the fields you want to export.\n",
    "# dataset.manifest()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.add_labels(\n",
    "    \"not_english\", filters=[((\"question\", \"lang_detection\"), \"not_equal\", \"en\")]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lilac has a lot more powerful capabilities like custom concepts and signals that you can apply. Check out their [docs](https://www.lilacml.com/) for more info, and see our [exploratory data analysis](../../exploratory-data-analysis/lilac/lilac.ipynb) noteboook for an introduction on using them with LangSmith datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Prepare the enriched dataset\n",
    "\n",
    "Now let's prepare the dataset for fine-tuning, we will fetch the deduplicated rows and filter out any rows that may contain PII."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original length: 400\n",
      "Filtered length: 314\n"
     ]
    }
   ],
   "source": [
    "df = dataset.to_pandas(\n",
    "    [\n",
    "        \"question\",\n",
    "        \"chat_history\",\n",
    "        \"context\",\n",
    "        \"output\",\n",
    "        \"question.pii\",\n",
    "        \"question.near_dup\",\n",
    "        \"user_score\",\n",
    "        \"not_english\",\n",
    "    ]\n",
    ")\n",
    "\n",
    "print(f\"Original length: {len(df)}\")\n",
    "\n",
    "# Flatten the dataframe\n",
    "df[\"cluster_id\"] = df[\"question.near_dup\"].apply(lambda x: x[\"cluster_id\"])\n",
    "df[\"contains_pii\"] = df[\"question.pii\"].apply(\n",
    "    lambda x: bool([v for l in x.values() for v in l])\n",
    ")\n",
    "df[\"not_english\"] = df[\"not_english\"].apply(\n",
    "    lambda x: x is not None and x.get(\"label\") == \"true\"\n",
    ")\n",
    "# Drop original dotted columns\n",
    "df.drop(columns=[\"question.near_dup\", \"question.pii\"], inplace=True)\n",
    "# Now filter for only rows for which contains_pii is false, user_score is 1.0\n",
    "df = df[(~df[\"contains_pii\"]) & (df[\"user_score\"] != \"0.0\") & (~df[\"output\"].isna())]\n",
    "# And drop the duplicate cluster IDs\n",
    "df = df.drop_duplicates(subset=\"cluster_id\", keep=\"first\")\n",
    "print(f\"Filtered length: {len(df)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>chat_history</th>\n",
       "      <th>context</th>\n",
       "      <th>output</th>\n",
       "      <th>user_score</th>\n",
       "      <th>not_english</th>\n",
       "      <th>cluster_id</th>\n",
       "      <th>contains_pii</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\"J'ai ajout\\u00e9 \\\"insurer\\\" au metadata de d...</td>\n",
       "      <td>[{\"content\": \"I tried to make a chatbot to hel...</td>\n",
       "      <td>\"&lt;doc id='0'&gt;Skip to main content\\ud83e\\udd9c\\...</td>\n",
       "      <td>\"Je m'excuse pour la confusion. Il semble que ...</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>\"Show me how to use a RecursiveUrlLoader?\"</td>\n",
       "      <td>[{\"content\": \"Show me how to use a RecursiveUr...</td>\n",
       "      <td>[{\"metadata\": {\"source\": \"http://www.hernandez...</td>\n",
       "      <td>\"I'm sorry, but the provided context does not ...</td>\n",
       "      <td>None</td>\n",
       "      <td>False</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"Embody the persona of Melinda Reed, my expert...</td>\n",
       "      <td>[]</td>\n",
       "      <td>\"&lt;doc id='0'&gt;history, so anything important mu...</td>\n",
       "      <td>\"```markdown\\n# Project Skeleton\\n\\n## File St...</td>\n",
       "      <td>None</td>\n",
       "      <td>False</td>\n",
       "      <td>2</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question  \\\n",
       "0  \"J'ai ajout\\u00e9 \\\"insurer\\\" au metadata de d...   \n",
       "1         \"Show me how to use a RecursiveUrlLoader?\"   \n",
       "2  \"Embody the persona of Melinda Reed, my expert...   \n",
       "\n",
       "                                        chat_history  \\\n",
       "0  [{\"content\": \"I tried to make a chatbot to hel...   \n",
       "1  [{\"content\": \"Show me how to use a RecursiveUr...   \n",
       "2                                                 []   \n",
       "\n",
       "                                             context  \\\n",
       "0  \"<doc id='0'>Skip to main content\\ud83e\\udd9c\\...   \n",
       "1  [{\"metadata\": {\"source\": \"http://www.hernandez...   \n",
       "2  \"<doc id='0'>history, so anything important mu...   \n",
       "\n",
       "                                              output user_score  not_english  \\\n",
       "0  \"Je m'excuse pour la confusion. Il semble que ...       None         True   \n",
       "1  \"I'm sorry, but the provided context does not ...       None        False   \n",
       "2  \"```markdown\\n# Project Skeleton\\n\\n## File St...       None        False   \n",
       "\n",
       "   cluster_id  contains_pii  \n",
       "0           0         False  \n",
       "1           1         False  \n",
       "2           2         False  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Finetune\n",
    "\n",
    "With the dataset filtered, we can now prepare it to a compatible format for fine-tuning.\n",
    "We will use OpenAI's fine-tuning endpoint for this, but you could also apply similar logic to finetune a Llama, T5, or other model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_messages(row):\n",
    "    chat_history = json.loads(row.chat_history or \"[]\") or []\n",
    "    roles = (\"assistant\", \"user\")\n",
    "    messages = [\n",
    "        {\"role\": \"system\", \"content\": \"Helpfully answer the questions about LangChain.\"}\n",
    "    ]\n",
    "    for i, msg in enumerate(chat_history):\n",
    "        messages.append({\"role\": roles[i % 2], \"content\": str(msg[\"content\"])})\n",
    "    messages.append({\"role\": \"user\", \"content\": row.question})\n",
    "    messages.append({\"role\": \"assistant\", \"content\": row.output})\n",
    "    return messages\n",
    "\n",
    "\n",
    "messages = df.apply(create_messages, axis=1).tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can fine-tune the model! This will take a while (20+ minutes), so we'd encourage you to further explore your local Lilac dataset\n",
    "while you wait."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File file-HtPWASZzc0LLlvEY7ENy0q13 ready after 106.96 seconds.\n",
      "Status=[running]... 1589.27s\r"
     ]
    }
   ],
   "source": [
    "import json\n",
    "from io import BytesIO\n",
    "import time\n",
    "\n",
    "import openai\n",
    "\n",
    "# We will write the jsonl file in memory\n",
    "my_file = BytesIO()\n",
    "for m in messages:\n",
    "    my_file.write((json.dumps({\"messages\": m}) + \"\\n\").encode(\"utf-8\"))\n",
    "\n",
    "my_file.seek(0)\n",
    "training_file = openai.File.create(file=my_file, purpose=\"fine-tune\")\n",
    "\n",
    "# OpenAI audits each training file for compliance reasons.\n",
    "# This make take a few minutes\n",
    "status = openai.File.retrieve(training_file.id).status\n",
    "start_time = time.time()\n",
    "while status != \"processed\":\n",
    "    print(f\"Status=[{status}]... {time.time() - start_time:.2f}s\", end=\"\\r\", flush=True)\n",
    "    time.sleep(5)\n",
    "    status = openai.File.retrieve(training_file.id).status\n",
    "print(f\"File {training_file.id} ready after {time.time() - start_time:.2f} seconds.\")\n",
    "\n",
    "job = openai.FineTuningJob.create(\n",
    "    training_file=training_file.id,\n",
    "    model=\"gpt-3.5-turbo\",\n",
    ")\n",
    "\n",
    "status = openai.FineTuningJob.retrieve(job.id).status\n",
    "start_time = time.time()\n",
    "while status != \"succeeded\":\n",
    "    print(f\"Status=[{status}]... {time.time() - start_time:.2f}s\", end=\"\\r\", flush=True)\n",
    "    time.sleep(5)\n",
    "    job = openai.FineTuningJob.retrieve(job.id)\n",
    "    status = job.status"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use fine-tuned model\n",
    "\n",
    "With the model fine-tuning complete, you can load the fine-tuned model directly in LangChain!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:langchain.llms.base:Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised ServiceUnavailableError: The server is overloaded or not ready yet..\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'\"The LangChain Expression Language is a domain-specific language (DSL) designed to work with LangChain. It allows users to create custom expressions and scripts for various purposes, such as data processing, text manipulation, and automation.\\\\n\\\\nWith the LangChain Expression Language, users can leverage a wide range of functions, operators, and variables to build complex and dynamic expressions. These expressions can be used within LangChain to transform data, generate text, make decisions, and perform other operations.\\\\n\\\\nOverall, the LangChain Expression Language provides a flexible and powerful tool for users to customize and extend the functionality of LangChain according to their specific needs.\"'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.chat_models import ChatOpenAI\n",
    "from langchain.prompts import ChatPromptTemplate\n",
    "\n",
    "model = ChatOpenAI(\n",
    "    model=job.fine_tuned_model,\n",
    "    temperature=1,\n",
    ")\n",
    "prompt = ChatPromptTemplate.from_messages(\n",
    "    [(\"system\", \"Helpfully answer the questions about LangChain.\"), (\"user\", \"{input}\")]\n",
    ")\n",
    "chain = prompt | model\n",
    "chain.invoke({\"input\": \"What's LangChain Expression Language?\"}).content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "LangSmith makes it easy to collect unstructured data seen by your production LLM application. Lilac can make it easier to filter and analyze with sophisticated methods.\n",
    "\n",
    "In this tutorial you created a dataset of run traces, filtered by near-duplicates and looking for PII, then used the filtered dataset to fine-tune a new model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}