pinecone-langchain-csv-dataloader-1.0.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ```.csv``` loader and splitter\n",
    "\n",
    "1. Create ```/csvs``` directory then place all .docx files inside \n",
    "2. Run app and all ```.csv``` files within the directory will be loaded into your vector store\n",
    "3. Use helper function to delet db \n",
    "4. Use Chat functions to test\n",
    "5. Trace using LangServe\n",
    "---\n",
    "\n",
    "### Directory Structure\n",
    "\n",
    "```\n",
    "├── pinecone-langchain-csv-dataloader.ipynb\n",
    "├── pdfs\n",
    "│   ├── client_file_01.csv\n",
    "│   ├── client_file_02.csv\n",
    "│   ├── client_file_03.csv\n",
    "```\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. ```check_and_load_csv_from_dir``` : checks directory for files,loads documents with the ```DirectoryLoader```, then splits the docs with langchains ```RecursiveCharacterTextSplitter```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install --upgrade --quiet  docx2txt\n",
    "! pip install langchain\n",
    "! pip install tiktoken\n",
    "! pip install pinecone-client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://api.python.langchain.com/en/latest/document_loaders/langchain_community.document_loaders.directory.DirectoryLoader.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DOCX - Check Path ald Load .pdf\n",
    "\n",
    "import os\n",
    "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
    "from langchain_community.document_loaders import DirectoryLoader\n",
    "\n",
    "def check_and_load_pdf_from_dir(directory):\n",
    "    # Ensure the directory path exists\n",
    "    if not os.path.exists(directory):\n",
    "        print(\"Directory does not exist.\")\n",
    "        return False\n",
    "\n",
    "    # Check if directory is actually a directory\n",
    "    if not os.path.isdir(directory):\n",
    "        print(\"The specified path is not a directory.\")\n",
    "        return False\n",
    "\n",
    "    # List all files in the directory\n",
    "    all_files = os.listdir(directory)\n",
    "    print(f'.csv files within the /csvs directory: {all_files}')\n",
    "    \n",
    "    # Check if each file ends with .docx\n",
    "    for file in all_files:\n",
    "        if not file.endswith(\".csv\"):\n",
    "            print(f\"Non-CSV file found: {file}\")\n",
    "            return False\n",
    " \n",
    " # load directory path\n",
    "    directory_path = directory \n",
    "    # add list comprehension for file os.path.join usage\n",
    "    loader = DirectoryLoader(directory_path,  glob=\"**/*.csv\")\n",
    "    \n",
    "    documents = loader.load()\n",
    "\n",
    "\n",
    "    text_splitter = RecursiveCharacterTextSplitter( \n",
    "        chunk_size=1000,  # Maximum size of each chunk\n",
    "        chunk_overlap=100,  # Number of overlapping characters between chunks\n",
    "    )\n",
    "\n",
    "    chunks = text_splitter.split_documents(documents)\n",
    "\n",
    "    return chunks\n",
    "\n",
    "chunks = check_and_load_pdf_from_dir('csvs')\n",
    "\n",
    "chunks\n",
    "# # Call Chunks \n",
    "# print(len(chunks))\n",
    "# print(chunks[0].page_content)\n",
    "# print(chunks[1].page_content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Tokens: 99487\n",
      "Embedding Cost in USD:0.039795\n"
     ]
    }
   ],
   "source": [
    "# Split Data Text With Cost Calculation\n",
    "# How much it costs to embed\n",
    "def calculate_and_display_embedding_cost(texts):\n",
    "    import tiktoken\n",
    "    enccoding = tiktoken.encoding_for_model(\"gpt-3.5-turbo\")\n",
    "    total_tokens = sum([len(enccoding.encode(page.page_content)) for page in texts])\n",
    "    print(f'Total Tokens: {total_tokens}')\n",
    "    print(f'Embedding Cost in USD:{total_tokens / 1000 * 0.0004:.6f}')\n",
    "\n",
    "calculate_and_display_embedding_cost(chunks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3. Create and delete pinecone index functions\n",
    "- ```delete_index_with_same_name```: deletes pincone index of the same name \n",
    "- ```load_or_create_embeddings_index```: If the index already exists it will just load data. If the idex is brand new it will create an then load the data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pinecone import Pinecone, ServerlessSpec\n",
    "from langchain_openai import OpenAIEmbeddings\n",
    "from langchain_pinecone import PineconeVectorStore\n",
    "import time\n",
    "\n",
    "pc = Pinecone(\n",
    "    api_key=os.environ.get(\"PINECONE_API_KEY\") or 'PINECONE_API_KEY'\n",
    ")\n",
    "\n",
    "def delete_index_with_same_name(index_name): \n",
    "    \n",
    "    # Delete index if any incdexes of the same name are present\n",
    "    if index_name in pc.list_indexes().names():\n",
    "        print(f'Deleting the {index_name} vector database')\n",
    "        pc.delete_index(index_name)\n",
    "\n",
    "\n",
    "def load_or_create_embeddings_index(index_name, chunks, namespace):\n",
    "    \n",
    "    if index_name in pc.list_indexes().names():\n",
    "        print(f'Index {index_name} already exists. Loading embeddings...', end='')\n",
    "        vector_store = PineconeVectorStore.from_documents(\n",
    "        documents=chunks, embedding=OpenAIEmbeddings(), index_name=index_name, namespace=namespace)\n",
    "        \n",
    "        while not pc.describe_index(index_name).status['ready']:\n",
    "            time.sleep(1)\n",
    "        \n",
    "        print('Done')\n",
    "    else:\n",
    "        print(f'Creating index {index_name} and embeddings ...', end = '')\n",
    "        pc.create_index(name=index_name, dimension=1536, metric='cosine',  spec=ServerlessSpec(\n",
    "                cloud='aws',\n",
    "                region='us-west-2'\n",
    "            ))\n",
    "        \n",
    "        while not pc.describe_index(index_name).status['ready']:\n",
    "            time.sleep(1)\n",
    "        # Add to vectorDB using LangChain \n",
    "        vector_store = PineconeVectorStore.from_documents(\n",
    "        documents=chunks, embedding=OpenAIEmbeddings(), index_name=index_name, namespace=namespace)\n",
    "        print('Done')   \n",
    "    return vector_store"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2. ```calculate_and_display_embedding_cost```: calculate embedding cost using using tiktoken\n",
    "- https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. Create your index and load all data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index csv-rag-test-2 already exists. Loading embeddings...Done\n"
     ]
    }
   ],
   "source": [
    "index_name='csv-rag-test-2'\n",
    "chunks = chunks\n",
    "namespace = \"csv_documents\"\n",
    "\n",
    "vector_store = load_or_create_embeddings_index(index_name=index_name, chunks=chunks, namespace=namespace)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5. Delete your index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete vector store \n",
    "index_name='csv-rag-test-1'\n",
    "delete_index_with_same_name(index_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6. Set up a LCEL conversation chain to test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langsmith import traceable\n",
    "from openai import Client\n",
    "\n",
    "openai = Client()\n",
    "\n",
    "# Experiment 3\n",
    "# Create Retrieval Chain Q+A Chain Chaining Retrieval with Create retrival/hitory \n",
    "\n",
    "@traceable(\n",
    "    run_type=\"llm\",\n",
    "    project_name=\"csv-rag-test-1\"\n",
    ")\n",
    "def create_history_aware_retriever_with_hub(vector_store, question, chat_history=[]):\n",
    "    from langchain_community.chat_models import ChatOpenAI\n",
    "    from langchain.chains.combine_documents import create_stuff_documents_chain\n",
    "    from langchain.chains import create_history_aware_retriever\n",
    "    from langchain.chains import create_retrieval_chain\n",
    "    from langchain.prompts import ChatPromptTemplate\n",
    "    from langchain_core.prompts import MessagesPlaceholder\n",
    "    from langchain import hub\n",
    "   \n",
    "    # https://smith.langchain.com/hub/langchain-ai/retrieval-qa-chat?organizationId=80b05ae8-a524-5b5f-b4d7-36207f821772\n",
    "    rephrase_prompt = hub.pull(\"langchain-ai/chat-langchain-rephrase\")\n",
    "\n",
    "    llm = ChatOpenAI(temperature=1)\n",
    "    \n",
    "    # Grab your Pinecone and set the search type    \n",
    "    retriever = vector_store.as_retriever(search_type='similarity', search_kwargs={'k':3})\n",
    "    \n",
    "    # Chain 1    \n",
    "    chat_retriever_chain = create_history_aware_retriever(\n",
    "        llm, retriever, rephrase_prompt\n",
    "    )\n",
    "    \n",
    "    qa_system_prompt = \"\"\"You are an assistant for question-answering tasks. \\\n",
    "    Use the following pieces of retrieved context to answer the question. \\\n",
    "    If you don't know the answer, just say that you don't know. \\\n",
    "    Use three sentences maximum and keep the answer concise.\\\n",
    "\n",
    "    {context}\"\"\"\n",
    "\n",
    "    qa_prompt = ChatPromptTemplate.from_messages(\n",
    "        [\n",
    "            (\"system\", qa_system_prompt),\n",
    "            MessagesPlaceholder(\"chat_history\"),\n",
    "            (\"human\", \"{input}\"),\n",
    "        ]\n",
    "    )\n",
    "    # Chain 2\n",
    "    question_answer_chain = create_stuff_documents_chain(llm, qa_prompt)\n",
    "    # combine both chains\n",
    "    rag_chain = create_retrieval_chain(chat_retriever_chain, question_answer_chain)\n",
    "    # invoke the chain to get a response\n",
    "    result = rag_chain.invoke({\"input\": question, \"chat_history\": chat_history })\n",
    "    # Append to Chat History \n",
    "    chat_history.append((question, result['answer']))\n",
    "    \n",
    "    return result, chat_history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "8. This does something with LangSmith trace\n",
    "https://docs.smith.langchain.com/tracing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# LangSmith Trace\n",
    "import os\n",
    "from dotenv import load_dotenv, find_dotenv\n",
    "_ = load_dotenv(find_dotenv()) # read local .env file\n",
    "\n",
    "LANGCHAIN_TRACING_V2 = 'true'\n",
    "LANGCHAIN_ENDPOINT = 'https://api.smith.langchain.com'\n",
    "LANGCHAIN_API_KEY = os.environ['LANGCHAIN_API_KEY']\n",
    "LANGCHAIN_PROJECT = os.environ['LANGCHAIN_PROJECT']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "9. Chat with your database to test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The employees who hold the position of Fire Suppression Captain in the provided data are MICHAEL CASTAGNOLA, MICHAEL DELANE, and GREG WYRSCH.\n",
      "[('What employee is the Fire Suppression Captain?', 'The employees who hold the position of Fire Suppression Captain in the provided data are MICHAEL CASTAGNOLA, MICHAEL DELANE, and GREG WYRSCH.')]\n"
     ]
    }
   ],
   "source": [
    "chat_history = []\n",
    "question = \"What employee is the Fire Suppression Captain?\"\n",
    "result, chat_history = create_history_aware_retriever_with_hub(vector_store, question, chat_history)\n",
    "print(result['answer'])\n",
    "print(chat_history)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Jeffrey Covitz makes a total of $160,383.74, according to the provided information for the year 2011.\n",
      "[('How much money does Jeffrey Covits make?', 'Jeffrey Covitz makes a total of $160,383.74, according to the provided information for the year 2011.')]\n"
     ]
    }
   ],
   "source": [
    "chat_history = []\n",
    "question = \"How much money does Jeffrey Covits make?\"\n",
    "result, chat_history = create_history_aware_retriever_with_hub(vector_store, question, chat_history)\n",
    "print(result['answer'])\n",
    "print(chat_history)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To calculate how much more money Jeffrey Covitz makes compared to Jonathan Huggins, we can look at their TotalPay:\n",
      "\n",
      "- Jeffrey Covitz's TotalPay: $160,383.74\n",
      "- Jonathan Huggins's TotalPay: $162,557.43\n",
      "\n",
      "Now we subtract Jonathan Huggins' TotalPay from Jeffrey Covitz's TotalPay:\n",
      "\n",
      "$160,383.74 - $162,557.43 = -$2,173.69\n",
      "\n",
      "Therefore, Jeffrey Covitz makes $2,173.69 less than Jonathan Huggins.\n",
      "[('How much more money does Jeffrey Covits make compared to Jonathan Huggins?', \"To calculate how much more money Jeffrey Covitz makes compared to Jonathan Huggins, we can look at their TotalPay:\\n\\n- Jeffrey Covitz's TotalPay: $160,383.74\\n- Jonathan Huggins's TotalPay: $162,557.43\\n\\nNow we subtract Jonathan Huggins' TotalPay from Jeffrey Covitz's TotalPay:\\n\\n$160,383.74 - $162,557.43 = -$2,173.69\\n\\nTherefore, Jeffrey Covitz makes $2,173.69 less than Jonathan Huggins.\")]\n"
     ]
    }
   ],
   "source": [
    "chat_history = []\n",
    "question = \"How much more money does Jeffrey Covits make compared to Jonathan Huggins?\"\n",
    "result, chat_history = create_history_aware_retriever_with_hub(vector_store, question, chat_history)\n",
    "print(result['answer'])\n",
    "print(chat_history)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "trasbotv0.1_env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}