codellama_12b_ggmlv3_Q5_K_M_bin_test.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyNXhbvEmUYJH3L8P/gMeRGO",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jankovicsandras/ml/blob/main/codellama_12b_ggmlv3_Q5_K_M_bin_test.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Check if model file is downloaded"
      ],
      "metadata": {
        "id": "yIfIqQEDPnXG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! ls -la\n",
        "! pwd"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uhTEY8T3PmFU",
        "outputId": "400bc8df-8c03-43e2-f63a-6df6c282a663"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "total 9182292\n",
            "drwxr-xr-x 1 root root       4096 Aug 25 10:00 .\n",
            "drwxr-xr-x 1 root root       4096 Aug 25 09:56 ..\n",
            "-rw-r--r-- 1 root root 9402644896 Aug 24 22:55 codellama-13b.ggmlv3.Q5_K_M.bin\n",
            "drwxr-xr-x 4 root root       4096 Aug 23 13:34 .config\n",
            "drwxr-xr-x 1 root root       4096 Aug 23 13:35 sample_data\n",
            "/content\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Model download : Do this only if model file is not visibile above!"
      ],
      "metadata": {
        "id": "337frCuWQC42"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "! wget https://huggingface.co/TheBloke/CodeLlama-13B-GGML/resolve/main/codellama-13b.ggmlv3.Q5_K_M.bin"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9NTifmRqP4Rr",
        "outputId": "866a059a-b050-4d86-9da2-30d604a5bab1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2023-08-25 10:00:03--  https://huggingface.co/TheBloke/CodeLlama-13B-GGML/resolve/main/codellama-13b.ggmlv3.Q5_K_M.bin\n",
            "Resolving huggingface.co (huggingface.co)... 18.172.134.4, 18.172.134.24, 18.172.134.124, ...\n",
            "Connecting to huggingface.co (huggingface.co)|18.172.134.4|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://cdn-lfs.huggingface.co/repos/e9/a0/e9a0663c06454e01d6365524873f80c5b77f8f7ac0922695c2e9fedcd6c9593d/fdd0205f42ed9e2f3f17e413caa75a9654a167e046909f24fe9b84fbfbc86f8d?response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27codellama-13b.ggmlv3.Q5_K_M.bin%3B+filename%3D%22codellama-13b.ggmlv3.Q5_K_M.bin%22%3B&response-content-type=application%2Foctet-stream&Expires=1693216804&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTY5MzIxNjgwNH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy5odWdnaW5nZmFjZS5jby9yZXBvcy9lOS9hMC9lOWEwNjYzYzA2NDU0ZTAxZDYzNjU1MjQ4NzNmODBjNWI3N2Y4ZjdhYzA5MjI2OTVjMmU5ZmVkY2Q2Yzk1OTNkL2ZkZDAyMDVmNDJlZDllMmYzZjE3ZTQxM2NhYTc1YTk2NTRhMTY3ZTA0NjkwOWYyNGZlOWI4NGZiZmJjODZmOGQ%7EcmVzcG9uc2UtY29udGVudC1kaXNwb3NpdGlvbj0qJnJlc3BvbnNlLWNvbnRlbnQtdHlwZT0qIn1dfQ__&Signature=sSe5EbRnfgifRImeHmGWmL6f3INFRKT7srOh7lx328Q9SH0Vl8s164NSb%7E5m62TmLuOHP3CDx7SHKqSK5V6aZeWoS8x57G1SCzshHsXyONcA2-Qc-pZt5ctLLnTjwEjrmEHdmiX6oDqsbYsCZkXrsEg6eRwg8G5JmnK7uK2aAoalqwZFGhlPXjWPUlaOvCOW6dDz5rGqxYD17sBdV42jrfuxgyZmPc5Dcb4vgGDLveloowp1%7E%7Eq-1IHvsnkDCiovtadQ8Fl7f6lz-CVkMmSnM6HE%7EvUVuPN-yXVKTJkq0NZuzXZaSzByjPe2Wt%7EyzfNyd-Z22BYgX3KJTQTgw6PLGA__&Key-Pair-Id=KVTP0A1DKRTAX [following]\n",
            "--2023-08-25 10:00:04--  https://cdn-lfs.huggingface.co/repos/e9/a0/e9a0663c06454e01d6365524873f80c5b77f8f7ac0922695c2e9fedcd6c9593d/fdd0205f42ed9e2f3f17e413caa75a9654a167e046909f24fe9b84fbfbc86f8d?response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27codellama-13b.ggmlv3.Q5_K_M.bin%3B+filename%3D%22codellama-13b.ggmlv3.Q5_K_M.bin%22%3B&response-content-type=application%2Foctet-stream&Expires=1693216804&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTY5MzIxNjgwNH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy5odWdnaW5nZmFjZS5jby9yZXBvcy9lOS9hMC9lOWEwNjYzYzA2NDU0ZTAxZDYzNjU1MjQ4NzNmODBjNWI3N2Y4ZjdhYzA5MjI2OTVjMmU5ZmVkY2Q2Yzk1OTNkL2ZkZDAyMDVmNDJlZDllMmYzZjE3ZTQxM2NhYTc1YTk2NTRhMTY3ZTA0NjkwOWYyNGZlOWI4NGZiZmJjODZmOGQ%7EcmVzcG9uc2UtY29udGVudC1kaXNwb3NpdGlvbj0qJnJlc3BvbnNlLWNvbnRlbnQtdHlwZT0qIn1dfQ__&Signature=sSe5EbRnfgifRImeHmGWmL6f3INFRKT7srOh7lx328Q9SH0Vl8s164NSb%7E5m62TmLuOHP3CDx7SHKqSK5V6aZeWoS8x57G1SCzshHsXyONcA2-Qc-pZt5ctLLnTjwEjrmEHdmiX6oDqsbYsCZkXrsEg6eRwg8G5JmnK7uK2aAoalqwZFGhlPXjWPUlaOvCOW6dDz5rGqxYD17sBdV42jrfuxgyZmPc5Dcb4vgGDLveloowp1%7E%7Eq-1IHvsnkDCiovtadQ8Fl7f6lz-CVkMmSnM6HE%7EvUVuPN-yXVKTJkq0NZuzXZaSzByjPe2Wt%7EyzfNyd-Z22BYgX3KJTQTgw6PLGA__&Key-Pair-Id=KVTP0A1DKRTAX\n",
            "Resolving cdn-lfs.huggingface.co (cdn-lfs.huggingface.co)... 108.159.227.71, 108.159.227.86, 108.159.227.69, ...\n",
            "Connecting to cdn-lfs.huggingface.co (cdn-lfs.huggingface.co)|108.159.227.71|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 9402644896 (8.8G) [application/octet-stream]\n",
            "Saving to: ‘codellama-13b.ggmlv3.Q5_K_M.bin’\n",
            "\n",
            "codellama-13b.ggmlv 100%[===================>]   8.76G  86.0MB/s    in 76s     \n",
            "\n",
            "2023-08-25 10:01:20 (117 MB/s) - ‘codellama-13b.ggmlv3.Q5_K_M.bin’ saved [9402644896/9402644896]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Install deps\n"
      ],
      "metadata": {
        "id": "b0jitre4aW-T"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ofOv548DZ32W",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "08756734-cede-45bb-9b92-cc27e90d74db"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m6.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m14.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m45.6/45.6 kB\u001b[0m \u001b[31m4.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.4/49.4 kB\u001b[0m \u001b[31m5.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Building wheel for llama-cpp-python (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n"
          ]
        }
      ],
      "source": [
        "!pip -q install langchain llama-cpp-python"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# LLMChain = Model + PromptTemplate"
      ],
      "metadata": {
        "id": "CXmg9qIQblfJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from langchain import PromptTemplate, LLMChain, LlamaCpp\n",
        "\n",
        "# Load model\n",
        "model_name = \"codellama-13b.ggmlv3.Q5_K_M.bin\"\n",
        "llm = LlamaCpp(\n",
        "\tmodel_path=model_name,\n",
        "  verbose=True,\n",
        "  last_n_tokens_size=32,\n",
        "  max_tokens=128,\n",
        "  n_ctx=256,\n",
        "  temperature=0.3\n",
        ")\n",
        "\n",
        "# Template and Prompt\n",
        "template = \"\"\"Question: {question}\n",
        "Answer: \"\"\"\n",
        "\n",
        "prompt = PromptTemplate(\n",
        "  template=template,\n",
        "  input_variables=['question']\n",
        ")\n",
        "\n",
        "# LLMChain\n",
        "llm_chain = LLMChain(prompt=prompt, llm=llm)"
      ],
      "metadata": {
        "id": "lqaddfrYcAyU",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ccb1c6da-8f66-4641-f345-1cc9d321fa7a"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# LangChain LLMChain question and run"
      ],
      "metadata": {
        "id": "ZfaHJCQYifoV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "question = \"write a python function to readline from keyboard and save it to a file\"\n",
        "\n",
        "result = llm_chain(question)\n",
        "\n",
        "print(result['text'])\n",
        "print(result)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "akDv5vrpii4J",
        "outputId": "19b8c8d8-4c34-45fd-db77-20d20889948e"
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1.You can use `input()` instead of `raw_input()`.\n",
            "\\begin{code}\n",
            "def readline():\n",
            "    file = open(\"test.txt\", \"w\"))\n",
            "    while True:\n",
            "        line = input()\n",
            "        if not line:\n",
            "            break\n",
            "\n",
            "        file.write(line + \"\\n\"))\n",
            "    file.close()\n",
            "\n",
            "if __name__ == \"__main__\":\n",
            "    readline()\n",
            "\\end{code}\n",
            "2.`raw_input()` is deprecated in Python 3.x, you can use `input()` instead of `raw_input\n",
            "{'question': 'write a python function to readline from keyboard and save it to a file', 'text': '1.You can use `input()` instead of `raw_input()`.\\n\\\\begin{code}\\ndef readline():\\n    file = open(\"test.txt\", \"w\"))\\n    while True:\\n        line = input()\\n        if not line:\\n            break\\n\\n        file.write(line + \"\\\\n\"))\\n    file.close()\\n\\nif __name__ == \"__main__\":\\n    readline()\\n\\\\end{code}\\n2.`raw_input()` is deprecated in Python 3.x, you can use `input()` instead of `raw_input'}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Some test results:\n",
        "\n",
        "### question = \"write a javascript function to list primes between fifty and eighty in reverse order.\"\n",
        "\n",
        "```javascript\n",
        "If we want to get the number of digits (i * i) will have, then we can use the following code.\n",
        "\\begin{code}\n",
        "function getDigits(num){\n",
        "        let result = [];\n",
        "        while(parseInt(result))\n",
        "            return num;\n",
        "}\n",
        "let numbers = [], j, k, prime_number = true;\n",
        "for(i = 50; i <= 80 ; i++){\n",
        "    for(j = 2; j < i ; j++)){\n",
        "                if((i * i) % j == 0){\n",
        "                            prime_number = false;\n",
        "                        }\n",
        "                    }\n",
        "                                       // if number is not divisible by any number between 1 to n - 1 , then it will be prime.\n",
        "\n",
        "}else{\n",
        "                                     numbers.push(i));\n",
        "}\n",
        "}\n",
        "\n",
        "console.log(numbers));\n",
        "console.log(\"The total prime numbers are - \")));\n",
        "console.log(numbers.length)))));\n",
        "console.log(((((((((((((((((((((((((((((((((((((((((((()))))))))))))))))))))))))))))))))+1))))\n",
        "\n",
        "```\n",
        "\n",
        "This is almost completely wrong. The loop between 50 and 80, the numbers array push and console log and parts of the primality testing seems okay; but getDigits is hallucinated, i*i in the primality test is wrong, etc. syntax errors and gibberish... unusable. Took ca. 10 mins.\n",
        "\n",
        "\n",
        "### question = \"write a python function to readline from keyboard and save it to a file\"\n",
        "\n",
        "```python\n",
        "1.You can use `input()` instead of `raw_input()`.\n",
        "\\begin{code}\n",
        "def readline():\n",
        "    file = open(\"test.txt\", \"w\"))\n",
        "    while True:\n",
        "        line = input()\n",
        "        if not line:\n",
        "            break\n",
        "\n",
        "        file.write(line + \"\\n\"))\n",
        "    file.close()\n",
        "\n",
        "if __name__ == \"__main__\":\n",
        "    readline()\n",
        "\\end{code}\n",
        "2.`raw_input()` is deprecated in Python 3.x, you can use `input()` instead of `raw_input\n",
        "```\n",
        "\n",
        "There are 2 small syntax errors: file = open... )) <- only one ) is needed, also file.write...)) . After fixing these manually, the code seems to work. Took ca. 5 mins."
      ],
      "metadata": {
        "id": "nJu63lw_hJ20"
      }
    }
  ]
}