examples/multimodal.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example will walk you throught the basic usage of MULTI-MODAL models in PromptBench. We hope that you can get familiar with the APIs and use it in your own projects later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, there is a unified import of `import promptbench as pb` that easily imports the package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda/envs/promptbench_1/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "import promptbench as pb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load dataset\n",
    "\n",
    "First, PromptBench supports easy load of datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All supported datasets: \n",
      "['vqav2', 'nocaps', 'science_qa', 'math_vista', 'ai2d', 'mmmu', 'chart_qa']\n",
      "Images already saved to local, loading file:  /home/v-mingxia/promptbench/promptbench/data/mmmu/validation.json\n",
      "{'images': [<PIL.PngImagePlugin.PngImageFile image mode=RGBA size=733x237 at 0x7F13BA2CD160>], 'image_paths': ['/home/v-mingxia/promptbench/promptbench/data/mmmu/validation/0_image_1.png'], 'answer': 'B', 'question': '<image 1> Baxter Company has a relevant range of production between 15,000 and 30,000 units. The following cost data represents average variable costs per unit for 25,000 units of production. If 30,000 units are produced, what are the per unit manufacturing overhead costs incurred?\\nA: $6\\nB: $7\\nC: $8\\nD: $9'}\n",
      "{'images': [<PIL.PngImagePlugin.PngImageFile image mode=RGBA size=342x310 at 0x7F13BA2CD550>], 'image_paths': ['/home/v-mingxia/promptbench/promptbench/data/mmmu/validation/1_image_1.png'], 'answer': 'C', 'question': 'Assume accounts have normal balances, solve for the one missing account balance: Dividends. Equipment was recently purchased, so there is neither depreciation expense nor accumulated depreciation. <image 1>\\nA: $194,815\\nB: $182,815\\nC: $12,000\\nD: $9,000'}\n",
      "{'images': [<PIL.PngImagePlugin.PngImageFile image mode=RGBA size=336x169 at 0x7F13BA2CD130>], 'image_paths': ['/home/v-mingxia/promptbench/promptbench/data/mmmu/validation/2_image_1.png'], 'answer': 'B', 'question': 'Maxwell Software, Inc., has the following mutually exclusive projects.Suppose the company uses the NPV rule to rank these two projects.<image 1> Which project should be chosen if the appropriate discount rate is 15 percent?\\nA: Project A\\nB: Project B'}\n",
      "{'images': [<PIL.PngImagePlugin.PngImageFile image mode=RGBA size=1222x237 at 0x7F13BA2CD460>], 'image_paths': ['/home/v-mingxia/promptbench/promptbench/data/mmmu/validation/3_image_1.png'], 'answer': 'D', 'question': \"Each situation below relates to an independent company's Owners' Equity. <image 1> Calculate the missing values of company 2.\\nA: $1,620\\nB: $12,000\\nC: $51,180\\nD: $0\"}\n",
      "{'images': [<PIL.PngImagePlugin.PngImageFile image mode=RGBA size=1219x217 at 0x7F13BA2CD400>], 'image_paths': ['/home/v-mingxia/promptbench/promptbench/data/mmmu/validation/4_image_1.png'], 'answer': 'B', 'question': 'The following data show the units in beginning work in process inventory, the number of units started, the number of units transferred, and the percent completion of the ending work in process for conversion. Given that materials are added at the beginning of the process, what are the equivalent units for conversion costs for each quarter using the weighted-average method? Assume that the quarters are independent.<image 1>\\nA: 132,625\\nB: 134,485\\nC: 135,332\\nD: 132,685'}\n"
     ]
    }
   ],
   "source": [
    "# print all supported datasets in promptbench\n",
    "print('All supported datasets: ')\n",
    "print(pb.SUPPORTED_DATASETS_VLM)\n",
    "\n",
    "# load a dataset, MMMMU, for instance.\n",
    "# if the dataset is not available locally, it will be downloaded automatically.\n",
    "dataset = pb.DatasetLoader.load_dataset(\"mmmu\")\n",
    "\n",
    "# print the first 5 examples\n",
    "for idx in range(5):\n",
    "    print(dataset[idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load models\n",
    "\n",
    "Then, you can easily load VLM models via promptbench."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All supported models: \n",
      "['Salesforce/blip2-opt-2.7b', 'Salesforce/blip2-opt-6.7b', 'Salesforce/blip2-flan-t5-xl', 'Salesforce/blip2-flan-t5-xxl', 'llava-hf/llava-1.5-7b-hf', 'llava-hf/llava-1.5-13b-hf', 'gemini-pro-vision', 'gpt-4-vision-preview', 'Qwen/Qwen-VL', 'Qwen/Qwen-VL-Chat', 'qwen-vl-plus', 'qwen-vl-max', 'internlm/internlm-xcomposer2-vl-7b']\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n",
      "Loading checkpoint shards: 100%|██████████| 3/3 [00:04<00:00,  1.48s/it]\n"
     ]
    }
   ],
   "source": [
    "# print all supported models in promptbench\n",
    "print('All supported models: ')\n",
    "print(pb.SUPPORTED_MODELS_VLM)\n",
    "\n",
    "# load a model, llava-1.5-7b, for instance.\n",
    "model = pb.VLMModel(model='llava-hf/llava-1.5-7b-hf', max_new_tokens=2048, temperature=0.0001, device='cuda')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Construct prompts\n",
    "\n",
    "Prompts are the key interaction interface to VLMs. You can easily construct a prompt by call the Prompt API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prompt API supports a list, so you can pass multiple prompts at once.\n",
    "prompts = pb.Prompt([\n",
    "    \"You are a helpful assistant. Here is the question:{question}\\nANSWER:\",\n",
    "    \"USER:{question}\\nANSWER:\",\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perform evaluation using prompts, datasets, and models\n",
    "\n",
    "Finally, you can perform standard evaluation using the loaded prompts, datasets, and labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/900 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 900/900 [17:35<00:00,  1.17s/it]  \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.333, 'You are a helpful assistant. Here is the question:{question}\\nANSWER:'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 900/900 [17:27<00:00,  1.16s/it]  "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.316, 'USER:{question}\\nANSWER:'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "for prompt in prompts:\n",
    "    preds = []\n",
    "    labels = []\n",
    "    for data in tqdm(dataset):\n",
    "        # process input\n",
    "        input_text = pb.InputProcess.basic_format(prompt, data)\n",
    "        input_images = data['images']  # please use data['image_paths'] instead of data['images'] for models that only support image path/url, such as GPT-4v\n",
    "        label = data['answer']\n",
    "        raw_pred = model(input_images, input_text)\n",
    "        # process output\n",
    "        pred = pb.OutputProcess.pattern_split(raw_pred, 'ANSWER:')\n",
    "        preds.append(pred)\n",
    "        labels.append(label)\n",
    "    \n",
    "    # evaluate\n",
    "    score = pb.Eval.compute_cls_accuracy(preds, labels)\n",
    "    print(f\"{score:.3f}, {repr(prompt)}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "promptbench",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}