notebooks/209-handwritten-ocr/209-handwritten-ocr.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0293f65e",
   "metadata": {},
   "source": [
    "# Handwritten Chinese and Japanese OCR with OpenVINO™\n",
    "\n",
    "In this tutorial, we perform optical character recognition (OCR) for handwritten Chinese (simplified) and Japanese. An OCR tutorial using the Latin alphabet is available in [notebook 208](../208-optical-character-recognition/208-optical-character-recognition.ipynb). This model is capable of processing only one line of symbols at a time.\n",
    "\n",
    "The models used in this notebook are [`handwritten-japanese-recognition-0001`](https://docs.openvino.ai/2023.3/omz_models_model_handwritten_japanese_recognition_0001.html) and [`handwritten-simplified-chinese-0001`](https://docs.openvino.ai/2023.3/omz_models_model_handwritten_simplified_chinese_recognition_0001.html). To decode model outputs as readable text [`kondate_nakayosi`](https://github.com/openvinotoolkit/open_model_zoo/blob/master/data/dataset_classes/kondate_nakayosi.txt) and [`scut_ept`](https://github.com/openvinotoolkit/open_model_zoo/blob/master/data/dataset_classes/scut_ept.txt) charlists are used. Both models are available on [Open Model Zoo](https://github.com/openvinotoolkit/open_model_zoo/).\n",
    "\n",
    "\n",
    "#### Table of contents:\n\n",
    "- [Imports](#Imports)\n",
    "- [Settings](#Settings)\n",
    "- [Select a Language](#Select-a-Language)\n",
    "- [Download the Model](#Download-the-Model)\n",
    "- [Load the Model and Execute](#Load-the-Model-and-Execute)\n",
    "- [Select inference device](#Select-inference-device)\n",
    "- [Fetch Information About Input and Output Layers](#Fetch-Information-About-Input-and-Output-Layers)\n",
    "- [Load an Image](#Load-an-Image)\n",
    "- [Visualize Input Image](#Visualize-Input-Image)\n",
    "- [Prepare Charlist](#Prepare-Charlist)\n",
    "- [Run Inference](#Run-Inference)\n",
    "- [Process the Output Data](#Process-the-Output-Data)\n",
    "- [Print the Output](#Print-the-Output)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b2642605",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install openvino-dev package\n",
    "%pip install -q \"openvino>=2023.1.0\"\n",
    "%pip install -q matplotlib numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fda2e1e0",
   "metadata": {},
   "source": [
    "## Imports\n",
    "[back to top ⬆️](#Table-of-contents:)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e0a6a0d5",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from collections import namedtuple\n",
    "from itertools import groupby\n",
    "\n",
    "import cv2\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import openvino as ov\n",
    "\n",
    "# Fetch `notebook_utils` module\n",
    "import urllib.request\n",
    "urllib.request.urlretrieve(\n",
    "    url='https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/main/notebooks/utils/notebook_utils.py',\n",
    "    filename='notebook_utils.py'\n",
    ")\n",
    "from notebook_utils import download_file"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a46517a",
   "metadata": {},
   "source": [
    "## Settings\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Set up all constants and folders used in this notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b50b2bad",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Directories where data will be placed.\n",
    "base_models_dir = \"models\"\n",
    "data_folder = \"data\"\n",
    "charlist_folder = f\"{data_folder}/text\"\n",
    "\n",
    "# Precision used by the model.\n",
    "precision = \"FP16\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3c93fed",
   "metadata": {},
   "source": [
    "To group files, you have to define the collection. In this case, use `namedtuple`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "17402fcc",
   "metadata": {},
   "outputs": [],
   "source": [
    "Language = namedtuple(\n",
    "    typename=\"Language\", field_names=[\"model_name\", \"charlist_name\", \"demo_image_name\"]\n",
    ")\n",
    "chinese_files = Language(\n",
    "    model_name=\"handwritten-simplified-chinese-recognition-0001\",\n",
    "    charlist_name=\"chinese_charlist.txt\",\n",
    "    demo_image_name=\"handwritten_chinese_test.jpg\",\n",
    ")\n",
    "japanese_files = Language(\n",
    "    model_name=\"handwritten-japanese-recognition-0001\",\n",
    "    charlist_name=\"japanese_charlist.txt\",\n",
    "    demo_image_name=\"handwritten_japanese_test.png\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c61827c",
   "metadata": {},
   "source": [
    "## Select a Language\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Depending on your choice you will need to change a line of code in the cell below.\n",
    "\n",
    "If you want to perform OCR on a text in Japanese, set `language = \"japanese\"`. For Chinese, set `language = \"chinese\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1d3b1190",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select the language by using either language=\"chinese\" or language=\"japanese\".\n",
    "language = \"chinese\"\n",
    "\n",
    "languages = {\"chinese\": chinese_files, \"japanese\": japanese_files}\n",
    "\n",
    "selected_language = languages.get(language)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "750355d4",
   "metadata": {},
   "source": [
    "## Download the Model\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "In addition to images and charlists, you need to download the model file. In the sections below, there are cells for downloading either the Chinese or Japanese model.\n",
    " \n",
    "If it is your first time running the notebook, the model will be downloaded. It may take a few minutes. \n",
    "\n",
    "Use `download_file` function from the utils package, which automatically creates a directory structure and downloads the selected model file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e8f266fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "path_to_model = download_file(\n",
    "    url=f'https://storage.openvinotoolkit.org/repositories/open_model_zoo/2023.0/models_bin/1/{selected_language.model_name}/{precision}/{selected_language.model_name}.xml',\n",
    "    directory=base_models_dir\n",
    ")\n",
    "_ = download_file(\n",
    "    url=f'https://storage.openvinotoolkit.org/repositories/open_model_zoo/2023.0/models_bin/1/{selected_language.model_name}/{precision}/{selected_language.model_name}.bin',\n",
    "    directory=base_models_dir\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d026308",
   "metadata": {},
   "source": [
    "## Load the Model and Execute\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "When all files are downloaded and language is selected, read and compile the network to run inference. The path to the model is defined based on the selected language."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "7114e467",
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "core = ov.Core()\n",
    "model = core.read_model(model=path_to_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ecb4551",
   "metadata": {},
   "source": [
    "## Select inference device\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "select device from dropdown list for running inference using OpenVINO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "aaef9cca-de02-49b3-9544-d1c9cfce7792",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d80e65b78ae44972b9f64e84f91c7fae",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(description='Device:', index=2, options=('CPU', 'GPU', 'AUTO'), value='AUTO')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import ipywidgets as widgets\n",
    "\n",
    "device = widgets.Dropdown(\n",
    "    options=core.available_devices + [\"AUTO\"],\n",
    "    value='AUTO',\n",
    "    description='Device:',\n",
    "    disabled=False,\n",
    ")\n",
    "\n",
    "device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "024ad3a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "compiled_model = core.compile_model(model=model, device_name=device.value)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3df5b652",
   "metadata": {},
   "source": [
    "## Fetch Information About Input and Output Layers\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Now that the model is loaded, fetch information about the input and output layers (shape)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "7d31411a",
   "metadata": {},
   "outputs": [],
   "source": [
    "recognition_output_layer = compiled_model.output(0)\n",
    "recognition_input_layer = compiled_model.input(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41307bc7",
   "metadata": {},
   "source": [
    "## Load an Image\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Next, load an image. The model expects a single-channel image as input, so the image is read in grayscale.\n",
    "\n",
    "After loading the input image, get information to use for calculating the scale ratio between required input layer height and the current image height. In the cell below, the image will be resized and padded to keep letters proportional and meet input shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "afa1d51a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download the image from the openvino_notebooks storage based on the selected model.\n",
    "file_name = download_file(\n",
    "    \"https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/image/\" + selected_language.demo_image_name,\n",
    "    directory=data_folder\n",
    ")\n",
    "\n",
    "# Text detection models expect an image in grayscale format.\n",
    "# IMPORTANT! This model enables reading only one line at time.\n",
    "\n",
    "# Read the image.\n",
    "image = cv2.imread(filename=str(file_name), flags=cv2.IMREAD_GRAYSCALE)\n",
    "\n",
    "# Fetch the shape.\n",
    "image_height, _ = image.shape\n",
    "\n",
    "# B,C,H,W = batch size, number of channels, height, width.\n",
    "_, _, H, W = recognition_input_layer.shape\n",
    "\n",
    "# Calculate scale ratio between the input shape height and image height to resize the image.\n",
    "scale_ratio = H / image_height\n",
    "\n",
    "# Resize the image to expected input sizes.\n",
    "resized_image = cv2.resize(\n",
    "    image, None, fx=scale_ratio, fy=scale_ratio, interpolation=cv2.INTER_AREA\n",
    ")\n",
    "\n",
    "# Pad the image to match input size, without changing aspect ratio.\n",
    "resized_image = np.pad(\n",
    "    resized_image, ((0, 0), (0, W - resized_image.shape[1])), mode=\"edge\"\n",
    ")\n",
    "\n",
    "# Reshape to network input shape.\n",
    "input_image = resized_image[None, None, :, :]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba364bee",
   "metadata": {},
   "source": [
    "## Visualize Input Image\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "After preprocessing, you can display the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f7a8e437",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20, 1))\n",
    "plt.axis(\"off\")\n",
    "plt.imshow(resized_image, cmap=\"gray\", vmin=0, vmax=255);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6d4c642",
   "metadata": {},
   "source": [
    "## Prepare Charlist\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "The model is loaded and the image is ready. The only element left is the charlist, which is downloaded. You must add a blank symbol at the beginning of the charlist before using it. This is expected for both the Chinese and Japanese models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e35eae1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download the image from the openvino_notebooks storage based on the selected model.\n",
    "used_charlist_file = download_file(\n",
    "    \"https://storage.openvinotoolkit.org/repositories/openvino_notebooks/data/data/text/\" + selected_language.charlist_name,\n",
    "    directory=charlist_folder\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "2cb18186",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get a dictionary to encode the output, based on model documentation.\n",
    "used_charlist = selected_language.charlist_name\n",
    "\n",
    "# With both models, there should be blank symbol added at index 0 of each charlist.\n",
    "blank_char = \"~\"\n",
    "\n",
    "with used_charlist_file.open(mode=\"r\", encoding=\"utf-8\") as charlist:\n",
    "    letters = blank_char + \"\".join(line.strip() for line in charlist)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6a4da65",
   "metadata": {},
   "source": [
    "## Run Inference\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Now, run inference. The `compiled_model()` function takes a list with input(s) in the same order as model input(s). Then, fetch the output from output tensors.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "e44a9de4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run inference on the model\n",
    "predictions = compiled_model([input_image])[recognition_output_layer]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03a40d22",
   "metadata": {},
   "source": [
    "## Process the Output Data\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "The output of a model is in the `W x B x L` format, where:\n",
    "\n",
    "* W - output sequence length\n",
    "* B - batch size\n",
    "* L - confidence distribution across the supported symbols in Kondate and Nakayosi.\n",
    "\n",
    "To get a more human-readable format, select a symbol with the highest probability. When you hold a list of indexes that are predicted to have the highest probability, due to limitations in [CTC Decoding](https://towardsdatascience.com/beam-search-decoding-in-ctc-trained-neural-networks-5a889a3d85a7), you will remove concurrent symbols and then remove the blanks.\n",
    "\n",
    "Finally, get the symbols from corresponding indexes in the charlist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "da1b8bc5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove a batch dimension.\n",
    "predictions = np.squeeze(predictions)\n",
    "\n",
    "# Run the `argmax` function to pick the symbols with the highest probability.\n",
    "predictions_indexes = np.argmax(predictions, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "f6730159",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the `groupby` function to remove concurrent letters, as required by CTC greedy decoding.\n",
    "output_text_indexes = list(groupby(predictions_indexes))\n",
    "\n",
    "# Remove grouper objects.\n",
    "output_text_indexes, _ = np.transpose(output_text_indexes, (1, 0))\n",
    "\n",
    "# Remove blank symbols.\n",
    "output_text_indexes = output_text_indexes[output_text_indexes != 0]\n",
    "\n",
    "# Assign letters to indexes from the output array.\n",
    "output_text = [letters[letter_index] for letter_index in output_text_indexes]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac88f622",
   "metadata": {},
   "source": [
    "## Print the Output\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Now, having a list of letters predicted by the model, you can display the image with predicted text printed below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "6f8a90f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(20, 1))\n",
    "plt.axis(\"off\")\n",
    "plt.imshow(resized_image, cmap=\"gray\", vmin=0, vmax=255)\n",
    "\n",
    "print(\"\".join(output_text))"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "ae617ccb002f72b3ab6d0069d721eac67ac2a969e83c083c4321cfcab0437cd1"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  },
  "openvino_notebooks": {
   "imageUrl": "https://github.com/openvinotoolkit/openvino_notebooks/blob/main/notebooks/209-handwritten-ocr/209-handwritten-ocr.png?raw=true",
   "tags": {
    "categories": [
     "Model Demos"
    ],
    "libraries": [],
    "other": [],
    "tasks": [
     "Image-to-Text"
    ]
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}