01_pytorch_basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PyTorch Basics: Tensors and Gradients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in links: https://download.pytorch.org/whl/torch_stable.html\n",
      "Requirement already satisfied: numpy in /home/priyanshu/bash/lib/python3.8/site-packages (1.18.5)\n",
      "Collecting torch==1.7.0+cpu\n",
      "  Downloading https://download.pytorch.org/whl/cpu/torch-1.7.0%2Bcpu-cp38-cp38-linux_x86_64.whl (159.4 MB)\n",
      "\u001b[K     |████████████████████████████████| 159.4 MB 4.8 kB/s eta 0:00:01    |█████▎                          | 26.2 MB 826 kB/s eta 0:02:42     |████████████████████▊           | 103.2 MB 4.0 MB/s eta 0:00:15     |█████████████████████████       | 124.8 MB 161 kB/s eta 0:03:34     |█████████████████████████▏      | 125.2 MB 3.3 MB/s eta 0:00:11     |███████████████████████████████▉| 158.4 MB 3.2 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting torchvision==0.8.1+cpu\n",
      "  Downloading https://download.pytorch.org/whl/cpu/torchvision-0.8.1%2Bcpu-cp38-cp38-linux_x86_64.whl (11.8 MB)\n",
      "\u001b[K     |████████████████████████████████| 11.8 MB 3.2 MB/s eta 0:00:01    |██████████████████▊             | 6.9 MB 2.3 MB/s eta 0:00:03\n",
      "\u001b[?25hCollecting torchaudio==0.7.0\n",
      "  Downloading torchaudio-0.7.0-cp38-cp38-manylinux1_x86_64.whl (7.6 MB)\n",
      "\u001b[K     |████████████████████████████████| 7.6 MB 102 kB/s eta 0:00:01     |████████████████████████████▉   | 6.9 MB 267 kB/s eta 0:00:03\n",
      "\u001b[?25hCollecting dataclasses\n",
      "  Downloading dataclasses-0.6-py3-none-any.whl (14 kB)\n",
      "Requirement already satisfied: typing-extensions in /home/priyanshu/bash/lib/python3.8/site-packages (from torch==1.7.0+cpu) (3.7.4.2)\n",
      "Requirement already satisfied: future in /home/priyanshu/bash/lib/python3.8/site-packages (from torch==1.7.0+cpu) (0.18.2)\n",
      "Requirement already satisfied: pillow>=4.1.1 in /home/priyanshu/bash/lib/python3.8/site-packages (from torchvision==0.8.1+cpu) (7.2.0)\n",
      "Installing collected packages: dataclasses, torch, torchvision, torchaudio\n",
      "Successfully installed dataclasses-0.6 torch-1.7.0+cpu torchaudio-0.7.0 torchvision-0.8.1+cpu\n"
     ]
    }
   ],
   "source": [
    "!pip install numpy torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensors\n",
    "\n",
    "At its core, PyTorch is a library for processing tensors. A tensor is a number, vector, matrix or any n-dimensional array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(4.)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t1 = torch.tensor(4.)\n",
    "t1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4. means 4.0 (floating point number)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.float32"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t1.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1., 2., 3.])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t2 = torch.tensor([1. ,2 ,3])\n",
    "t2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 2.],\n",
       "        [3., 4.],\n",
       "        [5., 6.]])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t3 = torch.tensor([[1, 2],\n",
    "                 [3, 4.],\n",
    "                 [5., 6]])\n",
    "t3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ 1,  2,  3],\n",
       "         [ 4,  5,  6]],\n",
       "\n",
       "        [[ 7,  8,  9],\n",
       "         [10, 11, 12]]])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 3d array\n",
    "\n",
    "t4 = torch.tensor([\n",
    "    [[1, 2, 3],\n",
    "    [4, 5, 6]],\n",
    "    [[7, 8, 9],\n",
    "    [10, 11, 12]]])\n",
    "t4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensors can have any number of dimensions, and different lengths along each dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(4.)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "torch.Size([])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(t1)\n",
    "t1.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[1., 2.],\n",
      "        [3., 4.],\n",
      "        [5., 6.]])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "torch.Size([3, 2])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(t3)\n",
    "t3.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([1., 2., 3.])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "torch.Size([3])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(t2)\n",
    "t2.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[[ 1,  2,  3],\n",
      "         [ 4,  5,  6]],\n",
      "\n",
      "        [[ 7,  8,  9],\n",
      "         [10, 11, 12]]])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 2, 3])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(t4)\n",
    "t4.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensor operations and gradients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(3.), tensor(4., requires_grad=True), tensor(5., requires_grad=True))"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.tensor(3.)\n",
    "w = torch.tensor(4., requires_grad = True)\n",
    "b = torch.tensor(5., requires_grad = True)\n",
    "\n",
    "x, w, b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(17., grad_fn=<AddBackward0>)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = w * x + b\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, y is a tensor with the value 3 * 4 + 5 = 17. What makes PyTorch special is that we can automatically compute the derivative of y w.r.t. the tensors that have requires_grad set to True i.e. w and b. To compute the derivatives, we can call the .backward method on our result y."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# compute derivative\n",
    "y.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The derivatives of y w.r.t. the input tensors are stored in the .grad property of the respective tensors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dy/dw: tensor(3.)\n",
      "dy/dx: None\n",
      "dy/db: tensor(1.)\n"
     ]
    }
   ],
   "source": [
    "# Display gradients\n",
    "print('dy/dw:', w.grad)\n",
    "print('dy/dx:', x.grad)\n",
    "print('dy/db:', b.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, dy/dw has the same value as x, i.e., 3, and dy/db has the value 1. Note that x.grad is None because x doesn't have requires_grad set to True.\n",
    "\n",
    "The \"grad\" in w.grad is short for gradient, which is another term for derivative. The term gradient is primarily used while dealing with vectors and matrices."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Note: A matrix is a special type of tensor but a tensor need not to be a matrix always. Tensor can be a vector, a number, a 3D array with regular shape."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensor Fuctions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apart from arithmetic operations, the torch module also contains many functions for creating and manipulating tensors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[34, 34],\n",
       "        [34, 34],\n",
       "        [34, 34]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t6 = torch.full((3, 2), 34)\n",
    "t6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.,  2.],\n",
       "        [ 3.,  4.],\n",
       "        [ 5.,  6.],\n",
       "        [34., 34.],\n",
       "        [34., 34.],\n",
       "        [34., 34.]])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# to concatenate two tensors, they must have same dimensions\n",
    "t7 = torch.cat((t3, t6))\n",
    "t7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.8415,  0.9093],\n",
       "        [ 0.1411, -0.7568],\n",
       "        [-0.9589, -0.2794],\n",
       "        [ 0.5291,  0.5291],\n",
       "        [ 0.5291,  0.5291],\n",
       "        [ 0.5291,  0.5291]])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t8 = torch.sin(t7)\n",
    "t8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.5403, -0.4161],\n",
       "        [-0.9900, -0.6536],\n",
       "        [ 0.2837,  0.9602],\n",
       "        [-0.8486, -0.8486],\n",
       "        [-0.8486, -0.8486],\n",
       "        [-0.8486, -0.8486]])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t9 = torch.cos(t7)\n",
    "t9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.5574, -2.1850],\n",
       "        [-0.1425,  1.1578],\n",
       "        [-3.3805, -0.2910],\n",
       "        [-0.6235, -0.6235],\n",
       "        [-0.6235, -0.6235],\n",
       "        [-0.6235, -0.6235]])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t10 = torch.tan(t7)\n",
    "t10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ 1.5574, -2.1850],\n",
       "         [-0.1425,  1.1578]],\n",
       "\n",
       "        [[-3.3805, -0.2910],\n",
       "         [-0.6235, -0.6235]],\n",
       "\n",
       "        [[-0.6235, -0.6235],\n",
       "         [-0.6235, -0.6235]]])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t11 = t10.reshape(3, 2, 2)\n",
    "t11"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://pytorch.org/docs/stable/torch.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interoperability with NumPy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numpy is a popular open-source library used for mathematical and scientific computing in Python. It enables efficient operations on large multi-dimensional arrays and has a vast ecosystem of supporting libraries, including:\n",
    "\n",
    "    Pandas for file I/O and data analysis\n",
    "    Matplotlib for plotting and visualization\n",
    "    OpenCV for image and video processing\n",
    "\n",
    "If you're interested in learning more about Numpy and other data science libraries in Python, check out this tutorial series: https://jovian.ai/aakashns/python-numerical-computing-with-numpy .\n",
    "\n",
    "Instead of reinventing the wheel, PyTorch interoperates well with Numpy to leverage its existing ecosystem of tools and libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 2.],\n",
       "       [4., 5.]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = np.array([[1, 2], [4, 5.]])\n",
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can convert a NumPy array to a PyTorch tensor using torch.from_numpy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 2.],\n",
       "        [4., 5.]], dtype=torch.float64)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = torch.from_numpy(x)\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(dtype('float64'), torch.float64)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.dtype, y.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can convert a PyTorch tensor to a Numpy array using the .numpy method of a tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 2.],\n",
       "       [4., 5.]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = y.numpy()\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('float64')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The interoperability between PyTorch and Numpy is essential because most datasets you'll work with will likely be read and preprocessed as Numpy arrays.\n",
    "\n",
    "You might wonder why we need a library like PyTorch at all since Numpy already provides data structures and utilities for working with multi-dimensional numeric data. There are two main reasons:\n",
    "\n",
    "Autograd: The ability to automatically compute gradients for tensor operations is essential for training deep learning models.\n",
    "\n",
    "GPU support: While working with massive datasets and large models, PyTorch tensor operations can be performed efficiently using a Graphics Processing Unit (GPU). Computations that might typically take hours can be completed within minutes using GPUs.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}