quickstart_Pendulum_v1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/AI4Finance-Foundation/ElegantRL/blob/master/quickstart_Pendulum_v1.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c1gUG3OCJ5GS"
   },
   "source": [
    "# **Pendulum-v1 Example in ElegantRL-HelloWorld**\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DbamGVHC3AeW"
   },
   "source": [
    "# **Part 1: Install ElegantRL**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "U35bhkUqOqbS",
    "outputId": "e509246b-ab75-40f3-9484-9ff67f115238"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting git+https://github.com/AI4Finance-LLC/ElegantRL.git\n",
      "  Cloning https://github.com/AI4Finance-LLC/ElegantRL.git to /tmp/pip-req-build-d9u596je\n",
      "  Running command git clone --filter=blob:none --quiet https://github.com/AI4Finance-LLC/ElegantRL.git /tmp/pip-req-build-d9u596je\n",
      "  Resolved https://github.com/AI4Finance-LLC/ElegantRL.git to commit 59fe4b4a1cca0a28e8a5fa4fb80eed9b5d472978\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: gym in ./.env-erl/lib/python3.10/site-packages (from elegantrl==0.3.6) (0.25.2)\n",
      "Requirement already satisfied: matplotlib in ./.env-erl/lib/python3.10/site-packages (from elegantrl==0.3.6) (3.7.2)\n",
      "Requirement already satisfied: numpy in ./.env-erl/lib/python3.10/site-packages (from elegantrl==0.3.6) (1.25.1)\n",
      "Requirement already satisfied: torch in ./.env-erl/lib/python3.10/site-packages (from elegantrl==0.3.6) (2.0.1)\n",
      "Requirement already satisfied: cloudpickle>=1.2.0 in ./.env-erl/lib/python3.10/site-packages (from gym->elegantrl==0.3.6) (2.2.1)\n",
      "Requirement already satisfied: gym-notices>=0.0.4 in ./.env-erl/lib/python3.10/site-packages (from gym->elegantrl==0.3.6) (0.0.8)\n",
      "Requirement already satisfied: box2d-py==2.3.5 in ./.env-erl/lib/python3.10/site-packages (from gym->elegantrl==0.3.6) (2.3.5)\n",
      "Requirement already satisfied: pygame==2.1.0 in ./.env-erl/lib/python3.10/site-packages (from gym->elegantrl==0.3.6) (2.1.0)\n",
      "Requirement already satisfied: swig==4.* in ./.env-erl/lib/python3.10/site-packages (from gym->elegantrl==0.3.6) (4.1.1)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in ./.env-erl/lib/python3.10/site-packages (from matplotlib->elegantrl==0.3.6) (1.4.4)\n",
      "Requirement already satisfied: packaging>=20.0 in ./.env-erl/lib/python3.10/site-packages (from matplotlib->elegantrl==0.3.6) (23.1)\n",
      "Requirement already satisfied: pillow>=6.2.0 in ./.env-erl/lib/python3.10/site-packages (from matplotlib->elegantrl==0.3.6) (10.0.0)\n",
      "Requirement already satisfied: fonttools>=4.22.0 in ./.env-erl/lib/python3.10/site-packages (from matplotlib->elegantrl==0.3.6) (4.41.0)\n",
      "Requirement already satisfied: python-dateutil>=2.7 in ./.env-erl/lib/python3.10/site-packages (from matplotlib->elegantrl==0.3.6) (2.8.2)\n",
      "Requirement already satisfied: pyparsing<3.1,>=2.3.1 in ./.env-erl/lib/python3.10/site-packages (from matplotlib->elegantrl==0.3.6) (3.0.9)\n",
      "Requirement already satisfied: contourpy>=1.0.1 in ./.env-erl/lib/python3.10/site-packages (from matplotlib->elegantrl==0.3.6) (1.1.0)\n",
      "Requirement already satisfied: cycler>=0.10 in ./.env-erl/lib/python3.10/site-packages (from matplotlib->elegantrl==0.3.6) (0.11.0)\n",
      "Requirement already satisfied: typing-extensions in ./.env-erl/lib/python3.10/site-packages (from torch->elegantrl==0.3.6) (4.7.1)\n",
      "Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in ./.env-erl/lib/python3.10/site-packages (from torch->elegantrl==0.3.6) (11.7.99)\n",
      "Requirement already satisfied: jinja2 in ./.env-erl/lib/python3.10/site-packages (from torch->elegantrl==0.3.6) (3.1.2)\n",
      "Requirement already satisfied: nvidia-nvtx-cu11==11.7.91 in ./.env-erl/lib/python3.10/site-packages (from torch->elegantrl==0.3.6) (11.7.91)\n",
      "Requirement already satisfied: nvidia-cufft-cu11==10.9.0.58 in ./.env-erl/lib/python3.10/site-packages (from torch->elegantrl==0.3.6) (10.9.0.58)\n",
      "Requirement already satisfied: nvidia-curand-cu11==10.2.10.91 in ./.env-erl/lib/python3.10/site-packages (from torch->elegantrl==0.3.6) (10.2.10.91)\n",
      "Requirement already satisfied: nvidia-cusolver-cu11==11.4.0.1 in ./.env-erl/lib/python3.10/site-packages (from torch->elegantrl==0.3.6) (11.4.0.1)\n",
      "Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in ./.env-erl/lib/python3.10/site-packages (from torch->elegantrl==0.3.6) (11.7.99)\n",
      "Requirement already satisfied: nvidia-cusparse-cu11==11.7.4.91 in ./.env-erl/lib/python3.10/site-packages (from torch->elegantrl==0.3.6) (11.7.4.91)\n",
      "Requirement already satisfied: sympy in ./.env-erl/lib/python3.10/site-packages (from torch->elegantrl==0.3.6) (1.12)\n",
      "Requirement already satisfied: nvidia-cuda-cupti-cu11==11.7.101 in ./.env-erl/lib/python3.10/site-packages (from torch->elegantrl==0.3.6) (11.7.101)\n",
      "Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in ./.env-erl/lib/python3.10/site-packages (from torch->elegantrl==0.3.6) (11.10.3.66)\n",
      "Requirement already satisfied: triton==2.0.0 in ./.env-erl/lib/python3.10/site-packages (from torch->elegantrl==0.3.6) (2.0.0)\n",
      "Requirement already satisfied: networkx in ./.env-erl/lib/python3.10/site-packages (from torch->elegantrl==0.3.6) (3.1)\n",
      "Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in ./.env-erl/lib/python3.10/site-packages (from torch->elegantrl==0.3.6) (8.5.0.96)\n",
      "Requirement already satisfied: filelock in ./.env-erl/lib/python3.10/site-packages (from torch->elegantrl==0.3.6) (3.12.2)\n",
      "Requirement already satisfied: nvidia-nccl-cu11==2.14.3 in ./.env-erl/lib/python3.10/site-packages (from torch->elegantrl==0.3.6) (2.14.3)\n",
      "Requirement already satisfied: wheel in ./.env-erl/lib/python3.10/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch->elegantrl==0.3.6) (0.40.0)\n",
      "Requirement already satisfied: setuptools in ./.env-erl/lib/python3.10/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch->elegantrl==0.3.6) (59.6.0)\n",
      "Requirement already satisfied: lit in ./.env-erl/lib/python3.10/site-packages (from triton==2.0.0->torch->elegantrl==0.3.6) (16.0.6)\n",
      "Requirement already satisfied: cmake in ./.env-erl/lib/python3.10/site-packages (from triton==2.0.0->torch->elegantrl==0.3.6) (3.26.4)\n",
      "Requirement already satisfied: six>=1.5 in ./.env-erl/lib/python3.10/site-packages (from python-dateutil>=2.7->matplotlib->elegantrl==0.3.6) (1.16.0)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in ./.env-erl/lib/python3.10/site-packages (from jinja2->torch->elegantrl==0.3.6) (2.1.3)\n",
      "Requirement already satisfied: mpmath>=0.19 in ./.env-erl/lib/python3.10/site-packages (from sympy->torch->elegantrl==0.3.6) (1.3.0)\n",
      "Building wheels for collected packages: elegantrl\n",
      "  Building wheel for elegantrl (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for elegantrl: filename=elegantrl-0.3.6-py3-none-any.whl size=197344 sha256=bae2b59a1422fe6be6b8956e540c5b256076cdd4e4f69caf5825446f07ae1558\n",
      "  Stored in directory: /tmp/pip-ephem-wheel-cache-d4krv4ya/wheels/85/a7/e7/37369f52de5c5e77ba80ec8d0dd898ef99fbb23707e77a3958\n",
      "Successfully built elegantrl\n",
      "Installing collected packages: elegantrl\n",
      "Successfully installed elegantrl-0.3.6\n"
     ]
    }
   ],
   "source": [
    "# install elegantrl library\n",
    "!pip install git+https://github.com/AI4Finance-LLC/ElegantRL.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UVdmpnK_3Zcn"
   },
   "source": [
    "# **Part 2: Import Packages**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "AAPdjovQrTpE"
   },
   "outputs": [],
   "source": [
    "import gym\n",
    "# from helloworld import *\n",
    "\n",
    "gym.logger.set_level(40)  # Block warning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3n8zcgcn14uq"
   },
   "source": [
    "# **Part 3: Specify Agent and Environment**\n",
    "\n",
    "*   **agent**: chooses a agent (DRL algorithm) from a set of agents in the [directory](https://github.com/AI4Finance-Foundation/ElegantRL/tree/master/elegantrl/agents).\n",
    "*   **env**: creates an environment for your agent.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "E03f6cTeajK4"
   },
   "outputs": [],
   "source": [
    "from elegantrl.train.config import Config\n",
    "from elegantrl.agents.AgentSAC import AgentSAC\n",
    "\n",
    "env = gym.make\n",
    "\n",
    "env_args = {\n",
    "    \"id\": \"Pendulum-v1\",\n",
    "    \"env_name\": \"Pendulum-v1\",\n",
    "    \"num_envs\": 1,\n",
    "    \"max_step\": 1000,\n",
    "    \"state_dim\": 3,\n",
    "    \"action_dim\": 1,\n",
    "    \"if_discrete\": False,\n",
    "    \"reward_scale\": 2**-1,\n",
    "    \"gpu_id\": 0, # if you have GPU\n",
    "}\n",
    "args = Config(AgentSAC, env_class=env, env_args=env_args)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rcFcUkwfzHLE"
   },
   "source": [
    "# **Part 4: Specify hyper-parameters**\n",
    "A list of hyper-parameters is available [here](https://elegantrl.readthedocs.io/en/latest/api/config.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "9WCAcmIfzGyE"
   },
   "outputs": [],
   "source": [
    "args.max_step = 1000\n",
    "args.reward_scale = 2**-1  # RewardRange: -1800 < -200 < -50 < 0\n",
    "args.gamma = 0.97\n",
    "args.target_step = args.max_step\n",
    "args.eval_times = 2**3\n",
    "args.gpu_id = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "z1j5kLHF2dhJ"
   },
   "source": [
    "# **Part 5: Train and Evaluate the Agent**\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "KGOPSD6da23k",
    "outputId": "1c9cc4fe-bbc8-4dc4-aa37-346ab110c5ab"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| Arguments Remove cwd: ./Pendulum-v1_SAC_0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/adhi/ElegantRL/.env-erl/lib/python3.10/site-packages/gym/utils/passive_env_checker.py:241: DeprecationWarning: `np.bool8` is a deprecated alias for `np.bool_`.  (Deprecated NumPy 1.24)\n",
      "  if not isinstance(terminated, (bool, np.bool8)):\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "| Evaluator:\n",
      "| `step`: Number of samples, or total training steps, or running times of `env.step()`.\n",
      "| `time`: Time spent from the start of training to this moment.\n",
      "| `avgR`: Average value of cumulative rewards, which is the sum of rewards in an episode.\n",
      "| `stdR`: Standard dev of cumulative rewards, which is the sum of rewards in an episode.\n",
      "| `avgS`: Average of steps in an episode.\n",
      "| `objC`: Objective of Critic network. Or call it loss function of critic network.\n",
      "| `objA`: Objective of Actor network. It is the average Q value of the critic network.\n",
      "################################################################################\n",
      "ID     Step    Time |    avgR   stdR   avgS  stdS |    expR   objC   objA   etc.\n",
      "0  5.12e+02       5 |-1177.22  352.9    200     0 |   -3.67   4.57  -0.17   0.35\n",
      "0  2.10e+04     188 |-1131.77   62.5    200     0 |   -2.88   2.20 -86.57   0.12\n"
     ]
    }
   ],
   "source": [
    "from elegantrl.train.run import train_agent\n",
    "\n",
    "train_agent(args)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JPXOxLSqh5cP"
   },
   "source": [
    "Understanding the above results::\n",
    "*   **Step**: the total training steps.\n",
    "*  **MaxR**: the maximum reward.\n",
    "*   **avgR**: the average of the rewards.\n",
    "*   **stdR**: the standard deviation of the rewards.\n",
    "*   **objA**: the objective function value of Actor Network (Policy Network).\n",
    "*   **objC**: the objective function value (Q-value)  of Critic Network (Value Network)."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "quickstart_Pendulum-v1.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}