Bhargav_Solanki_assignment_12.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please fill in your name and that of your teammate.\n",
    "\n",
    "You: Bhargav Solanki\n",
    "\n",
    "Teammate: Carmen Sangro Prieto"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Welcome to the twelfth lab. It's finally time to look closely at the foundations of Reinforcement Learning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to pass the lab?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below you find the exercise questions. Each question awarding points is numbered and states the number of points like this: **[0pt]**. To answer a question, fill the cell below with your answer (markdown for text, code for implementation). Incorrect or incomplete answers are in principle worth 0 points: to assign partial reward is only up to teacher discretion. Over-complete answers do not award extra points (though they are appreciated and will be kept under consideration). Save your work frequently! (`ctrl+s`)\n",
    "\n",
    "**You need at least 14 points (out of 21 available) to pass** (66%)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Fundamentals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.1 **[1pt]** Explain the Reinforcement Learning paradigm in English. Use the words Environment, Agent, Action and Feedback."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reinforcement learning is the third paradigm in ML, other two being Supervised learning and Unsupervised learning. RL is different than the other two because it does not necessariliy need labels to train. On the contraray it performs exploration. \n",
    "It follows the Markow's decision models. There is a $agent$ in a $environment$. The agent can perform an $action$ in the environment. The agent can observe either the entire environment or partial. The goal of RL is to maximise rewards. Based on exploration and exploitation, the environment reacts and gives a $feedback$ that determines the next state in time. The agent's goal is to maximise reward by choosing the best action, based on previous exploitation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.2 **[1pt]** Explain the equation for (pseudo-)Regret in English."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$R (T) = T max(\\mu_1, \\mu_2) - (E (N_1)\\mu_1 + E(N_2)\\mu_2)$$\n",
    "Pseudo regret is difference in maximum observed mean across the observations minus the sum of the expected values for a given action times mean for each action. It basically means the difference between an optimal agent vs current agent. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3 **[1pt]** Explain in English the importance of _exploration vs. exploitation_ in MAB."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Exploration: Agent decides to explore the environment so that it has more values and to get a mean closer to the true mean.\n",
    "Exploitation: It uses the previously found means and chooses the action with maximum mean with aiming to maximise the reward.\n",
    "It is important to have a balance between the because, pure exploration ad pure explotation alone do not maximise the reward. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.4 **[1pt]** Explain what would happen if you had a discount factor $\\gamma \\gt 1$ in the Ice Maze example from the lecture."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If $\\gamma \\gt 1$ , \n",
    "1)  The values will not converge and will go to infinity\n",
    "2) discount factor , when 0 gives more importance to short term rewards and 1 gives importance to long term rewards. So if it is greater than 1, it will not only focus completely on long term rewards, but will actually scale it, and will be incorrect."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.5 **[2pt]** The Bellman Equation is at the heart of the classical Reinforcement Learning framework. Write it below in Latex, then explain each of the terms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Equation:\n",
    "$$ \\hat{\\mathcal{Q}}(s_t, a_t) = ( 1 - \\alpha) \\hat{\\mathcal{Q}}(s_t, a_t) + \\alpha(R(s_t, a_t) + \\gamma \\hat{\\mathcal{Q}}(s_{t+1}, a_{t+1})) $$\n",
    "Explanation:\n",
    "\n",
    "The action to be taken by the agent should be the one that is the most optimal i.e. the one that maximizes the reward (R) after a certain action (a) taking into account future rewards, given a certain discount factor ($\\gamma$)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Q-Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The [OpenAI Gym](https://gym.openai.com/) maintains a broad set of Reinforcement Learning benchmarks. It is ready to import on Colab, or you can add to your local installation the environments we will use for this assignment using `pipenv install gym[toy_text,classic_control]`.\n",
    "- You can find the Ice Maze (named Frozen Lake) [here](https://github.com/openai/gym/blob/master/gym/envs/toy_text/frozen_lake.py).\n",
    "- The Classic Control games are also relatively easy to solve, and much nicer to render. Feel free to give it a try. You can save a video with `env = gym.wrappers.Monitor(env, 'video', force = True)` right after the `gym.make()` call. More on this topic [here](https://hub.packtpub.com/openai-gym-environments-wrappers-and-monitors-tutorial/).\n",
    "- To get you started, here is a working example of a random policy on the framework, which I took from their front page and customized."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1 **[2pt]** Write a Python function `choose_action` that takes as arguments `Q`, `state`, `epsilon`, `actions` and returns an action chosen according to Q-values using an epsilon-greedy policy.\n",
    "\n",
    "- `Q` is a dictionary that has states as keys. Each value is a numpy array with the Q-values corresponding to each action from the state used as key.\n",
    "- You cannot initialize `Q` with all possible states, because they can be infinite or unreachable. Using `Q[state]` with missing key `state` will throw an error; use instead `Q.get(state)` then check if its return `is None`.\n",
    "- `state` can be simply a number.\n",
    "- `actions` is a list of possible actions between which to choose."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from random import uniform, randint\n",
    "import numpy as np\n",
    "\n",
    "def choose_action(q, state, epsilon, actions):\n",
    "    val = uniform(0,1)\n",
    "    #print(val)\n",
    "    if val > epsilon and q.get(state):\n",
    "        return np.argmax(q.get(state))\n",
    "    else:\n",
    "        return randint(0, len(actions)-1)\n",
    "        \n",
    "#Q = {1:[0,0,1], 2:[0,1,0]}\n",
    "#choose_action(Q, 1, 0.0, [])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import gym\n",
    "env = gym.make(\"LunarLander-v2\")\n",
    "env.action_space.seed(42)\n",
    "\n",
    "observation= env.reset()\n",
    "\n",
    "for _ in range(1000):\n",
    "    #env.render()\n",
    "    observation, reward, done, info = env.step(env.action_space.sample())\n",
    "\n",
    "    if done:\n",
    "        observation = env.reset()\n",
    "\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HINT: here is some code I used for testing `choose_action()` and expected outputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22222222222222222222222222222222222222222222222222\n",
      "11111111111111111111111111111111111111111111111111\n",
      "02101100012201201001112210112220122122101120212122"
     ]
    }
   ],
   "source": [
    "Q = {1:[0,0,1], 2:[0,1,0]}\n",
    "for _ in range(50):\n",
    "    print(choose_action(Q, 1, 0.0, []), end='') #=> 2\n",
    "print()\n",
    "for _ in range(50):\n",
    "    print(choose_action(Q, 2, 0.0, []), end='') #=> 1\n",
    "print()\n",
    "for _ in range(50):\n",
    "    print(choose_action(Q, 2, 1.0, range(3)), end='') #=> random action"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.2 **[2pt]** Write a Python function `update_Q` that takes as arguments `Q`, `state`, `action`, `next_state`, `reward` and `num_actions`, and updates the `Q` dictionary according to Q-Learning. \n",
    "\n",
    "- Here you will need to initialize `Q[state]` if previously unexplored.\n",
    "- The future expected reward is the highest Q value from the next state -- or zero if unexplored. That's what the last argument is for.\n",
    "- I'll leave the testing to you here. You may feel like just running the next question for testing (called _integration testing_ ), but testing it in isolation ( _unit testing_ ) is a much more controlled verification which should not be skipped (i.e. _easier_ )."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1: [0, 0, 1], 2: [0, 0.6, 0]}\n"
     ]
    }
   ],
   "source": [
    "def update_Q(q, state, action, next_state, reward, num_actions, alpha, gamma):\n",
    "    if state not in q:\n",
    "        q[state] = [0]*num_actions\n",
    "    #old_q = 0\n",
    "    # I tried with choosing the action as old_q, but performance was so much better \n",
    "    # if I chose max. Non intuitive, but still works\n",
    "    #old_q = q.get(state)[action]\n",
    "    old_q = max(q.get(state))\n",
    "    q_next_step_list = q.get(next_state)\n",
    "    q_next_step = 0\n",
    "    calculated_q = 0\n",
    "    if q_next_step_list and len(q_next_step_list)>0:\n",
    "        q_next_step = max(q_next_step_list)\n",
    "    #calculated_q +=   (1-alpha)*(old_q ) + (alpha * (reward + (gamma*q_next_step)))\n",
    "    calculated_q =   (1-alpha)*old_q  +  (alpha)*(reward + (gamma*q_next_step))\n",
    "    q[state][np.argmax(q.get(state))] = calculated_q\n",
    "Q = {1:[0,0,1], 2:[0,1,0]} \n",
    "update_Q(q=Q, state=2,action=1,next_state=2,reward=0, num_actions=2,alpha=1,gamma=0.6)\n",
    "\n",
    "print(Q)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.3 **[1pt]** Run the code below to successfully use Q-Learning to solve the OpenAI Gym `FrozenLake` environment.\n",
    "\n",
    "- The code below is already correct, it just needs the two functions from above.\n",
    "- If you got both previous answers right, yes this is a free extra point for you :) just run it to produce the outputs.\n",
    "- If this does not run correctly, something is likely wrong with your implementation above. Go back and test a bit more.\n",
    "- The parameters should work as they are, but feel free to play with them and make sure by the last epochs the environment is solved more often than not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllWllllllllllllllllllllllllllllllllllllllllWllWlllllllWllllllllWllWlllllllllllllWlllllWlWllWlllllllWlllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllllWlWllllllllllllllllllllllllllllllWlllllllllllllllllllllllllllllllllllllllllllllllllWlllWllWllWllllWllllWlllWlllllWllWWllllWllllllllllllllWlllllllllllllllllWlllllllllllllWllllllllll\n"
     ]
    }
   ],
   "source": [
    "# Initialization and settings\n",
    "import gym\n",
    "# My environment dint allow FrozenLake-v0, saying it was deprecated. So I switched to FrozenLake-v1\n",
    "env = gym.make(\"FrozenLake-v0\", is_slippery=False)\n",
    "num_episodes = 1000\n",
    "max_nsteps = 30\n",
    "gamma = 0.9\n",
    "alpha = 0.6\n",
    "epsilon = 0.3\n",
    "\n",
    "num_actions = env.action_space.n\n",
    "Q = {} # hashing states to per-action reward arrays\n",
    "\n",
    "# Loop for each episode\n",
    "for ith_episode in range(num_episodes):\n",
    "\n",
    "    # Reset the environment (and obtain first observation)\n",
    "    state = env.reset()\n",
    "    total_reward = 0\n",
    "\n",
    "    # For each timestep\n",
    "    for t in range(max_nsteps):\n",
    "        # Choose action according to Q-values using epsilon-greedy policy\n",
    "        # THIS SHOULD USE YOUR IMPLEMENTATION\n",
    "        action = choose_action(Q, state, epsilon, range(num_actions))\n",
    "        # Execute action: get reward, move env to next state\n",
    "        next_state, reward, done, info = env.step(action)\n",
    "        # Add negative reward for falling in hole\n",
    "        if done and reward == 0: reward = -1\n",
    "\n",
    "        # Some useful printing to verify progression - (W)in or (l)ose\n",
    "        if reward == -1: print('l', end='', flush=True)\n",
    "        if reward == 1: print('W', end='', flush=True)\n",
    "\n",
    "        # Accumulate reward. Note this environment only rewards on termination though.\n",
    "        total_reward += reward\n",
    "        \n",
    "        # Update Q function for current state and action\n",
    "        # THIS SHOULD USE YOUR IMPLEMENTATION\n",
    "        update_Q(Q, state, action, next_state, reward, num_actions, alpha, gamma)\n",
    "        \n",
    "        # Update internal state\n",
    "        state = next_state\n",
    "        \n",
    "        # Terminate if episode ended\n",
    "        if done: \n",
    "            break\n",
    "print()\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.4 **[1pt]** Write a Python code snippet that satisfies the following requirements: (i) runs a fully-greedy policy, (ii) using the Q-values learned in question 2.3, (iii) renders the environment in each step, and (iv) write less than 10 lines of code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The point of this question is for you to go back to 2.3, study it and understand it. If you get how it works, this snippet will take but a second.\n",
    "- For the subpoint (iii) you should look at the method `render()` of the OpenAI environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{0: [-0.058019679864676876, -0.6953917535587233, -0.7275265741509098, -0.7358703061305361], 4: [-0.9163046766924723, -0.9474150506070621, -0.9359999999999999, -0.07557190691702242], 8: [-0.8819366975930727, -0.8784, -0.8858959949803245, -0.014150142696297451], 9: [-0.7185108827363973, 0.46835939623848555, -0.7492767983816536, -0.84], 1: [-0.03573313052552253, -0.9145391063893599, -0.9449264037188789, -0.935168294360051], 13: [-0.84, -0.7873797504, 0.7611393100022474, -0.692543340674254], 14: [-0.03440046034913537, -0.03440046034913537, 0.8940054622860665, 0], 2: [-0.17894660802901763, -0.5749374986785645, -0.6261412909657503, -0.6525292809003611], 3: [-0.21096011014211197, -0.91584, -0.8923633394250927, -0.88701590016], 6: [-0.84, -0.8895303373949051, -0.84, -0.2602475663536953], 10: [-0.7220149325063397, 0.6772854823641865, -0.6, -0.6974309737031343]}\n",
      "\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Right)\n",
      "S\u001b[41mF\u001b[0mFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Down)\n",
      "SFFF\n",
      "\u001b[41mF\u001b[0mHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Up)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Right)\n",
      "S\u001b[41mF\u001b[0mFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "  (Left)\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n"
     ]
    }
   ],
   "source": [
    "import gym\n",
    "def run_experiment():\n",
    "    state = env.reset()\n",
    "    for i in range(0, 1):\n",
    "        done = False\n",
    "        count = 30\n",
    "        while not done and count >=0:\n",
    "            count -=1\n",
    "            env.render()\n",
    "            action = choose_action(Q, state, epsilon, range(num_actions))\n",
    "            next_state, reward, done, info = env.step(action)\n",
    "            #update_Q(Q, state, action, next_state, reward, num_actions, alpha, gamma)\n",
    "            state = next_state\n",
    "            if reward>0:\n",
    "                print(\"Victory is mine!!\")\n",
    "print(Q)\n",
    "run_experiment()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.5 **[1pt]** Modify the code from question 2.3 to record the cumulative reward and number of time-steps for each episode. Then plot them with two line plots using Seaborn.\n",
    "\n",
    "- Think carefully: you need to augment the code above, by first initializing a proper data structure, then recording the statistics at the end of each episode. Finally plot them below.\n",
    "- If the plot looks uniform while you expect improvement, try re-running it a few times. Remember that the exploration capabilities of this methods are pretty limited, so short runs (such as this) depend significantly on the initialization conditions (i.e. luck)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "llllllllllllllllllllllllllllllllllllllllllWllllllWlWWWWllWlWWWWWlWllWlllWWWWWlWWlWllWWlWlllWWlWlWllWllWWlWWWlWWllllllllllllllllllllllllllllWllWWllWWlWllllllllWlWWWWlWWllWWWWllllllWllllllllWllllWlllllllllllllWWllllllWlWWWllWWlWllWWlWlWlWWllWWlWWlllWWWWlWWWllllWlWWWWWWlWlWWllWWlWlllWWWWlWllWWllllllWWlWWWWWWWWWWlWWlllWWlWWWlllWlWWWWWWlWlWWWWWWlWWlWlWllWWllWWWlWWlWWWWWlWWlWWWWWllllWlWWlWllWllWWWlWWWllWlllllWWWWlWllWWllWWWWWlWWWWWWlWWWllWWlWWWlWWWWWlWlWWlWllllWlWWWWWlWWlWWllllWllWlWllWWllllWlWllllllllWllWllllWllWlWllWlWllWlllllllllWllllllWlWlllllllllWWWWlWlllllWWllWlWWWlllWlWlWlWWWWllWllWlWlWWllllWlllWlWlllllWllllWWlWWWlllWlWllWlllWlllWlllWllllllWWWllWllWlllllllWllWllWlWllllWWlllWlWlWWllllllWlllllWWWlWWllllllWWWWlWWWWWlWWllWllllWWlllllllllWllWlWlllWWWlllllllllWllWWlWWlWW"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fcac8844670>]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD3CAYAAAAT+Z8iAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAvZUlEQVR4nO3deWBU9bn/8feZmcxktixAwh4gQFiNbAaQzWgVUbFqrVdyf9S2t/0ptbWgYBBlqWIVUbTaa9WqV6Ui0uJt/dlaWwFZAkRAFkE2I2sIkG2S2dfv749oFGVLSDjJzPP6y8zMmfM8mfDx5JtznqMppRRCCCHilkHvAoQQQjQvCXohhIhzEvRCCBHnJOiFECLOSdALIUScM+ldwOnEYjGi0cadDGQ0ao3etrWSnhOD9JwYLqTnpCTjaR9vkUEfjSpcLl+jtk1LszV629ZKek4M0nNiuJCeMzKcp31clm6EECLOSdALIUSck6AXQog4J0EvhBBxToJeCCHi3HkF/fbt25k8eTIAhw4dYtKkSRQUFDB37lxisRgAy5Yt45ZbbuG2225j1apVAAQCAX71q19RUFDAz3/+c6qqqpqpDSGEEGdyzqD/4x//yEMPPUQwGATgscceY+rUqSxZsgSlFCtWrKC8vJzFixezdOlSXnnlFRYtWkQoFOKtt94iJyeHJUuWcNNNN/H88883e0NCCCFOdc6gz8rK4rnnnqv/eteuXeTl5QEwduxY1q9fz44dOxg8eDBmsxmn00lWVhZ79uxhy5YtjBkzpv61GzZsaKY2hBCidTNW7sX8xfvN8t7nvGBq/PjxHD16tP5rpRSapgFgt9txu914PB6czq9P1Lfb7Xg8nlMe/+q158No1EhLszWoka+3NTR629ZKek4M0nOcioYwFD2NoWgRpHVDDbu1yXtu8JWxBsPXvwR4vV5SUlJwOBx4vd5THnc6nac8/tVrz4dcGdsw0nNikJ7jj+nEVpwrp2Os2kug9014xjxMaqzx+ddkV8b279+f4uJiANasWcOwYcPIzc1ly5YtBINB3G43JSUl5OTkMGTIEFavXl3/2qFDhzaqeCGEiCthP/aiR0hb/n20YA0117+G+5rfo6xtmmV3DT6iLywsZPbs2SxatIjs7GzGjx+P0Whk8uTJFBQUoJRi2rRpWCwWJk2aRGFhIZMmTSIpKYmnnnqqOXoQQohWI+loEc5V92OsPYR/wGS8l89CmU9/JN5UtJZ4z9hwOCpLNw0gPScG6bl104K12NfPx/rZEiKp3fHkLyTceeR3XtccQ81a5PRKIYSIJ+YD/8axeiYGXzm+wXfhvew+SLJetP1L0AshRDPR/JU41s4hef/fiLTti+u6V4lkXnrR65CgF0KIpqYUlv1/xbF2DlrIgzdvOr4hvwCjWZdyJOiFEKIJGdzHcKx+AMuhFYTbD8Gdv5Bo2z661iRBL4QQTUHFSN71Jvb1j6KpKJ7R8/Bf8hMwnP72fheTBL0QQlwgo+sLHKvux3xsI6Euo3FfsYBYaje9y6onQS+EEI0Vi2Dd9kfsHz+JMlpw5z9JoN9/wJdjYloKCXohhGgEY8VnOFfNIOnkdoI9xuMZ9ygxewe9yzotCXohhGiIaBDb5mexffLfKEsaNeNfINTz+hZ3FP9NEvRCCHGeTMe34Fw5A1P1PgJ9foBn9DxUcrreZZ2TBL0QQpxL2Ie9+Ams218h5uhIzQ1vEOp2pd5VnTcJeiGEOIukI2vrhpC5j+C/5A68Ix5AmR16l9UgEvRCCHEaWrAGe9HDWHe/TSS1B66blxPuNFzvshpFgl4IIb7F/MU/cax+EIO/At+Qu/FeNhVMF28IWVOToBdCiC9pvnIca2aTXPIekbb9cV3/P0Qyc/Uu64JJ0AshhFJY9i3HsXYeWtiHd3ghvsF3gTFJ78qahAS9ECKhGdylOD8qxHz4I8IdhuG+8kmi6b30LqtJSdALIRKTipG88w3sGx5DUwr3mIcJXPJj0Bp8K+0WT4JeCJFwjNUldeMLyj4m1HVs3RCylK56l9VsJOiFEIkjFsG69QXsm55GmZKpvXIRwb4/bNHjC5qCBL0QIiEYy3fhXDWdpPJPCWZPwD32UZQ9U++yLgoJeiFEfIsEsG3+HbZPnkclt6Hm2hfrhpAlEAl6IUTcMpVtwrlyOiZXCYG+t+EZNbtVDCFrahL0Qoj4E/Ji3/g41k9fI+bsjGvim4SzxuldlW4k6IUQcSXp8GqcHxVicJfiv+THeEfMBLNd77J0JUEvhIgLWqAaR9EjJO9ZRiStJ65b3iHS8TK9y2oRJOiFEK2eueTvOFc/hBaowjv0V/iG/RpMyXqX1WJI0AshWi3NexLn2oewlPyDcLuBuCf+iWjGAL3LanEk6IUQrY9SWPb8GUfRb9AiATwjH8A/6E4wSKSdjnxXhBCtiqH2SN0QsiNrCHfMw52/kGh6T73LatEk6IUQrYOKkfzpazg2PI7SNNxjHyUwcHJcDiFrahL0QogWz1i1v24I2fHNhLKuwD3ucWIpXfQuq9VoVNCHw2FmzpxJaWkpBoOBRx55BJPJxMyZM9E0jd69ezN37lwMBgPLli1j6dKlmEwmpkyZQn5+flP3IISIV9Ewts3PYdv0NCrJRu33niGY84O4H0LW1BoV9KtXryYSibB06VKKiop45plnCIfDTJ06leHDhzNnzhxWrFjBoEGDWLx4McuXLycYDFJQUMCoUaMwm81N3YcQIs6Yyj/F9OcZJJ3cSaDXRDxjHkbZMvQuq1VqVND36NGDaDRKLBbD4/FgMpnYtm0beXl5AIwdO5aioiIMBgODBw/GbDZjNpvJyspiz5495Oa2/nswCiGaScSPfdPTWLe+CLZ21Ex4mVD2tXpX1ao1KuhtNhulpaVMmDCB6upqXnjhBTZt2oT25a9Tdrsdt9uNx+PB6XTWb2e32/F4POd8f6NRIy3N1pjSMBoNjd62tZKeE0Mi9Kwd3oDx779Gq/qc2KX/B66Zj82cQnx3farm+JwbFfSvvfYao0eP5r777qOsrIw77riDcDhc/7zX6yUlJQWHw4HX6z3l8W8G/5lEowqXy9eY0khLszV629ZKek4M8dyzFvJg3/AY1p2vE3V2xX3jW4S7jiHNHL89n8mFfM4ZGafP10adl5SSklIf2KmpqUQiEfr3709xcTEAa9asYdiwYeTm5rJlyxaCwSBut5uSkhJycnIa1YAQIj6ZD60k/a0rSd75Br5Lf0bVpBWEu47Ru6y40qgj+h//+MfMmjWLgoICwuEw06ZNY+DAgcyePZtFixaRnZ3N+PHjMRqNTJ48mYKCApRSTJs2DYvF0tQ9CCFaIS1QjWPdPJL3LieS3hvXD/5KpMNQvcuKS5pSSuldxLeFw1FZumkA6TkxxE3PStUNIVvzEFrQhW/I3fiG3QPG7x4Exk3PDdAcSzdywZQQ4qIxeI/jWP0glgMfEM7IxX3jEqLt+utdVtyToBdCND+lSN69FHvRI2jRIJ6RD+If9HMZQnaRyHdZCNGsDDWH6oaQHV1HqNNwPPkLiaZl611WQpGgF0I0j1gU66f/g33jApRmxD3uMQID/lOGkOlAgl4I0eSMVftwrpxO0olPCHa7Es+4x4k5O+ldVsKSoBdCNJ1oCNsnz2Pb/CzKbKf26ucI9r5JhpDpTIJeCNEkTCe24Vw1HVPlHgK9v183hMzaVu+yBBL0QogLFfZj3/QU1m0vEbNlUHPdq4R6XKN3VeIbJOiFEI2WVLoBx6oZmGoO4u9fgPfyh1CWFL3LEt8iQS+EaDAt5Ma+/rdYdy0mmtIN1/ffJtxllN5liTOQoBdCNIj54Aocq2di8J7Ad+n/xTt8BiRZ9S5LnIUEvRDivGj+Shxr55K8/69E2vTBNf5FIh2G6F2WOA8S9EKIs1MKy+fv4lgzGy3kxnvZvfiG/hKMckvQ1kKCXghxRgZPGY7Vs7Ac/DfhzEG4r3ySaNu+epclGkiCXgjxXUqR/NkS7Ovno8XCeEbNwZ/7X2Aw6l2ZaAQJeiHEKQw1B3Guuh9z6XpCnS/Hnf8EsdTuepclLoAEvRCiTiyKdfvL2D9eiDIk4b5iAYH+BTK+IA5I0AshMFbuwbnyPpJObifY/Wo8435LzNFR77JEE5GgFyKRRUPYtjyHbcvvUZYUaq95nmCviXIUH2ck6IVIUKYTW3GunI6pai+BnJvxjP4NytpG77JEM5CgFyLRhP3Yixdi3fEyMXt7aq5/nVD3q/SuSjQjCXohEkjS0SKcq+7HWHsI/4DJeC+fhTI79S5LNDMJeiESgBaswb7+UayfLSGS2h3XTX8m3Hmk3mWJi0SCXog4Zz7wLxyrH8DgK8c3+C68l90nQ8gSjAS9EHFK81XgWDuH5M/fJdK2L67rXiWSeaneZQkdSNALEW+UwrLvf3Gsm4sW8uIdPgPf4CkyhCyBSdALEUcM7mM4Vj+A5dAKwu2H1A0ha5Ojd1lCZxL0QsQDFSN515vY1z+KpqJ4Rs/Df8lPZAiZACTohWj1jK4vcKyagflYMaEuo3FfsYBYaje9yxItiAS9EK1VLIJ120vYP34KZbTgzn+SQL//kPEF4jsk6IVohYwVn+FcOZ2k8h0Ee4zHM+5RYvYOepclWigJeiFak2gQ2+ZnsX3y3yhLGjXjXyDU83o5ihdn1eigf/HFF1m5ciXhcJhJkyaRl5fHzJkz0TSN3r17M3fuXAwGA8uWLWPp0qWYTCamTJlCfn5+U9YvRMLQjn5M+ru/wlS9n0CfW/GMnotKTte7LNEKGBqzUXFxMVu3buWtt95i8eLFHD9+nMcee4ypU6eyZMkSlFKsWLGC8vJyFi9ezNKlS3nllVdYtGgRoVCoqXsQIr6FfdjXzsX4+gS0sJeaG97A/b1nJOTFeWvUEf26devIycnh7rvvxuPxcP/997Ns2TLy8vIAGDt2LEVFRRgMBgYPHozZbMZsNpOVlcWePXvIzc096/sbjRppabbGlIbRaGj0tq2V9By/tAMfYfz7VLSaw6jLfk5s3EPYLE7iv/M6ifI5f1Nz9NyooK+urubYsWO88MILHD16lClTpqCUQvtyndBut+N2u/F4PDidX0/Gs9vteDyec75/NKpwuXyNKY20NFujt22tpOf4owVc2Nc/gnX320TSsvHcvBx7//y6nv3x2/e3xfvnfDoX0nNGxuknkTYq6NPS0sjOzsZsNpOdnY3FYuH48eP1z3u9XlJSUnA4HHi93lMe/2bwCyG+y/zF+zhWP4jBX4lvyN14L5sGpmS9yxKtWKPW6IcOHcratWtRSnHixAn8fj8jR46kuLgYgDVr1jBs2DByc3PZsmULwWAQt9tNSUkJOTlyObYQp6P5ynH+8y5S3/85MVsGrh++h3fkAxLy4oI16og+Pz+fTZs2ceutt6KUYs6cOXTp0oXZs2ezaNEisrOzGT9+PEajkcmTJ1NQUIBSimnTpmGxWJq6ByFaN6Ww7F1eN4Qs7Mc7vBDf4LvAmKR3ZSJOaEoppXcR3xYOR2WNvgGk59bL4C7F+VEh5sMfEe4wrG4IWXqv0742XnpuCOm5YZp0jV4IcYFUjOSdb2Df8BiaUrjHPEzgkh+D1qjVVCHOSoJeiIvMWF2Cc9UMkso+JtR1HO4rHieW0lXvskQck6AX4mKJhrFuexH7pqdRpmRqr3qaYJ9bZXyBaHYS9EJcBKbynThWTiepYifBntfhHjMfZc/UuyyRICTohWhOkQC2zb/D9snzqOQ21Fz7Yt0QMiEuIgl6IZqJqWwTzpXTMblKCPS9Dc+o2TKfRuhCgl6IphbyYt/4ONZPXyPm7Ixr4puEs8bpXZVIYBL0QjShpMMf4VxViMFzDH/uT/AOLwSzXe+yRIKToBeiCWiBahxFD5O8589E0nvhuuUdIh0v07ssIQAJeiEumLnk7zhXP4QWqMI79B58w+6R+TSiRZGgF6KRDN4TONY8hOWL9wm3G4h74p+IZgzQuywhvkOCXoiGUgrLnj/jKPoNWiSAZ+QD+AfdCQb55yRaJvnJFKIBDLVH6oaQHVlDuGMe7vyFRNN76l2WEGclQS/E+YhFsX76GvaNC1CahnvsowQGTpYhZKJVkKAX4hyMVfvrhpAd30wo6wrcVywg5uysd1lCnDcJeiHOJBrGtvUP2DY9g0qyUfu9Zwjm/ECGkIlWR4JeiNMwlX+Kc8V9mCo/I9BrIp4xD6NsGXqXJUSjSNAL8U0RP/ZNT2Pd+iIxa1tqJrxMKPtavasS4oJI0AvxpaRjG3GsnIGp5gD+frfjvfwhVHKa3mUJccEk6EXC00Ju7Bsex7rzdaIpWbhufItw1zF6lyVEk5GgFwnNfGgljo9mYvCU4bv0Z3iH3w9JNr3LEqJJSdCLhKQFqnGsm0fy3uVE0nNw/eCvRDoM1bssIZqFBL1ILEph+fw9HGsfQgvW4B3267ohZEaL3pUJ0Wwk6EXCMHiP41j9IJYDHxDOyMV941tE2/XXuywhmp0EvYh/SpG8eyn2okfQokE8Ix/EP+jnMoRMJAz5SRdxzVBzCOeq+zGXFhHqNAJP/hNE07L1LkuIi0qCXsSnWBTrp//z5RAyI+5xjxMYUCBDyERCkqAXccdYuRfnqukkndhKsNtVeK54jJijk95lCaEbCXoRP6IhbJ88j23z71BmB7VXP0ew900yhEwkPAl6ERdMJ7bhXDUdU+UeAr2/XzeEzNpW77KEaBEk6EXrFvZj//hJrNv/SMyWQc11rxLqcY3eVQnRolzQX6YqKysZN24cJSUlHDp0iEmTJlFQUMDcuXOJxWIALFu2jFtuuYXbbruNVatWNUnRQgAkla4n/e2rsW17kUC/SVRPWiUhL8RpNDrow+Ewc+bMITk5GYDHHnuMqVOnsmTJEpRSrFixgvLychYvXszSpUt55ZVXWLRoEaFQqMmKF4lJC9Zi+Me9pP31NjSlcH3/bTz5C1CWFL1LE6JFanTQL1iwgNtvv53MzEwAdu3aRV5eHgBjx45l/fr17Nixg8GDB2M2m3E6nWRlZbFnz56mqVwkJPPBFaS/dSWGbW/gG3QnVbd/SLjLKL3LEqJFa9Qa/TvvvEObNm0YM2YML730EgBKKbQvz26w2+243W48Hg9Op7N+O7vdjsfjOef7G40aaWmNmyBoNBoavW1rlRA9eysw/nsWhl1/QWX0Q93+JkkdBpOmd10XUUJ8zt8iPTeNRgX98uXL0TSNDRs2sHv3bgoLC6mqqqp/3uv1kpKSgsPhwOv1nvL4N4P/TKJRhcvla0xppKXZGr1taxXXPSuFZf/fcKydgxZy473sXnxDf0la27T47fkM4vpzPgPpuWEyMk6fr40K+jfffLP+vydPnsy8efNYuHAhxcXFDB8+nDVr1jBixAhyc3N55plnCAaDhEIhSkpKyMnJaVQDIvEYPMdwrJ6F5eCHhDMH4b7ySaJt++pdlhCtTpOdXllYWMjs2bNZtGgR2dnZjB8/HqPRyOTJkykoKEApxbRp07BYZBysOAcVI/mzJdjXP4oWC+MZNQd/7n+Bwah3ZUK0SppSSuldxLeFw1FZummAeOrZ4DqA86P7MZduINT5ctz5TxBL7f6d18VTz+dLek4MLWbpRogmF4ti3f4y9o8XogxJuPOfINBvkowvEKIJSNAL3Rkr9+BceR9JJ7cT7H41nnG/JeboqHdZQsQNCXqhn2gQ25bfY9vye5QlhdprnifYa6IcxQvRxCTohS5Mxz/BuWoGpqq9BHJuxjP6NyhrG73LEiIuSdCLiyvsw168EOv2l4k5OlBz/euEul+ld1VCxDUJenHRJB0twrnqfoy1h/AP/BHekQ+gzOe+gE4IcWEk6EWz04I12NfPx/rZW0RSu+O66c+EO4/UuywhEoYEvWhW5gP/wrH6AQy+cnyDp+DNuxdMVr3LEiKhSNCLZqH5KnCsnUPy5+8SadsX13WvEsm8VO+yhEhIEvSiaSmFZd87ONbORQv78A6fgW/wFDCa9a5MiBbNH44SjsaaZSKrBL1oMgb3MRyrZ2I5tJJw+yF1Q8jayBA7Ic5m29Ealm07xsaD1ViTDBQVXtnk+5CgFxdOxUje9Sfs63+LpqJ4Rs/Df8lPZAiZEF9y+cJU+EJsO1rDv/acJKZAAYeqfNQEIliTDAzvls7/GdalWfYvQS8uiNH1BY5VMzAfKybUZQzu/AXEUrL0LkuIiyYaUxyo8hGOxthwoJoK79e3S63xh9lZVsux2mD9Y70z7KRZkwAY0jWNwV1Sub5/JinJSc1WowS9aJxYBOu2l7B//BTKaMGd/ySBfv8h4wtE3IvGFP+7o4x/7y3nYJWPKl/4lOdTkk189a/AaNDo2c7Otf0y6dmuLuAvy0qrvxvfxSJBLxrMWPEZzpXTSSrfQbDHeDzjHiVm76B3WUI0q6MuPy9vOMQ/d58kqqBzajKXdk6lW7qVTqnJpFmT6JKWTO8Mh96lfocEvTh/0SC2zc9i++S/UZY0asa/QKjn9XIUL+JSIBzl3Z3H+efukwQjMQ5X+4nEFN/rk8FVORlc0avtRT8ybywJenFeTMe34Fw5HVP1fgJ9bsUzei4qOV3vsoS4IL5QlKIDVWw+7CIUjVHhDVHhCaFQHKn2E4oquqVb6dbGRr8OTn42IosOKcl6l91gEvTi7EJe7MVPYN3xKjFHJ1w3LCbcLV/vqoRoEJcvzKdltRytCXC02s/mI3XBftQVAMBiMpBuTcJk1OjexobJoNG3vZMJfTPJ63bx19SbmgS9OKOkI2twrirE6D6C/5I78I54AGVueeuPQnzlYKWPbaU1HKzyU+ENEo4q9p70UFoTOOV1w7qmkmpNYmzPtgzunMrIHm2wmAw6Vd38JOjFd2gBF/b1j2Dd/TaRtGxcNy8n3Gm43mUJAUA4GuN4bZBjNQEOVvk46QlRVhuoX4b5SqcUC0aDRsfUZK7pm8HAjin0b+/AZjZhMyfWNR4S9OIU5i/ex7H6QQz+SnxD7sZ72TQwtb41SRE/wtEY7392kn3lHjYddnHUVbd2/k0dUyxYTAYmDenMtf0yyW5rIzkpscL8bCToBQCa9yTOtbOxlPydcLsB1N7wOpGMS/QuSySIYCTGvpMe3MEIAO5AhGO1AY57w3y4+wS1gQgaMLCjk+sHtKdnWzvd29hon2Khg9MioX4OEvSJTikse5fjWDcXLezHO7wQ3+C7wNh8V+mJxKWU4rg7yP5yL0VfVHGsJsChah8V3hDhbx2lQ90FR6N7tOHGSzowsns6Scb4XUdvThL0CcxQexTn6kLMh1cT7jCsbghZei+9yxJxJqYUf91Rxj/3lPNFhZeaQN1Ru8mg0aOtjT6ZDkbZzfTr4KR7GxsaYNCgR1s7mW3teNyBs+9AnJMEfSJSMZJ3voF9w2NoSuEe8wiBS+4ATY6WxIWLKcWWIy5qAxG2l9ayYl85Jz0huqYlM6J7Otlt7Qzo4KRnhp129rOPrzbJEXyTkKBPMMbqEpyrZpBU9jGhruNwX/E4sZSuepclWpkafxhvKApAKBpjbUklLn+Eg1U+ymoD7C/31r/20k4p/HJsD67tm9nqz0dvrSToE0U0jHXbi9g3PY0yJVN71dME+9wq4wvEd3iCEVz+8GmfW7W/giVbSk+Z0PgVowZt7WY6piTzwPd6MaBjCu0dFtJs8vcevUnQJwBT+U4cK6eTVLGTYM/rcI+Zj7Jn6l2WaAJKKfad9LK1tIYT7uA5XgvHagP1Z7Z8pcITPCW4vcEo3/2z6NdGdEvnhgHt6ZpurZ/SmJPpoE+mXEzXUknQx7NIAPumZ7Bu/QMquQ01175YN4RMtDoxpThQ4WXrgUoqPCFKa/wcqPSxs8xdH9wmg4bJcPbf0GxmI13TrKf8ItfOYeGyrHSMX25r/nIMgPE075XpsDC0a6oswbQyEvRxylS2qW4ImasEf9//wDvqIRlC1oIddfkprQmw/kAV4ahiZ1ktByp99c+HY4po7OvjbJNBo63dTF63NHI7pZDXLZ2ebW0SwOK0JOjjjBbyYN/4OMmfvk7M2RnXxDcJZ43TuyzxDS5/mPd2neBwtY+DlT7KaoMc/3LZxaiB1Wwkw25h4sAOp8xf6dHeSftkE53TkumYknzaI24hTkeCPo4kHf4I56pCDJ5j+HN/gnd4IZjtepclvuQJRnih6CB/+/Q4gUgMh8VIx5Rkeraz88NBneiSbuWyrmk4k0//zzItzYbL5Tvtc0KcTaOCPhwOM2vWLEpLSwmFQkyZMoVevXoxc+ZMNE2jd+/ezJ07F4PBwLJly1i6dCkmk4kpU6aQny8jbpucvxrnh4Uk7/0LkfReuG55h0jHy/SuKuF5ghH+364THKz04QtH2XCgCncwQn7vdkwa0plLO6fqXaJIEI0K+nfffZe0tDQWLlxIdXU1N998M3379mXq1KkMHz6cOXPmsGLFCgYNGsTixYtZvnw5wWCQgoICRo0ahdl89oskxPkzf/4epnWzMfmq8A69B9+we2QImY5iSvHXT4+ztqSSDQericYUdrMRi8lAvw5O7hrVnQEdnHqXKRJMo4L+2muvZfz48fVfG41Gdu3aRV5eHgBjx46lqKgIg8HA4MGDMZvNmM1msrKy2LNnD7m5uWd9f6NRIy3N1pjSMBoNjd62VXEfx/jB/Rj2vofqkEvk9r9g7nAJifK/0Jb2OVd6gry+4RB/236MYzUBMhwWJl7SkduGdWFYt/Qm+SNpS+v5YpCem0ajgt5ur1v39Xg83HPPPUydOpUFCxbU/zDb7Xbcbjcejwen03nKdh6P55zvH42qRq9Fxv06plJY9izDUfQwWiSAZ+QDWK6Yhqs2BPHc97e0hM9ZKcXmIy5e//gIW4/WEIoqhnZN5c7LuzGh39dXgdbU+Jtkfy2h54tNem6YjIzT/7bY6D/GlpWVcffdd1NQUMDEiRNZuHBh/XNer5eUlBQcDgder/eUx78Z/KJhDLVHcH5UiPnIGkIdh+O5ciHRtGwsBhPw3SsVRfNQSnHEFWDhys/ZeLCadGsSE/q35+bcjrIsI1qkRgV9RUUFP/3pT5kzZw4jR44EoH///hQXFzN8+HDWrFnDiBEjyM3N5ZlnniEYDBIKhSgpKSEnJ6dJG0gIsSjWT1/DvnEBStNwj32UwMDJMoTsIovGFHtOuFmw4nN2n/BgTTJwz9ge3HRJxzOeKSNES9Con84XXniB2tpann/+eZ5//nkAHnzwQebPn8+iRYvIzs5m/PjxGI1GJk+eTEFBAUoppk2bhsViadIG4p2xaj/OVdNJOr6FUNYVuK9YQMzZWe+yEkZMKVbsq2DzYRdFB6o44Q7SxpbEXaO6cV3/9nRMkT98i5ZPU0qdbayFLsLhqKzRR8PYtv4B26ZnUEk2PGPmEcz5wWmHkMVNzw3QHD0rpdhaWsP20lq+qPQRjsY4VhNg9wkPRg0GdEzhqpx2XN+/PanWiz+oSz7nxNCi1uhF8zGd3IFz5X2YKncT6DURz5iHUbYMvcuKO9GYotwT5ECVjw/3lrPuiyqqfHVTG50WE+0cZpIMGrOu7s0NA9rL3Y1EqyVB35JE/Ng3PY1164vErO2omfAyoexr9a4qruw67qasJsDqkkrWllTWz1Q3GjTye7VjYEcnE/pnkmZNwiBzY0SckKBvIZKObcSxcgammgP4+92Od9RslEWunGwqn1d4eWXDYT7cVw7UzZTJ753BgI5OerS10bOtjQ6y3i7ilAS9zrSQG/uGx7DufINoShauG5cS7jpa77LiQoU3xJrPK3jrk1IOVvlJMmrceXk3Lu/Rhqx0Kw6L/PiLxCA/6ToyH1yBY/UDGDxl+C79Gd7h90NSYl0F2JQqvSG2HHFxoNLHZyfcFB9yEY0purex8vORWdw4sIMctYuEJEGvA81fhWPdPJL3vUMkPQfXD/5KpMNQvctqdbyhCGtLqthf7uHTY7VsP1bLVyPb06xJ3NC/PRMHtmdAB6fcZFokNAn6i0kpLJ+/h2PtQ2jBGrzDfl03hMwo1xY0xKEqH7P+sYdVe8uJxBQa0M5h5oeDOnFlTjt6tbOTkiz3KRXiKxL0F4nBexzH6gexHPiAcOaluG98i2i7/nqX1WocrPLxwe6TVHhD/OOzEyQZDUzol0l+73aM6J4upz4KcRYS9M1NKZJ3L8Ve9AhaNIjn8ofwX/ozMMi3/nzU+MMs+aSU1z8+Uj/yd2zPtvzmpoGYozG9yxOiVZC0aUaGmkM4V92PubSIUKcRuPMXEkvroXdZLV4gHOVvnx7njU1HOOmpG9Y2oV8mU0Z3rx85kOZMTrgrJoVoLAn65hCLYt3xKvbiBSjNhHvc4wQGFMgQsjMIRWKs3F/Bin3lrD9QRSha9xfV/h2cTBzYgRHd0hnURa4pEKKxJOibmLFyb90QshNbCXa7Cs8VjxFzdNK7rBZr69Ea5v9rH4er/TgsRq7uk0F7p4XBXVLJ65YuV6cK0QQk6JtKNITtk//GtvlZlNlB7dXPEex902mHkCWqmFLsP+klHItxuNrPuzuPs+VIDZ1SLMy/ri9X9cnAZJDvlxBNTYK+CZhObMO5ajqmyj0Een+/bgiZta3eZbUIu8pqee3jIxx1BagNhOvX3AHa2s3ckdeV/xqRhTXJqGOVQsQ3CfoLEfZj//hJrNv/SMyWSc11/0Oox9V6V6Wr0ho/qz+v5MO95XhCUQ5V+bCZjVzaKZWu6VZ+1CWVLulWzEaNwV3S5AheiItAgr6RkkrX41w5A2PtIfz9/xPv5Q+iLCl6l6Wb9Qeq+NPmo2w+7EIB3dKtZLezMya7LT8Z3lXmygihI/nX10BasBb7+kexfvYm0ZRuuL7/NuEuo/Qu66KLKcXuEx4qPEH+ubucD/eV08aWxO1DOnN1nwwGdnTW3xxbCKEvCfoGMB/8EMdHMzH4TuIbdCfevOmQZNW7rIsqphT/u6OMxZuOUloTAOpmuf/XiCx+OjwLs0lOIRWipZGgPw+avxLH2jkk7/8bkTZ9cE34I5H2g/Uu66LxhiLsPOZm6dZSdp/wUOkN0bOdjfuv6kXPdjaZLSNECydBfzZKYdn/Nxxr56CF3Hgvuxff0F+C0ax3Zc0uFInx0ecV/HtvOWtKKokpsCYZGJPdltE923Bt30xZmhGilZCgPwOD5xiO1bOwHPyQcOYg3Fc+SbRtX73LalaBcJT1B6v5x64TbD7iwhuKkmwy8P1LOpDbKYXv5WSQLKdBCtHqSNB/m4qRvGsJ9vXz0VQEz6i5+HN/CobWHXA1/jD+cPQ7j3923M2+ci/eUJRV+ys44Q5iNxsZ1aMNV+a0Y2zPtjIZUohWToL+GwyuAzg/uh9z6QZCnS/Hnf8EsdTuepd1TocqvbxdfIjIl3fdOFYToDYQAcATjHDEFcDlD5/1PQwa9G3v5N78nuRlpcnpkELEEfnXDBCLYN3+MvbihSijGXf+EwT6TWqx4ws8wQi7jrtZ90UVu8pq2XncDQoMX158ZE0ykJVuQ6MuwEdltyHDbqZzavJ3WnJaTIzt1U4uXBIijiV80Bsrd+NcOZ2kk9sJdr8az7jfEnN01LssAE66g+w4VktM1R2pByMxig5UsWp/BTEFGtC3vYP/zMti0qUdyXTKnaqEEN+VuEEfDWLb/By2T36PsqRSe83zBHtNbPKjeKUUpTUBIl+O3j2Xan+YLyq9rNxXwZajNURjp25nNmrcOLADw7ulk9sphUynhbQ0m8xmF0KcUUIGven4JzhXzcBUtZdAzi14Rs9DWds06T6iMcXakkpe2XiYPSc9Dd6+nd3MxAHtua5/e9KtX5+j3iHFIme+CCEaJLGCPuzDXrwQ6/aXiTk6UHP964S6X9Vkb+/yh/lofwWfHK1hW2kNZbVBOjgt/GJ0dzqnJp/XexgNGt3a2OiWbpWzXYQQTSJhgj7pyDqcH92PsfYw/oE/wjvyAZTZecHve7DKx7/3lFN8qJo9Jz0EIzGsSQb6d3Dy4+FZXN+/PRYZCyCE0FHcB70WrMG+fj7Wz94iktod101/Jtx55AW9p1KKv392gsWbjvJFZd3aeO8MO1fltOOGAe0Z0iUNo5zFIoRoIeI66M1ffIBj9SwM/nJ8g6fgzbsXTI0bQuYJRij6ooqdx918dtzNjmO1dE5N5ldjejCiezo5mY4mrl4IIZpGswd9LBZj3rx57N27F7PZzPz58+nWrVuz7lPzVdQNIfv8XSJt++G6/lUimZc26r384Sgr9pXz3JoDVPnCGDTolJrMrKt7c9MlHWTeixCixWv2oP/www8JhUK8/fbbbNu2jccff5w//OEPzbMzpdA+XUabD2aihX14h8/AN/gXYDy/yYrRmKq/urSsJsBrHx/m75+dBKBfewdzru3DkC6pcts7IUSr0uxBv2XLFsaMGQPAoEGD2LlzZ7Pt68Bf7iPj5DJ2Gfqw0PJLDu3sCju3n/f2J2qD+L4xD8Zo0LgltyMjuqczpmdbuXpUCNEqNXvQezweHI6v16+NRiORSAST6cy7Nho10tJsDd5XrONglqhOrHVeR7JmpE8Dt7+0SxrZGXYMmobZZODKPhl0a2tvcB0Xm9FoaNT3qzWTnhOD9Nw0mj3oHQ4HXq+3/utYLHbWkAeIRlWjrvTsOXoyQ9NsXN2EV4m2hitOE/HKWOk5MUjPDZORcfpTxpv9BO8hQ4awZs0aALZt20ZOTk5z71IIIcQ3NPsR/dVXX01RURG33347Sil++9vfNvcuhRBCfEOzB73BYODhhx9u7t0IIYQ4A7k2Xwgh4pwEvRBCxDkJeiGEiHMS9EIIEeck6IUQIs5pSqnzu8edEEKIVkmO6IUQIs5J0AshRJyToBdCiDgnQS+EEHFOgl4IIeKcBL0QQsQ5CXohhIhzzT698mLQ4wbkF1M4HGbWrFmUlpYSCoWYMmUKvXr1YubMmWiaRu/evZk7dy4Gg4Fly5axdOlSTCYTU6ZMIT8/X+/yG62yspJbbrmFV199FZPJFPf9Arz44ousXLmScDjMpEmTyMvLi+u+w+EwM2fOpLS0FIPBwCOPPBK3n/X27dt58sknWbx4MYcOHTrvHgOBADNmzKCyshK73c6CBQto06ZNw3au4sAHH3ygCgsLlVJKbd26Vd111106V9S0/vKXv6j58+crpZSqqqpS48aNU3feeafauHGjUkqp2bNnq3/961/q5MmT6oYbblDBYFDV1tbW/3drFAqF1C9+8Qt1zTXXqM8//zzu+1VKqY0bN6o777xTRaNR5fF41LPPPhv3ff/73/9W99xzj1JKqXXr1qlf/vKXcdnzSy+9pG644Qb1wx/+UCmlGtTjq6++qp599lmllFLvvfeeeuSRRxq8/7hYurmYNyDXw7XXXsuvf/3r+q+NRiO7du0iLy8PgLFjx7J+/Xp27NjB4MGDMZvNOJ1OsrKy2LNnj15lX5AFCxZw++23k5mZCRD3/QKsW7eOnJwc7r77bu666y6uuOKKuO+7R48eRKNRYrEYHo8Hk8kUlz1nZWXx3HPP1X/dkB6/mW9jx45lw4YNDd5/XAT9mW5AHi/sdjsOhwOPx8M999zD1KlTUUqhaVr98263G4/Hg9PpPGU7j8ejV9mN9s4779CmTZv6H24grvv9SnV1NTt37uR3v/sdv/nNb5g+fXrc922z2SgtLWXChAnMnj2byZMnx2XP48ePP+Ve2Q3p8ZuPf/XahoqLNfrG3IC8tSkrK+Puu++moKCAiRMnsnDhwvrnvF4vKSkp3/k+eL3eU35wWovly5ejaRobNmxg9+7dFBYWUlVVVf98vPX7lbS0NLKzszGbzWRnZ2OxWDh+/Hj98/HY92uvvcbo0aO57777KCsr44477iAcDtc/H489Q92d975yrh6/+fhXr23w/i68ZP3F+w3IKyoq+OlPf8qMGTO49dZbAejfvz/FxcUArFmzhmHDhpGbm8uWLVsIBoO43W5KSkpa5ffizTff5E9/+hOLFy+mX79+LFiwgLFjx8Ztv18ZOnQoa9euRSnFiRMn8Pv9jBw5Mq77TklJqQ/s1NRUIpFIXP9sf6UhPQ4ZMoTVq1fXv3bo0KEN3l9cTK/86qybffv21d+AvGfPnnqX1WTmz5/P+++/T3Z2dv1jDz74IPPnzyccDpOdnc38+fMxGo0sW7aMt99+G6UUd955J+PHj9ex8gs3efJk5s2bh8FgYPbs2XHf7xNPPEFxcTFKKaZNm0aXLl3ium+v18usWbMoLy8nHA7zox/9iIEDB8Zlz0ePHuXee+9l2bJlHDhw4Lx79Pv9FBYWUl5eTlJSEk899RQZGRkN2ndcBL0QQogzi4ulGyGEEGcmQS+EEHFOgl4IIeKcBL0QQsQ5CXohhIhzEvRCCBHnJOiFECLO/X8Se/R8FbRoXQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Initialization and settings\n",
    "# Assuming here that by cumulative u mean, only postive rewards\n",
    "import gym\n",
    "import pandas as pd\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "env = gym.make(\"FrozenLake-v0\", is_slippery=False)\n",
    "num_episodes = 1000\n",
    "max_nsteps = 30\n",
    "gamma = 0.9\n",
    "alpha = 0.6\n",
    "epsilon = 0.3\n",
    "\n",
    "num_actions = env.action_space.n\n",
    "Q = {} # hashing states to per-action reward arrays\n",
    "scorecard_cums = []\n",
    "scorecard_iterations = []\n",
    "# Loop for each episode\n",
    "total_reward = 0\n",
    "for ith_episode in range(num_episodes):\n",
    "\n",
    "    # Reset the environment (and obtain first observation)\n",
    "    state = env.reset()\n",
    "    \n",
    "\n",
    "    # For each timestep\n",
    "    for t in range(max_nsteps):\n",
    "        # Choose action according to Q-values using epsilon-greedy policy\n",
    "        # THIS SHOULD USE YOUR IMPLEMENTATION\n",
    "        action = choose_action(Q, state, epsilon, range(num_actions))\n",
    "        # Execute action: get reward, move env to next state\n",
    "        next_state, reward, done, info = env.step(action)\n",
    "        # Add negative reward for falling in hole\n",
    "        if done and reward == 0: reward = -1\n",
    "\n",
    "        # Some useful printing to verify progression - (W)in or (l)ose\n",
    "        if reward == -1: print('l', end='', flush=True)\n",
    "        if reward == 1: print('W', end='', flush=True)\n",
    "\n",
    "        # Accumulate reward. Note this environment only rewards on termination though.\n",
    "        if reward>0:\n",
    "            total_reward += reward\n",
    "        \n",
    "        # Update Q function for current state and action\n",
    "        # THIS SHOULD USE YOUR IMPLEMENTATION\n",
    "        update_Q(Q, state, action, next_state, reward, num_actions, alpha, gamma)\n",
    "        \n",
    "        # Update internal state\n",
    "        state = next_state\n",
    "        \n",
    "        # Terminate if episode ended\n",
    "        if done: \n",
    "            break\n",
    "    scorecard_cums.append(total_reward)\n",
    "    scorecard_iterations.append(ith_episode)\n",
    "temp = {\"Cumulative\":scorecard_cums, \"epoch\": scorecard_iterations}    \n",
    "df = pd.DataFrame(temp)\n",
    "#sns.lineplot(data=df, x=\"Cumulative\")\n",
    "sns.set_style(\"darkgrid\")\n",
    "plt.plot(scorecard_cums)\n",
    "plt.plot(scorecard_iterations)\n",
    "#print(scorecard_cums,scorecard_iterations )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. DQN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- DQN basically implements Q-Learning but uses a (deep) neural network to learn the rewards (with a few tricks). The main advantage versus the dictionary-based approach above is *generalization*: the network can output an expected reward for **all actions**, even those that were not yet explored.\n",
    "- If this code takes too long to execute, you may want to use [TPUs with Colab](https://www.tensorflow.org/guide/tpu)\n",
    "  - Though if the `env.render()` does not work in that case, just comment it out\n",
    "  - You can use the `render` option to save a video instead, find it in your Google Drive\n",
    "- This code was originally taken from [this DQN tutorial](https://towardsdatascience.com/reinforcement-learning-w-keras-openai-dqns-1eed3a5338c) (with few edits). Study it to answer the following questions.\n",
    "- Learn more about the CartPole [here](https://github.com/openai/gym/blob/master/gym/envs/classic_control/cartpole.py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-01-15 09:56:40.944446: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    }
   ],
   "source": [
    "## IMPLEMENTATION ##\n",
    "\n",
    "import gym\n",
    "import numpy as np\n",
    "import random\n",
    "\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "\n",
    "# Do you know what a deque is? https://en.wikipedia.org/wiki/Deque\n",
    "# Though here it's used just to cap its number of elements\n",
    "from collections import deque\n",
    "\n",
    "class DQN:\n",
    "    def __init__(self, env):\n",
    "        self.env     = env\n",
    "        self.memory  = deque(maxlen=200) # after 2000 will delete oldest record(s)\n",
    "\n",
    "        self.gamma = 0.85\n",
    "        self.epsilon = 1.0\n",
    "        self.epsilon_min = 0.01\n",
    "        self.epsilon_decay = 0.995\n",
    "        self.learning_rate = 0.01\n",
    "        self.tau = .125\n",
    "        self.batch_size = 64 # try different batch sizes\n",
    "        \n",
    "        self.model        = self.create_model()\n",
    "        self.target_model = self.create_model()\n",
    "\n",
    "    def create_model(self):\n",
    "        model = Sequential()\n",
    "        state_shape  = self.env.observation_space.shape\n",
    "        model.add(Dense(1,input_dim=state_shape[0], activation=\"relu\"))\n",
    "        model.add(Dense(2, activation=\"relu\"))\n",
    "        model.add(Dense(2, activation=\"relu\"))\n",
    "        model.add(Dense(self.env.action_space.n))\n",
    "        model.compile(loss=\"mean_squared_error\",\n",
    "            optimizer=Adam(learning_rate=self.learning_rate))\n",
    "        return model\n",
    "\n",
    "    def act(self, state):\n",
    "        self.epsilon *= self.epsilon_decay\n",
    "        self.epsilon = max(self.epsilon_min, self.epsilon)\n",
    "        if np.random.random() > self.epsilon:\n",
    "            return self.env.action_space.sample()\n",
    "        return np.argmax(self.model.predict(state)[0])\n",
    "\n",
    "    def remember(self, state, action, reward, new_state, done):\n",
    "        self.memory.append([state, action, reward, new_state, done])\n",
    "\n",
    "    def replay(self):\n",
    "        # This trick records past experience and re-plays for accelerate the training\n",
    "        if len(self.memory) < self.batch_size: return\n",
    "        samples = random.sample(self.memory, self.batch_size)\n",
    "        for sample in samples:\n",
    "            state, action, reward, new_state, done = sample\n",
    "            target = self.target_model.predict(state)\n",
    "            if done:\n",
    "                target[0][action] = reward\n",
    "            else:\n",
    "                Q_future = max(self.target_model.predict(new_state)[0])\n",
    "                target[0][action] = reward + Q_future * self.gamma\n",
    "            self.model.fit(state, target, epochs=1, verbose=0)\n",
    "\n",
    "    # This part can be confusing: can you explain what is happening line by line?\n",
    "    def target_train(self):\n",
    "        weights = self.model.get_weights()\n",
    "        target_weights = self.target_model.get_weights()\n",
    "        for i in range(len(target_weights)):\n",
    "            target_weights[i] = weights[i] * self.tau + target_weights[i] * (1 - self.tau)\n",
    "        self.target_model.set_weights(target_weights)        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trial 1:\n",
      "1/1 [==============================] - 0s 69ms/step\n",
      "0"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-01-15 09:56:47.431187: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "0 Reward: -92.0\n",
      "Trial 2:\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "0 Reward: -92.0\n",
      "Trial 3:\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "0 Reward: -91.0\n",
      "Trial 4:\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "0 Reward: -91.0\n",
      "Trial 5:\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "0 Reward: -92.0\n",
      "Trial 6:\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "00 Reward: 10.0\n",
      "Trial 7:\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 46ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 19ms/step\n",
      "1/1 [==============================] - 0s 23ms/step\n",
      "1/1 [==============================] - 0s 24ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 18ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      " Reward: 10.0\n",
      "Trial 8:\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 20ms/step\n",
      "1/1 [==============================] - 0s 22ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 22ms/step\n",
      "1/1 [==============================] - 0s 18ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 19ms/step\n",
      "1/1 [==============================] - 0s 19ms/step\n",
      "1/1 [==============================] - 0s 29ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 21ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 19ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 20ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 18ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 18ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 25ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 22ms/step\n",
      "1/1 [==============================] - 0s 22ms/step\n",
      "1/1 [==============================] - 0s 18ms/step\n",
      "1/1 [==============================] - 0s 19ms/step\n",
      "1/1 [==============================] - 0s 19ms/step\n",
      "1/1 [==============================] - 0s 18ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 18ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 19ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 18ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 18ms/step\n",
      "1/1 [==============================] - 0s 19ms/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 21ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 25ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 18ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 18ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 19ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      " Reward: 10.0\n",
      "Trial 9:\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 19ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 19ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 21ms/step\n",
      "1/1 [==============================] - 0s 19ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 19ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      " Reward: 10.0\n",
      "Trial 10:\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 11ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 17ms/step\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 16ms/step\n",
      "1/1 [==============================] - 0s 15ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 13ms/step\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - 0s 12ms/step\n",
      "1/1 [==============================] - ETA: 0s"
     ]
    }
   ],
   "source": [
    "## MAIN ##\n",
    "\n",
    "# Your Python kernel can crash if you redefine your `env` variable without closing\n",
    "# the gym environment properly. But also crashes if you call a method on `env` if\n",
    "# it has not been defined yet. The `try except` construct in Python allows you to \n",
    "# literally try to execute some code while catching errors and exceptions if\n",
    "# something goes wrong. Search online for what a NameError is!\n",
    "try: env.close()\n",
    "except NameError: pass # `pass` literally does nothing\n",
    "from time import sleep\n",
    "env     = gym.make(\"CartPole-v1\")\n",
    "#env = gym.wrappers.Monitor(env, 'video', force = True)\n",
    "gamma   = 0.9\n",
    "# Sometimes you will find code that uses (1-epsilon) for epsilon\n",
    "# Other times you will find a _decaying_ epsilon, lowering over time\n",
    "epsilon = .5 # it was .9 originally, play with it and understand its role\n",
    "\n",
    "ntrials   = 20 # I can see the beginning of learning with 20, feel free to raise this\n",
    "max_nstep = 10 # This environment is capped at 200 anyway, but you can try shorter\n",
    "\n",
    "dqn_agent = DQN(env=env)\n",
    "for trial in range(ntrials):\n",
    "    print(f\"Trial {trial+1}:\")\n",
    "    cur_state = env.reset().reshape(1,-1) # never forget to reset the env\n",
    "    tot_reward = 0\n",
    "    for step in range(max_nstep):\n",
    "        # print('.', end='', flush=True)\n",
    "        action = dqn_agent.act(cur_state)\n",
    "        #env.render(mode = \"human\")\n",
    "        print(action, end='', flush=True)\n",
    "        sleep(0.1) \n",
    "        new_state, reward, done, _ = env.step(action)\n",
    "        if done: reward = -100 # It's much easier to learn if you punish failing\n",
    "        tot_reward += reward\n",
    "\n",
    "        new_state = new_state.reshape(1,-1)\n",
    "        dqn_agent.remember(cur_state, action, reward, new_state, done)\n",
    "\n",
    "        dqn_agent.replay()       # Internally iterates default (prediction) model\n",
    "        dqn_agent.target_train() # What does this do?\n",
    "        cur_state = new_state\n",
    "        if done: break\n",
    "    print(f\" Reward: {tot_reward}\")\n",
    "\n",
    "env.close()\n",
    "\n",
    "# Here is how to save trained Keras models\n",
    "dqn_agent.model.save(\"trained.model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "#### 3.1 **[3pt]** There are three bugs in the implementation. Fix them until it runs successfully.\n",
    "\n",
    "- One is in `## IMPLEMENTATION ##`, easy to find because it stops it from running.\n",
    "- One is just harder, messes with the policy. It's also in `## IMPLEMENTATION ##` but there's a hint in `## MAIN ##`.\n",
    "- One is a missing line in `## MAIN ##`: something there makes no sense.\n",
    "- Of course you can just look at the original implementation. But you will learn less and will not be able do it at the exam, so give it a fair try first, because it is a fair exercise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.2 **[1pt]** Compare the shape of the model used with that of an autoencoder. Give one reason as to why they do or do not look alike."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Autoencoders have hour glass shape. First we try to compress and then make it big again. here, we are simply building a NN with input,24,48 and 24. You can say its bigger in the middle."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.3 **[1pt]** How many inputs and outputs neurons does the model have? Write the code that answers the question. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "model.add(Dense(24,input_dim=state_shape[0], activation=\"relu\"))\n",
    "model.add(Dense(48, activation=\"relu\"))\n",
    "model.add(Dense(24, activation=\"relu\"))\n",
    "model.add(Dense(self.env.action_space.n))\n",
    "```\n",
    "Input is the size of the entry state, and the output is dimension for next step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.4 **[3pt]** Write Python code to (i) create a new DQN agent, (ii) load a trained model into the agent's `model`, (iii) change the agent's epsilon-related parameters to enforce a greedy policy, and (iv) runs the greedy policy rendering each frame.\n",
    "\n",
    "- Remember to instantiate the environment first, and to close it at the end, or you may have problems.\n",
    "- Loading the model is easy -- if you saved it to a file in the `main` cell.\n",
    "- To enforce the greedy policy you need to set three agent variables, to sensible values.\n",
    "- You saw a similar evaluation loop when implementing Q-Learning. Just double check the differences with the DQN implementation to make sure it runs. You need to use `env.render()` here (if you're local; if you work on Colab just write a comment about it or save a video).\n",
    "- You can use `time.sleep()` to slow down the loop if the rendering is too fast. Remember also that the env's rendering window will close when you close the environment, so a pause there can also help."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-01-15 09:54:59.156833: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2023-01-15 09:55:01.839 python[23067:704418] Warning: Expected min height of view: (<NSPopoverTouchBarItemButton: 0x7fcaae521a20>) to be less than or equal to 30 but got a height of 32.000000. This error will be logged once per view in violation.\n",
      "2023-01-15 09:55:01.839 python[23067:704418] Warning: Expected min height of view: (<NSButton: 0x7fcab141ee40>) to be less than or equal to 30 but got a height of 32.000000. This error will be logged once per view in violation.\n",
      "2023-01-15 09:55:01.841 python[23067:704418] Warning: Expected min height of view: (<NSPopoverTouchBarItemButton: 0x7fcab141f2e0>) to be less than or equal to 30 but got a height of 32.000000. This error will be logged once per view in violation.\n",
      "2023-01-15 09:55:01.842 python[23067:704418] Warning: Expected min height of view: (<NSPopoverTouchBarItemButton: 0x7fcaabc362c0>) to be less than or equal to 30 but got a height of 32.000000. This error will be logged once per view in violation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1010001100100010000Next game\n",
      "1/1 [==============================] - 0s 196ms/step\n",
      "10000111100011001011Next game\n",
      "0001111011101001011011000001101010100000000110001011Next game\n",
      "0000001111Next game\n",
      "01000111110111111000Next game\n",
      "10000001010Next game\n",
      "11110110111Next game\n",
      "00000101100Next game\n",
      "1110000001101101111100111001000Next game\n",
      "0010101110111110111Next game\n",
      "0101100001110101000100111001Next game\n",
      "0010011011010101000001Next game\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "1Next game\n",
      "101110101110011Next game\n",
      "10001011000100001000Next game\n",
      "01101110100000010111000000011Next game\n",
      "0000111000010111Next game\n",
      "010110001111100000001111101001111Next game\n",
      "1/1 [==============================] - 0s 14ms/step\n",
      "100101100110111101010111Next game\n",
      "1110111101Next game\n",
      "Closing now\n"
     ]
    }
   ],
   "source": [
    "try: env.close()\n",
    "except NameError: pass\n",
    "from tensorflow.keras.models import load_model\n",
    "from time import sleep\n",
    "env     = gym.make(\"CartPole-v1\")\n",
    "dqn_agent2 = DQN(env=env)\n",
    "my_model = load_model(\"trained.model\")\n",
    "dqn_agent2.model = my_model\n",
    "dqn_agent2.target_model = my_model\n",
    "# step 3\n",
    "\n",
    "dqn_agent2.epsilon = 0.2\n",
    "dqn_agent2.epsilon_min = 0.01\n",
    "dqn_agent2.epsilon_decay = 0.5\n",
    "dqn_agent2.gamma = 0.85\n",
    "dqn_agent2.tau = 1\n",
    "ntrials   = 20\n",
    "max_nstep = 400\n",
    "\n",
    "for trial in range(ntrials):\n",
    "    state = env.reset().reshape(1,-1)\n",
    "    \n",
    "    for step in range(max_nstep):\n",
    "        env.render(mode = \"human\")\n",
    "        sleep(0.1) \n",
    "        action = dqn_agent2.act(state)\n",
    "        print(action, end='', flush=True)\n",
    "        new_state, reward, done, _ = env.step(action)\n",
    "        state = new_state.reshape(1,-1)\n",
    "        if done: \n",
    "            break\n",
    "    print(\"Next game\")\n",
    "            \n",
    "print(\"Closing now\")\n",
    "sleep(0.5)\n",
    "#env.render(close=True)\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# At the end of the exercise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bonus question with no points! Answering this will have no influence on your scoring, not at the assignment and not towards the exam score -- really feel free to ignore it with no consequence. But solving it will reward you with skills that will make the next lectures easier, give you real applications, and will be good practice towards the exam.\n",
    "\n",
    "The solution for this questions will not be included in the regular lab solutions pdf, but you are welcome to open a discussion on the Moodle: we will support your addressing it, and you may meet other students that choose to solve this, and find a teammate for the next assignment that is willing to do things for fun and not only for score :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import time\n",
    "import flappy_bird_gym\n",
    "try: env.close()\n",
    "except NameError: pass\n",
    "env = flappy_bird_gym.make(\"FlappyBird-v0\")\n",
    "dqn_agent3 = DQN(env=env)\n",
    "ntrials = 10\n",
    "\n",
    "obs = env.reset()\n",
    "for trial in range(ntrials):\n",
    "    print(f\"Trial {trial+1}:\")\n",
    "    count = 10\n",
    "    cur_state = env.reset().reshape(1,-1) # never forget to reset the env\n",
    "    tot_reward = 0\n",
    "    while count>=0:\n",
    "        # Next action:\n",
    "        # (feed the observation to your agent here)\n",
    "        #action = ...  # env.action_space.sample() for a random action\n",
    "        action = dqn_agent3.act(cur_state)\n",
    "        #env.render(mode = \"human\")\n",
    "        #print(action, end='', flush=True)\n",
    "        #sleep(0.1) \n",
    "        #time.sleep(1 / 30)\n",
    "        new_state, reward, done, _ = env.step(action)\n",
    "        if done: reward = -100 # It's much easier to learn if you punish failing\n",
    "        tot_reward += reward\n",
    "\n",
    "        new_state = new_state.reshape(1,-1)\n",
    "        dqn_agent3.remember(cur_state, action, reward, new_state, done)\n",
    "        count -=1\n",
    "        dqn_agent3.replay()       # Internally iterates default (prediction) model\n",
    "        dqn_agent3.target_train() # What does this do?\n",
    "        cur_state = new_state\n",
    "        if done: break\n",
    "        #print(f\" Reward: {tot_reward}\")\n",
    "        # Processing:\n",
    "        #obs, reward, done, info = env.step(action)\n",
    "\n",
    "        # Rendering the game:\n",
    "        # (remove this two lines during training)\n",
    "        #env.render()\n",
    "        #time.sleep(1 / 30)  # FPS\n",
    "\n",
    "        # Checking if the player is still alive\n",
    "        #if done:\n",
    "            #break\n",
    "env.close()\n",
    "dqn_agent3.model.save(\"floppy.model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### BONUS **[ZERO pt]** Solve another of the [OpenAI Gym environments](https://gym.openai.com/envs/#classic_control) by copying and modifying the above DQN implementation. This is very useful to learn to use different environments, and to see first-hand the limits of DRL. Can you imagine why the Mountain Car is hard?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try: env.close()\n",
    "except NameError: pass\n",
    "import flappy_bird_gym\n",
    "from tensorflow.keras.models import load_model\n",
    "from time import sleep\n",
    "env     = flappy_bird_gym.make(\"FlappyBird-v0\")\n",
    "dqn_agent2 = DQN(env=env)\n",
    "my_model = load_model(\"floppy.model\")\n",
    "dqn_agent2.model = my_model\n",
    "dqn_agent2.target_model = my_model\n",
    "# step 3\n",
    "\n",
    "dqn_agent2.epsilon = 0.2\n",
    "dqn_agent2.epsilon_min = 0.01\n",
    "dqn_agent2.epsilon_decay = 0.5\n",
    "dqn_agent2.gamma = 0.85\n",
    "dqn_agent2.tau = 1\n",
    "ntrials   = 20\n",
    "max_nstep = 400\n",
    "\n",
    "for trial in range(ntrials):\n",
    "    state = env.reset().reshape(1,-1)\n",
    "    \n",
    "    for step in range(max_nstep):\n",
    "        env.render(mode = \"human\")\n",
    "        sleep(0.1) \n",
    "        action = dqn_agent2.act(state)\n",
    "        print(action, end='', flush=True)\n",
    "        new_state, reward, done, _ = env.step(action)\n",
    "        state = new_state.reshape(1,-1)\n",
    "        if done: \n",
    "            break\n",
    "    print(\"Next game\")\n",
    "            \n",
    "print(\"Closing now\")\n",
    "sleep(0.5)\n",
    "#env.render(close=True)\n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### BONUS **[ZERO pt]** Augment the DQN implementation to track episode size and cumulative reward and plot them with lineplots."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Final considerations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Reinforcement Learning is both the name of the learning paradigm and of the framework classically used to address such problems.\n",
    "- The amount of research currently dedicated to solve RL problems by improving both DL and the RL framework is staggering. Results still keep coming, but at a much slower pace than in purer SL applications (in front of orders of magnitude more investment).\n",
    "- Next week we will explore the limitations of the RL framework, and beyond."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}