Project 1: Navigation

Overview

In this project, an intelligent agent will be trained to navigate a large, square world and collect bananas (yes, bananas).

The agent is rewarded +1 for each yellow banana and -1 for each blue banana. Thus, it is trained to maximize the amount of yellow bananas while avoiding blue bananas.

The state space has 37 dimensions and contains the agent's velocity, along with ray-based perception of objects around agent's forward direction. The agent uses this information to select the next best action. Four discrete actions are available: forward, backward, left, and right.

The task is episodic, and in order to solve the environment, the trained agent must get an average score of +13 over 100 consecutive episodes.

Prerequisites

The following instructions should be completed prior to running the code:

0. Ensure the following are already installed:

• Python 3.7 or higher
  • numpy, matplotlib, unityagents, torch
• Anaconda
  • Jupyter notebooks
  • Git

1. Create (and activate) a new environment with Python 3.6.

   • Linux or Mac:

   conda create --name drlnd python=3.6
   source activate drlnd

   • Windows:

   conda create --name drlnd python=3.6 
   activate drlnd

2. If running in Windows, ensure you have the "Build Tools for Visual Studio 2019" installed from this site. This article may also be very helpful. This was confirmed to work in Windows 10 Home.

3. To install the base Gym library, use pip install gym. Supports Python 3.7, 3.8, 3.9 and 3.10 on Linux and macOS.

4. Clone the repository (if you haven't already!), and navigate to the python/ folder. Then, install several dependencies.

   git clone https://github.com/udacity/deep-reinforcement-learning.git
   cd deep-reinforcement-learning/python
   pip install .

5. Create an IPython kernel for the drlnd environment.

   python -m ipykernel install --user --name drlnd --display-name "drlnd"

6. Before running code in a notebook, change the kernel to match the drlnd environment by using the drop-down Kernel menu.

7. Download the environment from one of the links below. You need only select the environment that matches your operating system:

   • Linux: click here
   • Mac OSX: click here
   • Windows (32-bit): click here
   • Windows (64-bit): click here

   (For Windows users) Check out this link if you need help with determining if your computer is running a 32-bit version or 64-bit version of the Windows operating system.

   (For AWS) If you'd like to train the agent on AWS (and have not enabled a virtual screen), then please use this link to obtain the environment.

8. Place the file in the DRLND GitHub repository, in the p1_navigation/ folder, and unzip (or decompress) the file.

File descriptions

• model.py - defines neural network used to train the agent
• dqn_agent.py - defines agent, with functions to act, learn and remember previous states in a replay buffer
• Deep_Q_Learning.py - sets up an agent in an environment, trains it, and displays results
• Banana_Collecting_DQL.py - sets hyperparameters and uses Deep_Q_Learning to train an agent in the Banana collecting environment
• Navigation.ipynb - Jupyter notebook to run the training and simulation
• Banana_Collecting_weights.pth - output file containing saved weights of the trained model

Instructions

The following instructions will run the environment simulation, train the agent and display results:

1. Open Anaconda and navigate to the folder containing Navigation.ipynb. Run the following commands to open the notebook:

   conda activate drlnd
   jupyter notebook Navigation.ipynb

2. The usable code starts in block 5, and all other blocks can be ignored. The hyperparameters can be altered as the user sees fit.