Skip to content

🐳 Implementation of various Distributional Reinforcement Learning Algorithms using TensorFlow2.

License

Notifications You must be signed in to change notification settings

marload/DistRL-TensorFlow2

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

35 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

TF Depend GYM Depend License Badge

Distributional RL in TensorFlow2

DistRL-TensorFlow2 is a repository that implements a variety of popular Distribution Reinforcement Learning Algorithms using TensorFlow2. Distributional RL is an algorithm suitable for stochastic environments. If you want to study the Distribution RL, this repository will be the best choice. The dist-rl-tf2 includes three Distributional RL algorithms published by DeepMind, a leading AI research institute.

Algorithms


C51

Paper A Distributional Perspective on Reinforcement Learning
Author Marc G. Bellemare, Will Dabney, Rémi Munos
Method OFF-Policy / Temporal-Diffrence / Model-Free
Action Discrete only

Core of Idea

# idea01. The output of the Q Network is a Distribution Vector, not a Scalar Value.
def create_model(self):
    input_state = Input((self.state_dim,))
    h1 = Dense(64, activation='relu')(input_state)
    h2 = Dense(64, activation='relu')(h1)
    outputs = []
    for _ in range(self.action_dim):
        outputs.append(Dense(self.atoms, activation='softmax')(h2))
    return tf.keras.Model(input_state, outputs)

Getting Start

# Discrete Action Space C51
$ python C51/C51.py

QRDQN

Paper Distributional Reinforcement Learning with Quantile Regression
Author Will Dabney, Mark Rowland, Marc G. Bellemare, Rémi Munos
Method OFF-Policy / Temporal-Diffrence / Model-Free
Action Discrete only

Core of Idea

# idea01. The output of the Q Network is Quantile Region, not the Distribution Vector.
def create_model(self):
    return tf.keras.Sequential([
        Input([self.state_dim, ]),
        Dense(64, activation='relu'),
        Dense(64, activation='relu'),
        Dense(self.action_dim * self.atoms, activation='linear'),
        Reshape([self.action_dim, self.atoms])
    ])

# idea02. Use Quantile Huber Loss instead of CategoryCrossEntropy Loss.
def quantile_huber_loss(self, target, pred, actions):
    pred = tf.reduce_sum(pred * tf.expand_dims(actions, -1), axis=1)
    pred_tile = tf.tile(tf.expand_dims(pred, axis=2), [1, 1, self.atoms])
    target_tile = tf.tile(tf.expand_dims(
        target, axis=1), [1, self.atoms, 1])
    huber_loss = self.huber_loss(target_tile, pred_tile)
    tau = tf.reshape(np.array(self.tau), [1, self.atoms])
    inv_tau = 1.0 - tau
    tau = tf.tile(tf.expand_dims(tau, axis=1), [1, self.atoms, 1])
    inv_tau = tf.tile(tf.expand_dims(inv_tau, axis=1), [1, self.atoms, 1])
    error_loss = tf.math.subtract(target_tile, pred_tile)
    loss = tf.where(tf.less(error_loss, 0.0), inv_tau *
                    huber_loss, tau * huber_loss)
    loss = tf.reduce_mean(tf.reduce_sum(
        tf.reduce_mean(loss, axis=2), axis=1))
    return loss

Getting Start

# Discrete Action Space QRDQN
$ python QR-DQN/QR-DQN.py

IQN

Paper Implicit Quantile Networks for Distributional Reinforcement Learning
Author Will Dabney, Georg Ostrovski, David Silver, Rémi Munos
Method OFF-Policy / Temporal-Diffrence / Model-Free
Action Discrete only

Core of Idea

# idea01. Use the quantile embedding network.
def call(self, state):
    x = self.feature_extraction(state)
    feature_dim = x.shape[1]
    tau = np.random.rand(self.atoms, 1)
    pi_mtx = tf.constant(np.expand_dims(
        np.pi * np.arange(0, self.quantile_dim), axis=0))
    cos_tau = tf.cos(tf.matmul(tau, pi_mtx))
    phi = self.relu(self.phi(cos_tau) +
                    tf.expand_dims(self.phi_bias, axis=0))
    phi = tf.expand_dims(phi, axis=0)
    x = tf.reshape(x, (-1, feature_dim))
    x = tf.expand_dims(x, 1)
    x = x * phi
    x = self.fc(x)
    x = self.fc_q(x)
    q = tf.transpose(x, [0, 2, 1])
    return q, tau

# idea02. Use the random sampled value instead of the specified value of the tau.
tau = np.random.rand(self.atoms, 1)

Getting Start

# Discrete Action Space IQN
$ python IQN/IQN.py

Reference

About

🐳 Implementation of various Distributional Reinforcement Learning Algorithms using TensorFlow2.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages