Skip to content

PKU-Alignment/omnisafe

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Organization PyPI tests Documentation Status Downloads GitHub Repo Stars codestyle License CodeCov Open In Colab

Documentation | Implemented Algorithms | Installation | Getting Started | License


OmniSafe is an infrastructural framework designed to accelerate safe reinforcement learning (RL) research. It provides a comprehensive and reliable benchmark for safe RL algorithms, and also an out-of-box modular toolkit for researchers. SafeRL intends to develop algorithms that minimize the risk of unintended harm or unsafe behavior.

OmniSafe stands as the inaugural unified learning framework in the realm of safe reinforcement learning, aiming to foster the Growth of SafeRL Learning Community. The key features of OmniSafe:

  • Highly Modular Framework. OmniSafe presents a highly modular framework, incorporating an extensive collection of tens of algorithms tailored for safe reinforcement learning across diverse domains. This framework is versatile due to its abstraction of various algorithm types and well-designed API, using the Adapter and Wrapper design components to bridge gaps and enable seamless interactions between different components. This design allows for easy extension and customization, making it a powerful tool for developers working with different types of algorithms.

  • High-performance parallel computing acceleration. By harnessing the capabilities of torch.distributed, OmniSafe accelerates the learning process of algorithms with process parallelism. This enables OmniSafe not only to support environment-level asynchronous parallelism but also incorporates agent asynchronous learning. This methodology bolsters training stability and expedites the training process via the deployment of a parallel exploration mechanism. The integration of agent asynchronous learning in OmniSafe underscores its commitment to providing a versatile and robust platform for advancing SafeRL research.

  • Out-of-box toolkits. OmniSafe offers customizable toolkits for tasks like training, benchmarking, analyzing, and rendering. Tutorials and user-friendly APIs make it easy for beginners and average users, while advanced researchers can enhance their efficiency without complex code.

Train video

If you find OmniSafe useful or use OmniSafe in your research, please cite it in your publications.

@article{JMLR:v25:23-0681,
  author  = {Jiaming Ji and Jiayi Zhou and Borong Zhang and Juntao Dai and Xuehai Pan and Ruiyang Sun and Weidong Huang and Yiran Geng and Mickel Liu and Yaodong Yang},
  title   = {OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Research},
  journal = {Journal of Machine Learning Research},
  year    = {2024},
  volume  = {25},
  number  = {285},
  pages   = {1--6},
  url     = {http://jmlr.org/papers/v25/23-0681.html}
}

Table of Contents


Quick Start

Installation

Prerequisites

OmniSafe requires Python 3.8+ and PyTorch 1.10+.

We support and test for Python 3.8, 3.9, 3.10 on Linux. Meanwhile, we also support M1 and M2 versions of macOS. We will accept PRs related to Windows, but do not officially support it.

Install from source

# Clone the repo
git clone https://github.com/PKU-Alignment/omnisafe.git
cd omnisafe

# Create a conda environment
conda env create --file conda-recipe.yaml
conda activate omnisafe

# Install omnisafe
pip install -e .

Install from PyPI

OmniSafe is hosted in PyPI / Status.

pip install omnisafe

Implemented Algorithms

Latest SafeRL Papers
List of Algorithms On Policy SafeRL Off Policy SafeRL Model-Based SafeRL Offline SafeRL Others

Examples

cd examples
python train_policy.py --algo PPOLag --env-id SafetyPointGoal1-v0 --parallel 1 --total-steps 10000000 --device cpu --vector-env-nums 1 --torch-threads 1

Algorithms Registry

Domains Types Algorithms Registry
On Policy Primal Dual TRPOLag; PPOLag; PDO; RCPO
TRPOPID; CPPOPID
Convex Optimization CPO; PCPO; FOCOPS; CUP
Penalty Function IPO; P3O
Primal OnCRPO
Off Policy Primal-Dual DDPGLag; TD3Lag; SACLag
DDPGPID; TD3PID; SACPID
Model-based Online Plan SafeLOOP; CCEPETS; RCEPETS
Pessimistic Estimate CAPPETS
Offline Q-Learning Based BCQLag; C-CRR
DICE Based COptDICE
Other Formulation MDP ET-MDP PPOEarlyTerminated; TRPOEarlyTerminated
SauteRL PPOSaute; TRPOSaute
SimmerRL PPOSimmerPID; TRPOSimmerPID

Supported Environments

Here is a list of environments that Safety-Gymnasium supports:

Category Task Agent Example
Safe Navigation Goal[012] Point, Car, Racecar, Ant SafetyPointGoal1-v0
Button[012]
Push[012]
Circle[012]
Safe Velocity Velocity HalfCheetah, Hopper, Swimmer, Walker2d, Ant, Humanoid SafetyHumanoidVelocity-v1
Safe Isaac Gym OverSafeFinger ShadowHand ShadowHandOverSafeFinger
OverSafeJoint
CatchOver2UnderarmSafeFinger
CatchOver2UnderarmSafeJoint

For more information about environments, please refer to Safety-Gymnasium.

Customizing your environment

We offer a flexible customized environment interface that allows users to achieve the following without modifying the OmniSafe source code:

  • Use OmniSafe to train algorithms on customized environments.
  • Create the the environment with specified personalized parameters.
  • Complete the recording of environment-specific information in Logger.

We provide step-by-step tutorials on Environment Customization From Scratch and Environment Customization From Community to give you a detailed introduction on how to use this extraordinary feature of OmniSafe.

Note: If you find trouble customizing your environment, please feel free to open an issue or discussion. Pull requests are also welcomed if you're willing to contribute the implementation of your environments interface.

Try with CLI

pip install omnisafe

omnisafe --help  # Ask for help

omnisafe benchmark --help  # The benchmark also can be replaced with 'eval', 'train', 'train-config'

# Quick benchmarking for your research, just specify:
# 1. exp_name
# 2. num_pool(how much processes are concurrent)
# 3. path of the config file (refer to omnisafe/examples/benchmarks for format)

# Here we provide an exampe in ./tests/saved_source.
# And you can set your benchmark_config.yaml by following it
omnisafe benchmark test_benchmark 2 ./tests/saved_source/benchmark_config.yaml

# Quick evaluating and rendering your trained policy, just specify:
# 1. path of algorithm which you trained
omnisafe eval ./tests/saved_source/PPO-{SafetyPointGoal1-v0} --num-episode 1

# Quick training some algorithms to validate your thoughts
# Note: use `key1:key2`, your can select key of hyperparameters which are recursively contained, and use `--custom-cfgs`, you can add custom cfgs via CLI
omnisafe train --algo PPO --total-steps 2048 --vector-env-nums 1 --custom-cfgs algo_cfgs:steps_per_epoch --custom-cfgs 1024

# Quick training some algorithms via a saved config file, the format is as same as default format
omnisafe train-config ./tests/saved_source/train_config.yaml

Getting Started

Important Hints

We have provided benchmark results for various algorithms, including on-policy, off-policy, model-based, and offline approaches, along with parameter tuning analysis. Please refer to the following:

Quickstart: Colab on the Cloud

Explore OmniSafe easily and quickly through a series of Google Colab notebooks:

  • Getting Started Introduce the basic usage of OmniSafe so that users can quickly hand it.
  • CLI Command Introduce how to use the CLI tool of OmniSafe.

We take great pleasure in collaborating with our users to create tutorials in various languages. Please refer to our list of currently supported languages. If you are interested in translating the tutorial into a new language or improving an existing version, kindly submit a PR to us.


Changelog

See CHANGELOG.md.

Publications using OmniSafe

We have compiled a list of papers that use OmniSafe for algorithm implementation or experimentation. If you are willing to include your work in this list, or if you wish to have your implementation officially integrated into OmniSafe, please feel free to contact us.

Papers Publisher
Off-Policy Primal-Dual Safe Reinforcement Learning ICLR 2024
Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model ICLR 2024
Iterative Reachability Estimation for Safe Reinforcement Learning NeurIPS 2023
Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulation AAAI 2024
Learning Safety Constraints From Demonstration Using One-Class Decision Trees AAAI 2024 WorkShops

The OmniSafe Team

OmniSafe is mainly developed by the SafeRL research team directed by Prof. Yaodong Yang. Our SafeRL research team members include Borong Zhang, Jiayi Zhou, JTao Dai, Weidong Huang, Ruiyang Sun, Xuehai Pan and Jiaming Ji. If you have any questions in the process of using OmniSafe, don't hesitate to ask your questions on the GitHub issue page, we will reply to you in 2-3 working days.

License

OmniSafe is released under Apache License 2.0.