v3.0.0: Sim-to-real RL pipeline

[stephane-caron]

This release brings a functional Sim-to-real reinforcement learning pipeline (usage video) for Upkies, implemented as an update to the PPO balancer and Gym environments in upkie.envs.

ppo_balancer_michel-strogoff.mp4

Added

• PPO balancer: Convenience Makefile to manage the training directory
• docs: Agents and Environments pages
• envs: Log action, observation and reward to "env" sub-dict of action
• envs: New upkie.envs.wrappers submodule
• envs: Wrapper to act on the discrete-derivative of an action
• envs: Wrapper to add the action to the observation vector
• envs: Wrapper to low-pass filter action (modeling a delay)
• envs: Wrapper to noisify actions
• envs: Wrapper to noisify observations
• logs: Convenience Makefile for log synchronization
• spines: Label pi3hat spine log filenames with the robot's hostname

Changed

• Breaking: Rename the "wheel" balancer agent to "PID balancer"
• Breaking: envs: Rewards are now part of individual environments
• Breaking: envs: info dictionary does not repeat "action" any more
• MPC balancer: Update height of control point to 58 cm
• PPO balancer: Update height of control point to 58 cm
• dependencies: Bump Gymnasium to 0.29.1
• dependencies: Bump loop-rate-limiters to 0.6.1
• dependencies: Bump NumPy to 1.24.3
• dependencies: Bump Vulp to 2.0.0
• envs: Bump UpkieGroundVelocity to version 3
• envs: Legs now return to their neutral configuration in UpkieGroundVelocity
• envs: rate attribute from base environment is now internal
• examples: Refactor lying genuflections
• examples: Refactor wheeled balancing

Removed

• docs: Code overview page, split into Agents, Environments and Observations
• envs: Acceleration filter from UpkieGroundVelocity (=> gymnasium.Wrapper)
• envs: Intermediate UpkieWheeledPendulum environment
• envs: Low-pass filter from UpkieGroundVelocity (=> gymnasium.Wrapper)
• envs: Survival reward, as environments now ship their own rewards

---

This discussion was created from the release v3.0.0.