Using and registering RoboHive environments to the Ray API #103
Comments
|
Hi Christos, I haven't used Ray so it's a bit hard for me to pinpoint. If I have to guess, I feel like the ray worker isn't able to find RoboHive. Two things I'd suggest --
|
|
Hello dear Dr. Vikash, How are you? I hope you and everyone in your family are all well and healthy!!! I am sending you this message to inform you that I managed to super pass the above issue with the Ray API. The problem got resolved after creating the conda environment with Python 3.9 (I have given more details regarding this environment on the open issue #102). In this conda environment setup I followed your second advise (the first one I had indeed tested it already) and I inserted the import robohive command inside the env_creator() function, at the line before the gym.make() command. Then the FrankaPickPlaceFixed-v0 environment was successfully created inside the Ray worker. Unfraternally, another error was thrown later regarding the building of the mujoco-py library: NFO:root:running build_ext During handling of the above exception, another exception occurred: ray::PPO.init() (pid=27623, ip=172.21.2.236, repr=PPO) (PPO pid=27623) 2023-07-12 15:33:05,354 ERROR worker.py:763 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::PPO.init() (pid=27623, ip=172.21.2.236, repr=PPO) File "/home/cocp5/anaconda3/envs/rhRL94/lib/python3.9/site-packages/ray/rllib/evaluation/worker_set.py", line 169, in init During handling of the above exception, another exception occurred: ray::PPO.init() (pid=27623, ip=172.21.2.236, repr=PPO) I have attempted uninstalling and reinstalling mujoco-py from the cloned GitHub repository, it did not work. I uninstalled and reinstalled the libffi library on my system. It also did not work. Then I specified the path to the library in my environment by setting it to the LD_LIBRARY_PATH using the conda env config vars set command. It did not work either. I can not understand why this issue occurs. mujoco-py is being build fine when I simply import it, either on a traditional Python .py file or from a Jupyter Notebook. The RoboHive demo commands also work fine (python -m robohive.utils.examine_env -e FrankaReachRandom-v0, python -m robohive.utils.examine_env -e FrankaPickPlaceFixed-v0). Could it be an issue with multi-processing that Ray API uses? Do you have any suggestions on how I might be able to overcome this issue? Thank you very much in advance for you valuable help and support!!! Kind regards, Christos Peridis |
|
Try the |
|
Dear Dr. Vikash, Thank you very much for your immediate response! Thank you very much for your valuable help!!! Kind regards, Christos Peridis |
|
Hello dear Dr. Vikash, I hope you are doing well! I am sending you this message in order to update you regarding my progress with the dev branch of the RoboHive API. I managed to successfully clone the dev branch in my WSL2 setup. the I proceed in the creation of a new conda environment to work there with the dev branch. I followed similar steps that I followed for the creation of the conda 3.9 environment that I had provided you with yesterday. However, I faced an issue with the rendering of the environments. I managed to fix the issue by changing the setting of the MUJOCO_GL and PYOPENGL_PLATFORM environment variables from egl, thath the PyTorch website was suggesting, to osmesa. By applying the above changes I managed to run successfully the example commands: python -m robohive.utils.examine_env -e FrankaPickPlaceFixed-v0 & python -m robohive.utils.examine_env -e FrankaReachRandom-v0 I further proceed in installing the Ray API. Initially I installed version 2.2.0 as in yesterdays Python 3.9 conda environment. Ray manged successfully to build the RoboHive environment and progressed further. Unfurtunaly, dependency errors did not allow the Ray Tune trials to complete. After conducting further research I installed Ray version 1.6.0 which did not had any dependency errors while running the code. I have made a new environment.yml file for the above mentioned conda environment with the RoboHive API made from the dev branch (version 0.6.0) and Ray API version 1.6.0. I have tested the .yml file and it successfully creates the desired conda environment with no pip or other errors. The new recreated environment has also been tested with the example commands, and everything worked fine. I am attaching that .yml file in this message for your reference. rhRL3906_RoboHive_dev_Ray160_environment.zip Unfortunately however, the code did not manage to run much further because the following error occurred: (pid=296669) 2023-07-14 16:37:53,591 ERROR worker.py:428 -- Exception raised in creation task: The actor died because of an error raised in its creation task, ray::SAC.init() (pid=296669, ip=172.21.2.236) Do you have any suggestions on how I might be able to resolve it ? Thank you very much for all your valuable help and support!!! Kind regards, Christos Peridis |
|
This seems like an error coming from policy initialization from Ray and SAC side. |
|
Hello dear Dr. Vikash, I hope you are doing well! I further investigated the error and it seems that the error occurs when the agent tries to create a "dummy batch" from the view requirements of the policy so to initialize the loss. This procedure takes place in the 'initialize_loss_from_dummy_batch' function. The dummy batch creation involves generating random samples from the observation and action spaces of the environment. The purpose of this is to initialize the loss function of the policy with some initial data .The batch creation procedure takes place in the 'get_dummy_batch_from_view_requirements' which is called by the 'initialize_loss_from_dummy_batch' _ During this procedure, an attempt to generate a random sample from a uniform distribution with bounds that are too large for the numpy's uniform function to handle generates the error. I have checked the observation that the FrankaPickPlaceFixed-v0 and it returns a Box(63,) of values of type float64. Based on research I have conducted, RLlib expects float32 inputs for its algorithms. Is there any hyperparameter on the configuration of the FrankaPickPlaceFixed-v0 or any utility function on the RoboHive API that would enable to normalize the observation and convert it to a type of float32? I would also like to ask what is the physical meaning behind the Box(63,) observation returned from the environment? Is it possible to use as a returned observation from the environment an image from the simulator ? The action space, which is Box(9,) of float32 does it represent the degrees of freedom of the robot (seven for the arm robot and two for the robot gripper)? Regarding the Gym version, I am using OpenAI Gym version 0.13.0 which is the one installed with the RoboHive API installation, and the one specified in the env.yaml file in the setup folder of the RoboHive repository. The requirements.txt for the Ray API version 1.6.0, do not specify any specific version dependency for the OpenAI Gym library. Is there anything more that I should check for the gym version based on your experience ? Thank you very much for all the valuable and continues help and support!!! I am always at your disposal for any further clarification or queries that might occur. Kind regards, Christos Peridis |
|
RoboHive currently supports action normalization (can be configured during registration) but doesn't support any observation normalization, or observation precision. The rationale is that observations are usually quite heterogenous (linear, angular, pixel, etc.) and it's not obvious is a default choice will work for all observations. @vmoens: Im curious what do we do in TorchRL? Are there any configurations to expose such choices to the user? |
|
The env-API/ MDP definitions are designed to abstract out the physical meaning to promote generalization in data-driven methods. However, it's still possible to peel the layer and understand these variables.
|
|
Hello dear Dr. Vikash, Thank you very much for your immediate response!!! Your instructions and the material you provided me with are very helpful and I have already started studying and working on them. Based on them I will progress further with the integration with the Ray API. I will be keeping you updated regarding my progress. Thank you very much for your valuable and continues help and support!!! Kind regards, Christos Peridis |
|
Thank you for your effort with the Ray Integration. It's going to be a big win. Keep me in the loop as you make progress. |
Hello dear Dr. Vikash,
I hope you and everyone in your family are doing well! For conducting Reinforcement Learning experiments I have been using the Ray API and more specifically the implemented algorithms of RLlib and the Tune library for hyper parameter tuninng. In the past I had integrated and register in the Ray API the CT-graph benchmark, as a custom gym environment. You can see how I have performed this in the example code here
I have developed the following code that works on the same principals as the code developed for the integration of the CT-graph benchmark:
import robohive
import gym
import ray
ray.init()
resources = ray.cluster_resources()
print(resources)
resources = ray.cluster_resources()
print(resources)
def env_creator(env_config={}):
env = gym.make('FrankaPickPlaceFixed-v0')
env.reset()
return env
from ray.tune.registry import register_env
register_env("RoboHive_Pick_Place_0", env_creator)
sac_config = {
"env": "FrankaPickPlaceFixed-v0", # Specify your environment class here
"framework": "torch",
"num_workers": 4,
"num_gpus": 1,
"monitor": True,
# Add more SAC-specific config here
}
from ray import tune
analysis = tune.run(
"SAC",
config=sac_config,
stop={"training_iteration": 100}, # Specify stopping criteria
checkpoint_at_end=True
)
However I am getting the following error:
The above code throughs the following error:
(RolloutWorker pid=318903) ray::RolloutWorker.init() (pid=318903, ip=158.125.234.46, repr=<ray.rllib.evaluation.rollout_worker.RolloutWorker object at 0x7f878b6db6a0>)
(RolloutWorker pid=318903) File "/home/lunet/cocp5/roboHive200/lib/python3.10/site-packages/ray/rllib/env/utils.py", line 54, in gym_env_creator
(RolloutWorker pid=318903) return gym.make(env_descriptor, **env_context)
(RolloutWorker pid=318903) File "/home/lunet/cocp5/roboHive200/lib/python3.10/site-packages/gym/envs/registration.py", line 156, in make
(RolloutWorker pid=318903) return registry.make(id, **kwargs)
(RolloutWorker pid=318903) File "/home/lunet/cocp5/roboHive200/lib/python3.10/site-packages/gym/envs/registration.py", line 100, in make
(RolloutWorker pid=318903) spec = self.spec(path)
(RolloutWorker pid=318903) File "/home/lunet/cocp5/roboHive200/lib/python3.10/site-packages/gym/envs/registration.py", line 142, in spec
(RolloutWorker pid=318903) raise error.UnregisteredEnv('No registered env with id: {}'.format(id))
(RolloutWorker pid=318903) gym.error.UnregisteredEnv: No registered env with id: FrankaPickPlaceFixed-v0
Why is the system unable to detect the RoboHive registered environment? Is it something hidden in the layer complexity of the API that I have not yet understand? Which tutorials or other material do you suggest me to study in order to better understand the structure of the RoboHive API, how it works and how I could modify it ?
Thank you very much in advance for the valuable help!!!
Kind regards,
Christos Peridis
The text was updated successfully, but these errors were encountered: