[Error] [carb.gym.plugin] Gym cuda error: invalid resource handle: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 6137

[GeYuhong]

/usr/bin/env /home/PJLAB/geyuhong/anaconda3/envs/policydissect/bin/python /home/PJLAB/geyuhong/.vscode/extensions/ms-python.python-2023.6.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher 39341 -- /home/PJLAB/geyuhong/policydissect/play/play_anymal.py
*** Warning: failed to preload CUDA lib
*** Warning: failed to preload PhysX libs
Importing module 'gym_37' (/home/PJLAB/geyuhong/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_37.so)
Setting GYM_USD_PLUG_INFO_PATH to /home/PJLAB/geyuhong/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
PyTorch version 1.10.0+cu113
Device count 1
/home/PJLAB/geyuhong/isaacgym/python/isaacgym/_bindings/src/gymtorch
Using /home/PJLAB/geyuhong/.cache/torch_extensions/py37_cu113 as PyTorch extensions root...
Emitting ninja build file /home/PJLAB/geyuhong/.cache/torch_extensions/py37_cu113/gymtorch/build.ninja...
Building extension module gymtorch...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module gymtorch...
pygame 2.3.0 (SDL 2.24.2, Python 3.7.16)
Hello from the pygame community. https://www.pygame.org/contribute.html
Setting seed: 1
Not connected to PVD
/buildAgent/work/99bede84aa0a52c2/source/physx/src/gpu/PxPhysXGpuModuleLoader.cpp (148) : internal error : libcuda.so!

[Warning] [carb.gym.plugin] Failed to create a PhysX CUDA Context Manager. Falling back to CPU.
Physics Engine: PhysX
Physics Device: cpu
GPU Pipeline: disabled
Backend TkAgg is interactive backend. Turning interactive mode on.
/home/PJLAB/geyuhong/anaconda3/envs/policydissect/lib/python3.7/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
[Error] [carb.gym.plugin] Gym cuda error: invalid resource handle: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 6137
[Error] [carb.gym.plugin] Must enable GPU pipeline to use state tensors
Traceback (most recent call last):
File "/home/PJLAB/geyuhong/anaconda3/envs/policydissect/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/PJLAB/geyuhong/anaconda3/envs/policydissect/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/PJLAB/geyuhong/.vscode/extensions/ms-python.python-2023.6.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/main.py", line 39, in
cli.main()
File "/home/PJLAB/geyuhong/.vscode/extensions/ms-python.python-2023.6.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
run()
File "/home/PJLAB/geyuhong/.vscode/extensions/ms-python.python-2023.6.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
runpy.run_path(target, run_name="main")
File "/home/PJLAB/geyuhong/.vscode/extensions/ms-python.python-2023.6.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 322, in run_path
pkg_name=pkg_name, script_name=fname)
File "/home/PJLAB/geyuhong/.vscode/extensions/ms-python.python-2023.6.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 136, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/home/PJLAB/geyuhong/.vscode/extensions/ms-python.python-2023.6.0/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
exec(code, run_globals)
File "/home/PJLAB/geyuhong/policydissect/play/play_anymal.py", line 46, in
play_anymal(args, activation_func=activation, map=forward_anymal, model_name="anymal_forward", parkour=False)
File "/home/PJLAB/geyuhong/policydissect/policydissect/utils/isaacgym_utils.py", line 351, in play_anymal
env, _ = task_registry.make_env(name=args.task, args=args, env_cfg=env_cfg)
File "/home/PJLAB/geyuhong/policydissect/policydissect/legged_gym/utils/task_registry.py", line 103, in make_env
headless=args.headless
File "/home/PJLAB/geyuhong/policydissect/policydissect/legged_gym/envs/anymal_c/anymal.py", line 51, in init
super().init(cfg, sim_params, physics_engine, sim_device, headless)
File "/home/PJLAB/geyuhong/policydissect/policydissect/legged_gym/envs/base/legged_robot.py", line 87, in init
self._init_buffers()
File "/home/PJLAB/geyuhong/policydissect/policydissect/legged_gym/envs/anymal_c/anymal.py", line 65, in _init_buffers
super()._init_buffers()
File "/home/PJLAB/geyuhong/policydissect/policydissect/legged_gym/envs/base/legged_robot.py", line 572, in _init_buffers
self.noise_scale_vec = self._get_noise_scale_vec(self.cfg)
File "/home/PJLAB/geyuhong/policydissect/policydissect/legged_gym/envs/anymal_c/anymal.py", line 126, in _get_noise_scale_vec
noise_vec = torch.zeros_like(self.obs_buf[0])
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

In /legged_gym/envs/base/legged_robot.py, line 488, i get '[Error] [carb.gym.plugin] Gym cuda error: invalid resource handle: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 6137', and after running the line 489, i get '[Error] [carb.gym.plugin] Must enable GPU pipeline to use state tensors'.
These two errors prevent me from creating tensors on 'cuda'. I don't know how to solve this problem.
Besides, I run the case in /isaacgym/python/examples/ and find that i will get the same error if the case has the code 'self.gym.refresh_dof_state_tensor(self.sim)' and 'self.gym.refresh_actor_root_state_tensor(self.sim)', e.g., franka_nut_bolt_ik_osc.py and franka_cube_ik_osc.py

[lizhiwen19900709]

I had the same problem