test_axis.py Timeout communicating with flight control plugin #40
Comments
xabierolaz
changed the title
|
99% of the time its because you don't have the plugins built and in the correct location. You get a time out because there are no motor plugins to talk to. The thesis branch is still WIP in progress so there is still stuff missing. Take a look here for instructions https://github.com/wil3/gymfc#directory-layout So youll need to update the directory to be something like, /modules/digitaltwins/nf1/plugin/bulid/ If you are using these plugins https://github.com/wil3/gymfc-aircraft-plugins youll need to create soft links to the built IMU and motor model plugins (libgazebo_imu_plugin.so and libgazebo_motor_model.so). |
|
There's actually an open issue to make this more clear, #37 |
|
@xabierolaz What output do you get when you pass the |
|
@wil3 thanks, will have a look at #37 and directory layout instructions @SwapnilPande this is what I get passing the --verbose |
|
@xabierolaz did building and linking the aircraft-plugins resolve the error? Can this be closed? |
|
Followed all steps building and linking the aircraft plugins and following the directory structure, and getting this now when running test_axis.py: Might be something related to SetAnchor? |
|
What's the output of, Show me the build contents of the aircraft plugin repo. |
|
This is the output And the content of the airclaft plugin repo's build folder Steps I followed:1-Created these directories 2- git clone inside nf1/plugins 3- inside gymfc-aircraft-plugins 4- symlink |
|
Thanks @xabierolaz. Everything looks fine but you do have something things different than how I have it setup so could you try a couple things? I don't think it should cause problems but I've never cloned the gymfc-aircraft-plugin repo inside a digital twin. To rule out all possibilities could you move it to Inside My build directory only has these two files. |
|
Reviewing the code, this error occurs when the python environment can not communicate with the Gazebo environment over the UDP network. The flight control plugin will wait until it gets data back from the motors and IMU, if those plugins don't exist it waits indefinitely and times out. This is usually the problem as I previously mentioned. If the above does not help it may require uncommenting some debug statements in the FlightControllerPlugin.cpp plugin to see if its possibly a networking issue. I did test out test_axis.py with the thesis branch and nf1 model and was able to run it without problems. |
|
Went to a previous snapshot where gymfc was clean and tried both with @SwapnilPande method and deleting (method 1) and trying your method from scratch (method 2) To sum up: Both ways throw same output, with the setanchor error Method 1 (Steps 1 to 9) 1- Created /plugins/build directories inside nf1 folder 5- BTW, this is the method @SwapnilPande mentioned in its repo https://github.com/SwapnilPande/gymfc/#setting-up-an-aircraft-model when running this, 4 files are created there, so I'm guessing both libcontrol_msgs.so and libsensor_msgs.so shouldn't be there? 6- Deleted all 4 files
9-Proceed to run test_axis.py Method 210- As this didn't seem to work, tried with your symlink method went to a previous snapshot, this time didnt run the symlink of @SwapnilPande and run yours instead (build folder was empty and the 2 files you mentioned were created this time) Output: |
|
I'm not familiar with that window manager you are using, you mentioned using a VM, what OS are you running? I suggest uncommenting the gzdbg statements in this function, rebuilding and providing the output, https://github.com/wil3/gymfc/blob/master/gymfc/envs/assets/gazebo/plugins/FlightControllerPlugin.cpp#L593 Your problem isn't obvious, its going to take some deep diving into the source code to find out whats going on with your environment. |
|
I'm using Ubuntu 18.04.3 LTS with the xfce desktop environment, thats why it looks that way It's indeed a VM with these specifications: I will be commenting output here by tomorrow, as Im rebuilding everything taking consideration of all the feedback you gave me |
|
Re-building process: (at the end, got same setanchor error and output with those gzdbg lines uncommented) 1- went to a clean state with just gazebo and dart installed error: invalid command 'bdist_wheel
(env) cuda@eim-alu-83252:~/projects/gymfc$ pip3 install -e . error: invalid command 'bdist_wheel' Failed building wheel for gym error: invalid command 'bdist_wheel' Failed building wheel for psutil error: invalid command 'bdist_wheel' Failed building wheel for future Did some research and gets fixed with pip install wheel before doing pip3 install -e . However, this time I did pip install wheel before installing gymfc in case that could be the reason. So, steps: no 'bdist_wheel error this time, as Im doing pip install wheel before(env) cuda@eim-alu-83252:~/projects/gymfc$ pip3 install -e . Obtaining file:///home/cuda/projects/gymfc Collecting gym (from gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/77/48/c43b8a72b916cc70896aa431b0fc00d1481ae34e28dc55e2144f4c77916b/gym-0.17.1.tar.gz (1.6MB) 100% |################################| 1.6MB 723kB/s Collecting numpy (from gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/07/08/a549ba8b061005bb629b76adc000f3caaaf881028b963c2e18f811c6edc1/numpy-1.18.2-cp36-cp36m-manylinux1_x86_64.whl (20.2MB) 100% |################################| 20.2MB 58kB/s Collecting protobuf (from gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/57/02/5432412c162989260fab61fa65e0a490c1872739eb91a659896e4d554b26/protobuf-3.11.3-cp36-cp36m-manylinux1_x86_64.whl (1.3MB) 100% |################################| 1.3MB 971kB/s Collecting psutil>=5.3.0 (from gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/c4/b8/3512f0e93e0db23a71d82485ba256071ebef99b227351f0f5540f744af41/psutil-5.7.0.tar.gz (449kB) 100% |################################| 450kB 2.6MB/s Collecting cloudpickle<1.4.0,>=1.2.0 (from gym->gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/ea/0b/189cd3c19faf362ff2df5f301456c6cf8571ef6684644cfdfdbff293825c/cloudpickle-1.3.0-py2.py3-none-any.whl Collecting pyglet<=1.5.0,>=1.4.0 (from gym->gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/70/ca/20aee170afe6011e295e34b27ad7d7ccd795faba581dd3c6f7cec237f561/pyglet-1.5.0-py2.py3-none-any.whl (1.0MB) 100% |################################| 1.0MB 1.2MB/s Collecting scipy (from gym->gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/dc/29/162476fd44203116e7980cfbd9352eef9db37c49445d1fec35509022f6aa/scipy-1.4.1-cp36-cp36m-manylinux1_x86_64.whl (26.1MB) 100% |################################| 26.1MB 44kB/s Collecting six (from gym->gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/65/eb/1f97cb97bfc2390a276969c6fae16075da282f5058082d4cb10c6c5c1dba/six-1.14.0-py2.py3-none-any.whl Requirement already satisfied: setuptools in ./env/lib/python3.6/site-packages (from protobuf->gymfc==0.1.2) Collecting future (from pyglet<=1.5.0,>=1.4.0->gym->gymfc==0.1.2) Downloading https://files.pythonhosted.org/packages/45/0b/38b06fd9b92dc2b68d58b75f900e97884c45bedd2ff83203d933cf5851c9/future-0.18.2.tar.gz (829kB) 100% |################################| 829kB 1.5MB/s Building wheels for collected packages: gym, psutil, future Running setup.py bdist_wheel for gym ... done Stored in directory: /home/cuda/.cache/pip/wheels/c0/84/61/523b92d88787ae29689b3cc08cf445d8d8186d7fbe1acbf87b Running setup.py bdist_wheel for psutil ... done Stored in directory: /home/cuda/.cache/pip/wheels/d7/69/b4/3200b95828d1f0ddb3cb5699083717f4fdbd9b4223d0644c57 Running setup.py bdist_wheel for future ... done Stored in directory: /home/cuda/.cache/pip/wheels/8b/99/a0/81daf51dcd359a9377b110a8a886b3895921802d2fc1b2397e Successfully built gym psutil future Installing collected packages: cloudpickle, numpy, future, pyglet, scipy, six, gym, protobuf, psutil, gymfc Running setup.py develop for gymfc Successfully installed cloudpickle-1.3.0 future-0.18.2 gym-0.17.1 gymfc numpy-1.18.2 protobuf-3.11.3 psutil-5.7.0 pyglet-1.5.0 scipy-1.4.1 six-1.14.0cd ~/projects/gymfc/gymfc/envs/assets/gazebo/plugins In this case, I will build build_plugin.sh without modifying the FlightController.cpp changed gymfc.ini to be pointing to new gazebo as usual cmake ../ (no errors)-- The C compiler identification is GNU 7.5.0 -- The CXX compiler identification is GNU 7.5.0 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Looking for pthread.h -- Looking for pthread.h - found -- Looking for pthread_create -- Looking for pthread_create - not found -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found -- Found Threads: TRUE -- Boost version: 1.65.1 -- Found the following Boost libraries: -- system -- thread -- timer -- chrono -- date_time -- atomic -- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1") -- Checking for module 'bullet>=2.82' -- Found bullet, version 2.87 -- Found Simbody: /usr/include/simbody -- Found CCD: /usr/include (found version "2.0") -- Found FCL: /usr/include (found version "0.5.0") -- Found ASSIMP: /usr/include (found version "4.1.0") -- Found DART: /usr/local/include (Required is at least version "6.6") found components: dart -- Boost version: 1.65.1 -- Found the following Boost libraries: -- thread -- system -- filesystem -- program_options -- regex -- iostreams -- date_time -- chrono -- atomic -- Found Protobuf: /usr/lib/x86_64-linux-gnu/libprotobuf.so;-lpthread (found version "3.0.0") -- Boost version: 1.65.1 -- Looking for OGRE... -- OGRE_PREFIX_WATCH changed. -- Checking for module 'OGRE' -- Found OGRE, version 1.9.0 -- Found Ogre Ghadamon (1.9.0) -- Found OGRE: optimized;/usr/lib/x86_64-linux-gnu/libOgreMain.so;debug;/usr/lib/x86_64-linux-gnu/libOgreMain.so -- Looking for OGRE_Paging... -- Found OGRE_Paging: optimized;/usr/lib/x86_64-linux-gnu/libOgrePaging.so;debug;/usr/lib/x86_64-linux-gnu/libOgrePaging.so -- Looking for OGRE_Terrain... -- Found OGRE_Terrain: optimized;/usr/lib/x86_64-linux-gnu/libOgreTerrain.so;debug;/usr/lib/x86_64-linux-gnu/libOgreTerrain.so -- Looking for OGRE_Property... -- Found OGRE_Property: optimized;/usr/lib/x86_64-linux-gnu/libOgreProperty.so;debug;/usr/lib/x86_64-linux-gnu/libOgreProperty.so -- Looking for OGRE_RTShaderSystem... -- Found OGRE_RTShaderSystem: optimized;/usr/lib/x86_64-linux-gnu/libOgreRTShaderSystem.so;debug;/usr/lib/x86_64-linux-gnu/libOgreRTShaderSystem.so -- Looking for OGRE_Volume... -- Found OGRE_Volume: optimized;/usr/lib/x86_64-linux-gnu/libOgreVolume.so;debug;/usr/lib/x86_64-linux-gnu/libOgreVolume.so -- Looking for OGRE_Overlay... -- Found OGRE_Overlay: optimized;/usr/lib/x86_64-linux-gnu/libOgreOverlay.so;debug;/usr/lib/x86_64-linux-gnu/libOgreOverlay.so -- Found Protobuf: /usr/lib/x86_64-linux-gnu/libprotobuf.so;-lpthread;-lpthread (found suitable version "3.0.0", minimum required is "2.3.0") -- Config-file not installed for ZeroMQ -- checking for pkg-config -- Checking for module 'libzmq >= 4' -- Found libzmq , version 4.2.5 -- Found ZeroMQ: TRUE (Required is at least version "4") -- Checking for module 'uuid' -- Found uuid, version 2.31.1 -- Found UUID: TRUE -- Checking for module 'tinyxml2' -- Found tinyxml2, version 6.0.0 -- Looking for dlfcn.h - found -- Looking for libdl - found -- Found DL: TRUE -- FreeImage.pc not found, we will search for FreeImage_INCLUDE_DIRS and FreeImage_LIBRARIES -- Checking for module 'gts' -- Found gts, version 0.7.6 -- Found GTS: TRUE -- Checking for module 'libswscale' -- Found libswscale, version 4.8.100 -- Found SWSCALE: TRUE -- Checking for module 'libavdevice >= 56.4.100' -- Found libavdevice , version 57.10.100 -- Found AVDEVICE: TRUE (Required is at least version "56.4.100") -- Checking for module 'libavformat' -- Found libavformat, version 57.83.100 -- Found AVFORMAT: TRUE -- Checking for module 'libavcodec' -- Found libavcodec, version 57.107.100 -- Found AVCODEC: TRUE -- Checking for module 'libavutil' -- Found libavutil, version 55.78.100 -- Found AVUTIL: TRUE -- Found CURL: /usr/lib/x86_64-linux-gnu/libcurl.so (found version "7.58.0") -- Checking for module 'jsoncpp' -- Found jsoncpp, version 1.7.4 -- Found JSONCPP: TRUE -- Checking for module 'yaml-0.1' -- Found yaml-0.1, version 0.1.7 -- Found YAML: TRUE -- Checking for module 'libzip' -- Found libzip, version 1.1.2 -- Found ZIP: TRUE -- Found Protobuf: /usr/lib/x86_64-linux-gnu/libprotobuf.so;-lpthread;-lpthread;-lpthread (found version "3.0.0") -- Checking for module 'protobuf' -- Found protobuf, version 3.0.0 Gazebo version: 10.1 -- Configuring done -- Generating done -- Build files have been written to: /home/cuda/projects/gymfc-aircraft-plugins/build make (no errors)cuda@eim-alu-83252:~/projects/gymfc-aircraft-plugins/build$ make [ 5%] Running C++ protocol buffer compiler on /home/cuda/local/include/gazebo-10/gazebo/msgs/proto/quaternion.proto [ 10%] Running C++ protocol buffer compiler on msgs/Float.proto [ 15%] Running C++ protocol buffer compiler on msgs/Imu.proto [ 20%] Running C++ protocol buffer compiler on msgs/MotorSpeed.proto [ 25%] Running C++ protocol buffer compiler on msgs/EscSensor.proto [ 30%] Running C++ protocol buffer compiler on /home/cuda/local/include/gazebo-10/gazebo/msgs/proto/vector3d.proto Scanning dependencies of target sensor_msgs [ 35%] Building CXX object CMakeFiles/sensor_msgs.dir/Float.pb.cc.o [ 40%] Building CXX object CMakeFiles/sensor_msgs.dir/Imu.pb.cc.o [ 45%] Building CXX object CMakeFiles/sensor_msgs.dir/MotorSpeed.pb.cc.o [ 50%] Building CXX object CMakeFiles/sensor_msgs.dir/EscSensor.pb.cc.o [ 55%] Building CXX object CMakeFiles/sensor_msgs.dir/vector3d.pb.cc.o [ 60%] Building CXX object CMakeFiles/sensor_msgs.dir/quaternion.pb.cc.o [ 65%] Linking CXX shared library libsensor_msgs.so [ 65%] Built target sensor_msgs [ 70%] Running C++ protocol buffer compiler on msgs/MotorCommand.proto Scanning dependencies of target control_msgs [ 75%] Building CXX object CMakeFiles/control_msgs.dir/MotorCommand.pb.cc.o [ 80%] Linking CXX shared library libcontrol_msgs.so [ 80%] Built target control_msgs Scanning dependencies of target gazebo_imu_plugin [ 85%] Building CXX object CMakeFiles/gazebo_imu_plugin.dir/src/gazebo_imu_plugin.cpp.o [ 90%] Linking CXX shared library libgazebo_imu_plugin.so [ 90%] Built target gazebo_imu_plugin Scanning dependencies of target gazebo_motor_model [ 95%] Building CXX object CMakeFiles/gazebo_motor_model.dir/src/gazebo_motor_model.cpp.o [100%] Linking CXX shared library libgazebo_motor_model.so [100%] Built target gazebo_motor_model Tried to link with, but wont let me as it says those file are already there build folder is again fillefdwith all these files (as in the image I posted) Observation: /home/cuda/projects/gymfc-aircraft-plugins/build/ creates all these files when building with cmake and make, shouldn't just those 2 .so files be created instead? However, as test_axis.py is still outputing same error, deleted /home/cuda/projects/gymfc/modules/digitaltwins/nf1/plugins/build/ Proceed to change the .cpp now, uncommenting those gzdbg lines ln -s ~/projects/gymfc-aircraft-plugins/build/libgazebo_imu_plugin.so libgazebo_imu_plugin.so text_axis.py output with those gzdb lines uncommented: Result: same output as before, with the SetAnchor error(env) cuda@eim-alu-83252:~/projects/gymfc/tests$ python3 test_axis.py ~/projects/gymfc/modules/digitaltwins/nf1/model.sdf --verbose Sending motor control signals to port 9095 Gazebo Model Path = /home/cuda/local/share/gazebo-10/models::/home/cuda/projects/gymfc/gymfc/envs/assets/gazebo/models:/home/cuda/projects/gymfc/modules/digitaltwins Gazebo Plugin Path = /home/cuda/local/lib/gazebo-10/plugins::/home/cuda/projects/gymfc/gymfc/envs/assets/gazebo/plugins/build:/home/cuda/projects/gymfc/modules/digitaltwins/nf1/plugins/build Starting gzserver with process ID= 12207 Roll Left ----------------------------------- Gazebo multi-robot simulator, version 10.1.0 Copyright (C) 2012 Open Source Robotics Foundation. Released under the Apache 2 License. http://gazebosim.org [Msg] Waiting for master. Simulation Statssteps 0 Killing Gazebo process with ID= 12207 |
|
If you aren't seeing the debug statements that you uncommented that could mean the packets sent from python are never getting to the plugin for some reason. You'll need to trace the plugin to see where it's getting too. If you aren't receiving any packets there's a networking problem somewhere on your setup. You can use Wireshark to capture the UDP packets being sent from the Python code. The log messages tell you which UDP port is used to you can filter the packet stream on that. If you need help making sense of the Wireshark output upload it here and I'll take a look. I've never tested it within a VM so I cant say if that's a source of the problem. Other option is to work off of the docker images that have been tested to work. |
|
Following your steps above you built the plugin and the made the code change. You made sure to rebuild the GymFC plugin after the code change right? (The build_plugin.sh script) |
|
@xabierolaz Sorry I've been on mobile, was able to read through your post in greater detail.
The purpose of this test was to trace in FlightController.cpp where you are getting so it can help you figure out what's going on in your setup. Since you didn't make any changes there's nothing new here to use to help debug. If uncommenting gzdbg statements cause error then put your own in that just print out where you are in the code. Most importantly you need to find out if you are getting here. If you aren't then somehow your client and server are not communicating. |
|
Okay, I was uncommenting the https://github.com/wil3/gymfc/blob/master/gymfc/envs/assets/gazebo/plugins/FlightControllerPlugin.cpp#L593 loop, not the https://github.com/wil3/gymfc/blob/master/gymfc/envs/assets/gazebo/plugins/FlightControllerPlugin.cpp#L749 EDIT: Some considerations before using Wireshark to trace the packages 1-Is it okay to pip install wheel before pip3 install -e . gymfc? should be gymfc able to get installed without any bdist_wheel errors ? 2- Should the virtual environment be installed in ~/PROJECTS/gymfc/ Code: |
As a sanity check I recommend printing out a debug statement in the plugin as soon as it starts to make sure you are using the new one you are modifying. Maybe like right after this statement, "[Dbg] [FlightControllerPlugin.cpp:225] Binding on port 9308". Wireshark can verify the python side is sending UDP packets, just filter the UDP port by the port specified in the logs, (i.e., 9308 in the example above). The init sets the hostname to localhost, also make sure thats being resolved. On your machine if you just ping localhost it'll do the translation. |
|
Quick update, installing wheel is not needed and therefore there are no errors when installing gymfc with pip3 install -e . without using any virtual environment. I will look forward to this while I try Wireshark and commenting that line in the .cpp |
|
Tried to run the docker, but my virtual machines docker engine seems unable to run even a simple hello word ,
|
Whenever I run text_axis.py
python3 test_start_sim.py ../modules/digitaltwins/nf1/model.sdfGazebo opens with the nf1 in the middle, but shells wont get pass from here:
After 1 minute, this message appears and gazebo ends.
Watching the youtube video in your Channel, I see that after all these hyphens, this should appear, but wont get pass from there.
The text was updated successfully, but these errors were encountered: