ppo_example.py #59
Comments
|
Hey @xabierolaz been meaning to reach out to you, everything is in the master branch including a PID and PPO demo all used in my thesis. Let me know if you have any further questions. Make sure to remember to build the aircraft plugin s for the NF1 model. I have another branch I'm working on to improve the installation of everything. |
|
As a follow up, do you have the PPO example running? |
|
Not yet, still struggling with some fundamental AI pipeline basics before getting it running, also have to install Tensorflow as it seems in the PPO code it's mandatory |
|
Before getting into the neuro-controllers which is far more complicated than traditional controllers, make sure you have the pid_example.py running and understand it. Gymfc_nf (in examples/) is just an OpenAI gym. If you are familiar with OpenAI gyms it's pretty straight forward. Gymfc_nf just provides a wrapper around GymFC, extending the base class to be able to interface with Gazebo. I suggest reading up on the OpenAI project and walking through all their environments and examples. Also have a look at the book "Reinforcement Learning: An Introduction Second edition" by Sutton and Barto if you are interested in more RL background. |
|
Thanks for the guidance Wil. Im stuck at the same point as @xabierolaz is. Now focussing on pid_example first. |
|
@xabierolaz have you been able to run the file pid_example? I am facing the error If you notice the top part of the terminal screen screenshot, the plugins have been built in the directory |
|
Unless you are installing via pip3 in edit/development mode (-e flag) you could run into problems as out lined in PR #60. Highly recommend using branch i58-dep-script until it is merged into master as it is more stable. @varunag18 did you resolved Issue #61? If so please provide a solution and close the issue. If you can't run test_start_sim.py you aren't going to be able to run anything else. How you installed the motor plugins should work but it conflicts with the procedure here so I'm not sure. Also whenever you are submitting an issue for a command not working please do not screen shot it if its text, text isn't searchable. Please also provide the entire verbose output for the command otherwise there's not enough information to help debug. |
|
Hi Wil, Issue #61 isnt resolved yet. Im able to run test_start_sim.py only by adding sudo. I have detailed it in the issue comments. With respect to the error in pid_example.py which I have mentioned above, i am getting it when running below code:
Also, running sudo python3 pid_example.py gives the below error:
I havent been able to trace the cause of this |
|
Hey @varunag18 we still need to resolve the sudo problem because having to run things sudo will cause issues down the line but the TypeError you are reporting may be a valid bug. Let me try reproducing and I'll get back to you. Having a quick look at the source ppo_example.py does set the sample_noise class attribute so if you want to test if that's the problem you can just copy this to below this line. |
|
See updates to PR: #60 which fixes this bug. |
|
Hi Wil, I have executed the pid_example.py script and obtained the graph, as expected. Now, Im studying the PPO algo's base paper and also trying to understand the code in the files ppo_example.py, pposgd_simple.py, etc. I tried running the file ppo_example.py and obtained the below output. Checkpoints folder got created and files added to it at repeated time intervals, as expected. However, Gazebo didnt load and execution ends abruptly. While Im debugging the code to know whats going wrong, im adding the terminal output here to know your thoughts. Also, nothing gets recorded in the log file created in /tmp folder. |
|
Is this the output of ppo_example.py without any modification or overrides? If so looks like its only executing 1/10 of the # of steps but no error is given. Which log file is supposed to be written to /tmp? Is this an OpenAI thing? I havn't done thorough testing of the ppo refactor so its possible there is a bug. Simulation stats only get output when it shuts down. If you look here calling close is the only instance calling shutdown happens when there is no issue which appears by the output you provided. So it seems somewhere the It this reproducible? What happens if you override timesteps to say 100e3 and 200e3? |
I did modify the timesteps to 10e5 and ckpt-freq to 100e2, thats why its running fr 1/10 of the steps.
There is a log file getting created in /tmp folder as per
Yes,
Not really, when i modified the timesteps to 100e3 or 200e3, there is again an abrupt end to the execution but even one step earlier. |
I'm confused then, why are you saying it's not working? |
|
I guess I posted the query a bit too early ;) |
|
All the PPO example does is train and produce checkpoints. If it's throwing you off it's not producing a plot like the pid example it's because it's not implented too, there's no current validation happening. There are a number of ways to do this. What I've done in the past is monitor the checkpoints directory and when a new checkpoint is created validate it on N number of episodes. Then you can get an idea how well the performance is. The neurocontroller I have in Neuroflight I believe was found in just 2 million steps. I have a script I've be meaning refactor and push will try and get to it in the next couple days. |
|
Got it. Indeed I was expecting PPO example to provide a graph or something to compare and show that its output is better than that of PID example, coz thats the whole point of doing a task through RL algos. I need some more time understanding this PPO code, meanwhile waiting fr this script ur planning to add to the repository. |
|
@varunag18 @xabierolaz PR #75 adds example evaluation and plotting scripts. |
|
@xabierolaz given the updates to the examples/README.md I think it covers your original question. Unless there is an objection I'm going to close this issue. If something else comes up feel free to open another issue. |
|
I'm facing same issue as @varunag18 was having while running the pid_example script |
|
You timed out which usually means the motor plugins were never built. Have you pulled the latest from the master branch? Is the pid example working? |
|
getting same error. Seems like I havent built the plugins yet, as pointed out in imu missing while running start_sim example. Do Ihave to install gymfc again as said in examples/readme? when running test_start_sim.py after having /examples nf1 aircraft plugins built (looks fine) |
|
Hi @xabierolaz If your test scripts are running fine, your pid_example should run too. I havent installed the plugins at more than 1 location. All scripts in test and examples folder are running fine for me. I will state the set of steps I did, maybe you can cross check once. |
|
Please install plugins according to the readme, if there is something wrong with installation instructions please submit a bug report. A model is reusable, you build the plugins once and then you just point to the models SDF file to use it with GymFC.
Not sure what you mean by this. You can only have a single Python package installed at a time. If you install via pip without edit mode (the -e flag) and you make modifications to the source then yes you need to uninstall and reinstall but for the examples you don't need to make any modifications. If there is confusion with installation of GymFC compared to GymFC-NF, GymFC does dynamically build the Gazebo C++ plugins during pip install because they are required by GymFC. Aircraft models (ie digital twins) and their corresponding motor and sensor plugins are totally independent entities by design. This allows you to test multiple aircraft models with GymFC by just pointing to a different model.sdf file. |
|
I also want to mention, there is no point in running the PPO Neuro Controller until you have the PID example working. PPO is much more complex. |
|
Sorry, let me get this clear still have few questions about the structure. What I meant is that inside examples folder there is another gymfc (gymfc_nf) installation, so Im assuming we have to either pick the root folders gymfc installation or the gymfc_nf one, right? 1-Whenever installing gymfc do we have to choose between installing the gymfc at the main root folder or installing examples/gymfc_nfc, which follows gymfc's installation process but also runs automatically the manual building process we had to do when installing gymfc at the end (gymfc/envs/assets/gazebo/plugins/build_plugin.sh) Is this right? should we run too pip3 install -e . inside the GymFC_NF as we did in the normal gymfc install? 2- In examples/readme, at the end of step 2
Should we have it installed by then or you mean 3- In examples/readme, pip3 install is mentioned at the end of step 2 and at the beginning of step 3. I know those might be trivial questions, but need to get this clear in order to know why still not working |
No, GymFC is the framework. It provides the interfaces to Gazebo and the digital twin. GymFC_NF is an implementation of the GymFC framework used for training neurocontrollers. They are completely independent python packages. GymFC is a dependency of GymFC_NF. I highly recommend you review the code.
Since this thread hasn't been related to the specific PPO example I'm going to close it. Before you get to the examples you need to verify your motor and sensor plugins are installed and operating correctly which you can do using the GymFC test scripts. Feel free to open a follow up issue if needed. |
Hi, I'm trying to start testing the PPO algorythm in the thesis-work branch.
Are there any steps on how to get started once GymFC is working? I'm using jupyter notebook, is there any documentation to look for regarding it's OpenAI baselines and how it's related to gymfc?
Thanks
The text was updated successfully, but these errors were encountered: