Apparent race condition with connext generator causes occasional failures

[nuclearsandwich]

This happens occasionally. Rebuilds don't help unless you first remove the cached docker image. The hit ratio is pretty high so once we have the chance to start digging in it should be possible to figure out what's happening.

Log extract

16:06:19 make[4]: Entering directory '/tmp/binarydeb/ros-crystal-actionlib-msgs-0.6.1/obj-x86_64-linux-gnu'
16:06:19 [ 44%] Generating DDS interfaces
16:06:19 /usr/bin/python3 /opt/ros/crystal/lib/rosidl_generator_dds_idl/rosidl_generator_dds_idl --generator-arguments-file /tmp/binarydeb/ros-crystal-actionlib-msgs-0.6.1/obj-x86_64-linux-gnu/rosidl_generator_dds_idl__dds_connext__arguments.json --subfolders dds_connext --extension
16:06:19 [ 45%] Generating C++ type support for RTI Connext (using '/opt/rti.com/rti_connext_dds-5.3.1/bin/rtiddsgen_server')
16:06:19 /usr/bin/python3 /opt/ros/crystal/lib/rosidl_typesupport_connext_cpp/rosidl_typesupport_connext_cpp --generator-arguments-file /tmp/binarydeb/ros-crystal-actionlib-msgs-0.6.1/obj-x86_64-linux-gnu/rosidl_typesupport_connext_cpp__arguments.json --dds-interface-base-path /tmp/binarydeb/ros-crystal-actionlib-msgs-0.6.1/obj-x86_64-linux-gnu/rosidl_generator_dds_idl --idl-pp /opt/rti.com/rti_connext_dds-5.3.1/bin/rtiddsgen_server
16:06:19 ERROR com.rti.ndds.nddsgen.Main Fail:  com.rti.ndds.nddsgen.Main$ArgumentException: File /tmp/binarydeb/ros-crystal-actionlib-msgs-0.6.1/obj-x86_64-linux-gnu/rosidl_generator_dds_idl/actionlib_msgs/msg/dds_connext/GoalID_.idl doesn't exist. Please make sure that you have specified the correct path.
16:06:19 INFO com.rti.ndds.nddsgen.Main Done (failures)
16:06:19 Traceback (most recent call last):
16:06:19   File "/opt/ros/crystal/lib/rosidl_typesupport_connext_cpp/rosidl_typesupport_connext_cpp", line 61, in <module>
16:06:19     sys.exit(main())
16:06:19   File "/opt/ros/crystal/lib/rosidl_typesupport_connext_cpp/rosidl_typesupport_connext_cpp", line 56, in main
16:06:19     service_specs,
16:06:19   File "/opt/ros/crystal/lib/python3.6/site-packages/rosidl_typesupport_connext_cpp/__init__.py", line 89, in generate_dds_connext_cpp
16:06:19     subprocess.check_call(cmd)
16:06:19   File "/usr/lib/python3.6/subprocess.py", line 291, in check_call
16:06:19     raise CalledProcessError(retcode, cmd)
16:06:19 subprocess.CalledProcessError: Command '['/opt/rti.com/rti_connext_dds-5.3.1/bin/rtiddsgen_server', '-I', '/tmp/binarydeb/ros-crystal-actionlib-msgs-0.6.1/obj-x86_64-linux-gnu/rosidl_generator_dds_idl', '-I', '/opt/ros/crystal/share', '-d', '/tmp/binarydeb/ros-crystal-actionlib-msgs-0.6.1/obj-x86_64-linux-gnu/rosidl_typesupport_connext_cpp/actionlib_msgs/msg/dds_connext', '-language', 'C++', '-namespace', '-update', 'typefiles', '-unboundedSupport', '/tmp/binarydeb/ros-crystal-actionlib-msgs-0.6.1/obj-x86_64-linux-gnu/rosidl_generator_dds_idl/actionlib_msgs/msg/dds_connext/GoalID_.idl']' returned non-zero exit status 255.
16:06:19 CMakeFiles/actionlib_msgs__rosidl_typesupport_connext_cpp.dir/build.make:104: recipe for target 'rosidl_typesupport_connext_cpp/actionlib_msgs/msg/goal_id__rosidl_typesupport_connext_cpp.hpp' failed
16:06:19 make[4]: *** [rosidl_typesupport_connext_cpp/actionlib_msgs/msg/goal_id__rosidl_typesupport_connext_cpp.hpp] Error 1
16:06:19 make[4]: *** Deleting file 'rosidl_typesupport_connext_cpp/actionlib_msgs/msg/goal_id__rosidl_typesupport_connext_cpp.hpp'
16:06:19 make[4]: Leaving directory '/tmp/binarydeb/ros-crystal-actionlib-msgs-0.6.1/obj-x86_64-linux-gnu'
16:06:19 CMakeFiles/Makefile2:1315: recipe for target 'CMakeFiles/actionlib_msgs__rosidl_typesupport_connext_cpp.dir/all' failed
16:06:19 make[3]: *** [CMakeFiles/actionlib_msgs__rosidl_typesupport_connext_cpp.dir/all] Error 2
16:06:19 make[3]: Leaving directory '/tmp/binarydeb/ros-crystal-actionlib-msgs-0.6.1/obj-x86_64-linux-gnu'
16:06:19 Makefile:143: recipe for target 'all' failed
16:06:19 make[2]: Leaving directory '/tmp/binarydeb/ros-crystal-actionlib-msgs-0.6.1/obj-x86_64-linux-gnu'
16:06:19 make[2]: *** [all] Error 2
16:06:19 dh_auto_build: cd obj-x86_64-linux-gnu && make -j1 returned exit code 2
16:06:19 debian/rules:34: recipe for target 'override_dh_auto_build' failed
16:06:19 make[1]: Leaving directory '/tmp/binarydeb/ros-crystal-actionlib-msgs-0.6.1'
16:06:19 make[1]: *** [override_dh_auto_build] Error 2
16:06:19 debian/rules:22: recipe for target 'build' failed
16:06:19 make: *** [build] Error 2
16:06:19 dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
16:06:19 E: Building failed
16:06:19 Traceback (most recent call last):
16:06:19   File "/tmp/ros_buildfarm/ros_buildfarm/binarydeb_job.py", line 138, in build_binarydeb
16:06:19     subprocess.check_call(cmd, cwd=source_dir)
16:06:19   File "/usr/lib/python3.6/subprocess.py", line 291, in check_call
16:06:19     raise CalledProcessError(retcode, cmd)
16:06:19 subprocess.CalledProcessError: Command '['apt-src', 'build', 'ros-crystal-actionlib-msgs']' returned non-zero exit status 1.

[tfoote]

Seen here too: http://build.ros2.org/view/Cbin_uB64/job/Cbin_uB64__stereo_msgs__ubuntu_bionic_amd64__binary/6/console

stereo_msgs dependency on DisparityImage though I see that DisparityImage was built noteably higher in the build at 19% where we're at 59%

16:43:42 [ 59%] Generating C++ type support for RTI Connext (using '/opt/rti.com/rti_connext_dds-5.3.1/bin/rtiddsgen_server')
16:43:42 /usr/bin/python3 /opt/ros/crystal/lib/rosidl_typesupport_connext_cpp/rosidl_typesupport_connext_cpp --generator-arguments-file /tmp/binarydeb/ros-crystal-stereo-msgs-0.6.1/obj-x86_64-linux-gnu/rosidl_typesupport_connext_cpp__arguments.json --dds-interface-base-path /tmp/binarydeb/ros-crystal-stereo-msgs-0.6.1/obj-x86_64-linux-gnu/rosidl_generator_dds_idl --idl-pp /opt/rti.com/rti_connext_dds-5.3.1/bin/rtiddsgen_server
16:43:43 ERROR com.rti.ndds.nddsgen.Main Fail:  com.rti.ndds.nddsgen.Main$ArgumentException: File /tmp/binarydeb/ros-crystal-stereo-msgs-0.6.1/obj-x86_64-linux-gnu/rosidl_generator_dds_idl/stereo_msgs/msg/dds_connext/DisparityImage_.idl doesn't exist. Please make sure that you have specified the correct path.
16:43:43 INFO com.rti.ndds.nddsgen.Main Done (failures)
16:43:43 Traceback (most recent call last):
16:43:43   File "/opt/ros/crystal/lib/rosidl_typesupport_connext_cpp/rosidl_typesupport_connext_cpp", line 61, in <module>
16:43:43     sys.exit(main())
16:43:43   File "/opt/ros/crystal/lib/rosidl_typesupport_connext_cpp/rosidl_typesupport_connext_cpp", line 56, in main
16:43:43     service_specs,
16:43:43   File "/opt/ros/crystal/lib/python3.6/site-packages/rosidl_typesupport_connext_cpp/__init__.py", line 89, in generate_dds_connext_cpp
16:43:43     subprocess.check_call(cmd)
16:43:43   File "/usr/lib/python3.6/subprocess.py", line 291, in check_call
16:43:43     raise CalledProcessError(retcode, cmd)
16:43:43 subprocess.CalledProcessError: Command '['/opt/rti.com/rti_connext_dds-5.3.1/bin/rtiddsgen_server', '-I', '/tmp/binarydeb/ros-crystal-stereo-msgs-0.6.1/obj-x86_64-linux-gnu/rosidl_generator_dds_idl', '-I', '/opt/ros/crystal/share', '-d', '/tmp/binarydeb/ros-crystal-stereo-msgs-0.6.1/obj-x86_64-linux-gnu/rosidl_typesupport_connext_cpp/stereo_msgs/msg/dds_connext', '-language', 'C++', '-namespace', '-update', 'typefiles', '-unboundedSupport', '/tmp/binarydeb/ros-crystal-stereo-msgs-0.6.1/obj-x86_64-linux-gnu/rosidl_generator_dds_idl/stereo_msgs/msg/dds_connext/DisparityImage_.idl']' returned non-zero exit status 255.
16:43:43 CMakeFiles/stereo_msgs__rosidl_typesupport_connext_cpp.dir/build.make:160: recipe for target 'rosidl_typesupport_connext_cpp/stereo_msgs/msg/disparity_image__rosidl_typesupport_connext_cpp.hpp' failed
16:43:43 make[4]: Leaving directory '/tmp/binarydeb/ros-crystal-stereo-msgs-0.6.1/obj-x86_64-linux-gnu'
16:43:43 make[4]: *** [rosidl_typesupport_connext_cpp/stereo_msgs/msg/disparity_image__rosidl_typesupport_connext_cpp.hpp] Error 1
16:43:43 make[4]: *** Deleting file 'rosidl_typesupport_connext_cpp/stereo_msgs/msg/disparity_image__rosidl_typesupport_connext_cpp.hpp'
16:43:43 CMakeFiles/Makefile2:1572: recipe for target 'CMakeFiles/stereo_msgs__rosidl_typesupport_connext_cpp.dir/all' failed

This is probably a reasonable place to track this for visibility but the fix is likely needed at some level in the cmake not actually here.

Update: this one passed on a straight rebuild: http://build.ros2.org/view/Cbin_uB64/job/Cbin_uB64__stereo_msgs__ubuntu_bionic_amd64__binary/7/

[nuclearsandwich]

A few more from last night's rebuild.

http://build.ros2.org/job/Cbin_uB64__nav_2d_msgs__ubuntu_bionic_amd64__binary/13
http://build.ros2.org/job/Cbin_uB64__realsense_camera_msgs__ubuntu_bionic_amd64__binary/7/
http://build.ros2.org/job/Cbin_uB64__stereo_msgs__ubuntu_bionic_amd64__binary/10/

[nuclearsandwich]

http://build.ros2.org/view/Cbin_uB64/job/Cbin_uB64__action_msgs__ubuntu_bionic_amd64__binary/11/
http://build.ros2.org/job/Cbin_uB64__lifecycle_msgs__ubuntu_bionic_amd64__binary/10/
http://build.ros2.org/job/Cbin_uB64__sensor_msgs__ubuntu_bionic_amd64__binary/12/

[nuclearsandwich]

http://build.ros2.org/job/Cbin_uB64__pendulum_msgs__ubuntu_bionic_amd64__binary/10