🧑‍🌾 Homebrew main, 7 and 6 dartsim test regressions

[Crola1702]

Environment

• OS Version: MacOS (Homebrew)
• Source main, 7, and 6

Description

• Expected behavior: Tests should pass
• Actual behavior: Multiple test related to dartsim are failing due to a Bus error

Steps to reproduce

1. Run a build in gz-physics main, 7 or 6 homebrew
2. See tests failures

Output

gz-physics7 was failing to build because of #660. This issue is already fixed. However, we still have multiple test regressions to address

Reference build:

• gz-physics-main-homebrew
• gz-physics-7-homebrew
• gz-physics-6-homebrew

Test regressions:

• ConstructEmptyWorldTestUpToEmptyLink/0.ConstructUpToEmptyWorld
  *COMMON_TEST_added_mass_dartsim.test_ran
• COMMON_TEST_addexternalforcetorque_dartsim.test_ran
• COMMON_TEST_collisions_dartsim.test_ran
• COMMON_TEST_detachable_joint_dartsim.test_ran
• COMMON_TEST_free_joint_features_dartsim.test_ran
• COMMON_TEST_joint_features_dartsim.test_ran
• COMMON_TEST_joint_transmitted_wrench_features_dartsim.test_ran
• COMMON_TEST_kinematic_features_dartsim.test_ran
• COMMON_TEST_link_features_dartsim.test_ran
• COMMON_TEST_shape_features_dartsim.test_ran
• COMMON_TEST_simulation_features_dartsim.test_ran
• COMMON_TEST_world_features_dartsim.test_ran
• INTEGRATION_DoublePendulum.test_ran
• AddedMassFeatures.AddedMass
• UNIT_Base_TEST.test_ran
• UNIT_EntityManagement_TEST.test_ran
• UNIT_SDFFeatures_TEST.test_ran

Test failures in log output:

The following tests FAILED:
	 19 - COMMON_TEST_added_mass_dartsim (Bus error)
	 27 - COMMON_TEST_addexternalforcetorque_dartsim (Bus error)
	 43 - COMMON_TEST_collisions_dartsim (Bus error)
	 51 - COMMON_TEST_construct_empty_world_dartsim (Failed)
	 59 - COMMON_TEST_detachable_joint_dartsim (Bus error)
	 67 - COMMON_TEST_free_joint_features_dartsim (Bus error)
	 75 - COMMON_TEST_joint_features_dartsim (Bus error)
	 91 - COMMON_TEST_joint_transmitted_wrench_features_dartsim (Bus error)
	 99 - COMMON_TEST_kinematic_features_dartsim (Bus error)
	107 - COMMON_TEST_link_features_dartsim (SEGFAULT)
	115 - COMMON_TEST_shape_features_dartsim (Bus error)
	123 - COMMON_TEST_simulation_features_dartsim (Bus error)
	131 - COMMON_TEST_world_features_dartsim (Bus error)
	137 - INTEGRATION_DoublePendulum (Bus error)
	168 - UNIT_AddedMassFeatures_TEST (Failed)
	170 - UNIT_Base_TEST (Bus error)
	172 - UNIT_EntityManagement_TEST (SEGFAULT)
	174 - UNIT_SDFFeatures_TEST (Bus error)

See reference builds for complete test output. This is probably happening after #660 fix

[scpeters]

I believe these regressions are related to the release of dartsim 6.14.2

[jslee02]

@scpeters Would it be beneficial to set up tests on the dartsim/dart repo to identify any issues with the Gazebo integration? Could someone help create a CI job for this in the dartsim/dart repo? Testing building and running Dart and Gazebo-Physics within a Docker environment would suffice.

[scpeters]

@scpeters Would it be beneficial to set up tests on the dartsim/dart repo to identify any issues with the Gazebo integration? Could someone help create a CI job for this in the dartsim/dart repo? Testing building and running Dart and Gazebo-Physics within a Docker environment would suffice.

good idea, I'll discuss with the simulation team

[scpeters]

running UNIT_EntityManagement_TEST on my local system shows invalid data for the link name:

/Users/scpeters/ws/dome/src/ign-physics/dartsim/src/EntityManagement_TEST.cc:88: Failure
Expected equality of these values:
  "empty link"
  link->GetName()
    Which is: "\xC9\xEF\xBF\xF7\x7F\0\0@\x82\xA6\0\x1\0\0\0\x14" "empt"

here's the full backtrace

[==========] Running 3 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 3 tests from EntityManagement_TEST
[ RUN      ] EntityManagement_TEST.ConstructEmptyWorld
/Users/scpeters/ws/dome/src/ign-physics/dartsim/src/EntityManagement_TEST.cc:88: Failure
Expected equality of these values:
  "empty link"
  link->GetName()
    Which is: "\xC9\xEF\xBF\xF7\x7F\0\0@\x82\xA6\0\x1\0\0\0\x14" "empt"
Warning [BodyNode.cpp:482] [BodyNode] A negative or zero mass [0] is set to BodyNode [???@??
dumm], which can cause invalid physical behavior or segfault. Consider setting positive value instead.
Process 21694 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x0000000101721713 libdart.6.14.dylib`dart::dynamics::BodyNode::aggregateExternalForces(Eigen::Matrix<double, -1, 1, 0, -1, 1>&) + 211
libdart.6.14.dylib`dart::dynamics::BodyNode::aggregateExternalForces:
->  0x101721713 <+211>: movq   (%rdi), %rax
    0x101721716 <+214>: callq  *0x68(%rax)
    0x101721719 <+217>: testq  %rax, %rax
    0x10172171c <+220>: je     0x10172188f               ; <+591>
Target 0: (UNIT_EntityManagement_TEST) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
  * frame #0: 0x0000000101721713 libdart.6.14.dylib`dart::dynamics::BodyNode::aggregateExternalForces(Eigen::Matrix<double, -1, 1, 0, -1, 1>&) + 211
    frame #1: 0x00000001017f71e2 libdart.6.14.dylib`dart::dynamics::Skeleton::addEntryToJointNameMgr(dart::dynamics::Joint*, bool) + 110
    frame #2: 0x0000000101800f89 libdart.6.14.dylib`dart::dynamics::Skeleton::registerJoint(dart::dynamics::Joint*) + 45
    frame #3: 0x00000001017f5cfd libdart.6.14.dylib`dart::dynamics::Skeleton::registerBodyNode(dart::dynamics::BodyNode*) + 1017
    frame #4: 0x000000010180353b libdart.6.14.dylib`dart::dynamics::Skeleton::moveBodyNodeTree(dart::dynamics::Joint*, dart::dynamics::BodyNode*, std::__1::shared_ptr<dart::dynamics::Skeleton>, dart::dynamics::BodyNode*) + 2869
    frame #5: 0x0000000100995108 libgz-physics7-dartsim-plugin.7.dylib`dart::dynamics::RevoluteJoint* dart::dynamics::Skeleton::moveBodyNodeTree<dart::dynamics::RevoluteJoint>(this=0x000000011103a000, _bodyNode=0x000000011103f000, _newSkeleton=std::__1::shared_ptr<dart::dynamics::Skeleton>::element_type @ 0x000000011103a000 strong=11 weak=8, _parentNode=0x0000000000000000, _joint=0x00007ff7bfefd7c0) at Skeleton.hpp:51:7 [opt]
    frame #6: 0x0000000100990ab4 libgz-physics7-dartsim-plugin.7.dylib`dart::dynamics::RevoluteJoint* dart::dynamics::BodyNode::moveTo<dart::dynamics::RevoluteJoint>(this=0x000000011103f000, _newParent=0x0000000000000000, _joint=0x00007ff7bfefd7c0) at BodyNode.hpp:50:36 [opt]
    frame #7: 0x00000001009905d9 libgz-physics7-dartsim-plugin.7.dylib`gz::physics::dartsim::JointFeatures::AttachRevoluteJoint(this=0x0000000111032e00, _childID=0x00007ff7bfefe780, _parent=<unavailable>, _name="revolute", _axis=<unavailable>) at JointFeatures.cc:613:11 [opt]
    frame #8: 0x0000000100990be5 libgz-physics7-dartsim-plugin.7.dylib`virtual thunk to gz::physics::dartsim::JointFeatures::AttachRevoluteJoint(gz::physics::Identity const&, gz::physics::EntityPtr<gz::physics::Link<gz::physics::FeaturePolicy<double, 3ul>, gz::physics::FeatureList<gz::physics::Feature>>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, Eigen::Matrix<double, 3, 1, 0, 3, 1> const&) at JointFeatures.cc:0 [opt]
    frame #9: 0x00000001000192f3 UNIT_EntityManagement_TEST`gz::physics::AttachRevoluteJointFeature::Link<gz::physics::FeaturePolicy<double, 3ul>, TestFeatureList>::AttachRevoluteJoint(this=<unavailable>, _parent=0x00007ff7bfefdb00, _name="revolute", _axis=0x00007ff7bfefdcc0) at RevoluteJoint.hh:57:17 [opt]
    frame #10: 0x00000001000101c0 UNIT_EntityManagement_TEST`EntityManagement_TEST_ConstructEmptyWorld_Test::TestBody(this=<unavailable>) at EntityManagement_TEST.cc:94:22 [opt]
    frame #11: 0x0000000100049621 UNIT_EntityManagement_TEST`void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) [inlined] void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(object=<unavailable>, method=0x00000000000000000000000000000000, location="the test body") at gtest.cc:2599:10 [opt]
    frame #12: 0x0000000100049611 UNIT_EntityManagement_TEST`void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(object=0x00006000000094a0, method=0x00000000000000000000000000000000, location="the test body") at gtest.cc:2635:14 [opt]
    frame #13: 0x0000000100049589 UNIT_EntityManagement_TEST`testing::Test::Run(this=0x00006000000094a0) at gtest.cc:2674:5 [opt]
    frame #14: 0x000000010004a4b9 UNIT_EntityManagement_TEST`testing::TestInfo::Run(this=0x000000010dd09370) at gtest.cc:2853:11 [opt]
    frame #15: 0x000000010004b2f7 UNIT_EntityManagement_TEST`testing::TestSuite::Run(this=0x000000010dd096e0) at gtest.cc:3012:30 [opt]
    frame #16: 0x000000010005b067 UNIT_EntityManagement_TEST`testing::internal::UnitTestImpl::RunAllTests(this=0x000000010dd09480) at gtest.cc:5870:44 [opt]
    frame #17: 0x000000010005a471 UNIT_EntityManagement_TEST`bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) [inlined] bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(object=<unavailable>, method=0x00000000000000000000000000000000, location="auxiliary test code (environments or event listeners)") at gtest.cc:2599:10 [opt]
    frame #18: 0x000000010005a461 UNIT_EntityManagement_TEST`bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(object=<unavailable>, method=0x00000000000000000000000000000000, location="auxiliary test code (environments or event listeners)") at gtest.cc:2635:14 [opt]
    frame #19: 0x000000010005a3fd UNIT_EntityManagement_TEST`testing::UnitTest::Run(this=0x00000001000842f0) at gtest.cc:5444:10 [opt]
    frame #20: 0x0000000100068c1a UNIT_EntityManagement_TEST`main [inlined] RUN_ALL_TESTS() at gtest.h:2293:73 [opt]
    frame #21: 0x0000000100068c0d UNIT_EntityManagement_TEST`main(argc=1, argv=0x00007ff7bfeff6c8) at gtest_main.cc:51:10 [opt]
    frame #22: 0x00007ff807acd41f dyld`start + 1903
(lldb) 

[scpeters]

This bug has been pretty tricky to track down. I tried building dartsim with gz-physics7 in a colcon workspace on macOS, but that didn't reproduce the problem. The only way I can reproduce it is by installing dartsim with brew install dartsim and compiling gz-physics7 against it.

The diff from 6.13.2 to 6.14.2 is pretty huge (dartsim/dart@v6.13.2...v6.14.2), and seems to have more changes than are listed in the changelog. To investigate further, I've been using git bisect and modifying the dartsim formula to install the recommended commits and then compiling gz-physics7 against it. So far my bisecting has narrowed the possible source of the bug to the following range of git commits: dartsim/dart@383140b...d494604, and I'm suspicious of dartsim/dart@d494604 due to the following comment:

https://github.com/dartsim/dart/blob/v6.14.2/CMakeLists.txt#L80-L88

[scpeters]

I just confirmed that building the homebrew formula with -DDART_ENABLE_SIMD=OFF appears to fix the issue. I'll submit a pull request to homebrew-core

[scpeters]

I just confirmed that building the homebrew formula with -DDART_ENABLE_SIMD=OFF appears to fix the issue. I'll submit a pull request to homebrew-core

Homebrew/homebrew-core#176368

[jslee02]

Ah, it's my bad. -DDART_ENABLE_SIMD=OFF should be the default. Let me create a fix for the next patch release (6.14.3).

[scpeters]

Homebrew/homebrew-core#176368 has been merged and gz-physics tests are passing again. I'll go ahead and re-trigger CI for the other branches and mark this as fixed

https://build.osrfoundation.org/job/gz_physics-ci-gz-physics7-homebrew-amd64/64/