Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] Threaded predict error. #11103

Open
trivialfis opened this issue Dec 15, 2024 · 0 comments
Open

[CI] Threaded predict error. #11103

trivialfis opened this issue Dec 15, 2024 · 0 comments

Comments

@trivialfis
Copy link
Member

trivialfis commented Dec 15, 2024

https://github.com/dmlc/xgboost/actions/runs/12339183942/job/34435083920

 [ RUN      ] Learner.MultiThreadedPredict
=================================================================
==3895==ERROR: AddressSanitizer: heap-use-after-free on address 0x603008e17ec8 at pc 0x55b82804543a bp 0x7f6d7f5fde60 sp 0x7f6d7f5fde50
READ of size 4 at 0x603008e17ec8 thread T15161
    #0 0x55b828045439 in xgboost::gbm::GBTree::PredictBatchImpl(xgboost::DMatrix*, xgboost::PredictionCacheEntry*, bool, int, int) const /workspace/src/gbm/gbtree.cc:490
    #1 0x55b828047051 in xgboost::gbm::GBTree::PredictBatch(xgboost::DMatrix*, xgboost::PredictionCacheEntry*, bool, int, int) /workspace/src/gbm/gbtree.cc:528
    #2 0x55b82814bf5f in xgboost::LearnerImpl::PredictRaw(xgboost::DMatrix*, xgboost::PredictionCacheEntry*, bool, unsigned int, unsigned int) const /workspace/src/learner.cc:1473
    #3 0x55b828148ecd in xgboost::LearnerImpl::Predict(std::shared_ptr<xgboost::DMatrix>, bool, xgboost::HostDeviceVector<float>*, int, int, bool, bool, bool, bool, bool) /workspace/src/learner.cc:1399
    #4 0x55b82992aadd in operator() /workspace/tests/cpp/test_learner.cc:274
    #5 0x55b82997275e in __invoke_impl<void, xgboost::Learner_MultiThreadedPredict_Test::TestBody()::<lambda()> > /usr/include/c++/10/bits/invoke.h:60
    #6 0x55b829972265 in __invoke<xgboost::Learner_MultiThreadedPredict_Test::TestBody()::<lambda()> > /usr/include/c++/10/bits/invoke.h:95
    #7 0x55b829971a6c in _M_invoke<0> /usr/include/c++/10/thread:264
    #8 0x55b8299706b4 in operator() /usr/include/c++/10/thread:271
    #9 0x55b82996bb31 in _M_run /usr/include/c++/10/thread:215
    #10 0x7f6e5c0ed252  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc252)
    #11 0x7f6e5bd73ac2  (/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
    #12 0x7f6e5be04a03 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x125a03)

0x603008e17ec8 is located 24 bytes inside of 32-byte region [0x603008e17eb0,0x603008e17ed0)
freed by thread T15161 here:
    #0 0x7f6e5c95824f in operator delete(void*, unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:172
    #1 0x55b8281ba2e9 in __gnu_cxx::new_allocator<std::_Sp_counted_ptr_inplace<xgboost::PredictionCacheEntry, std::allocator<xgboost::PredictionCacheEntry>, (__gnu_cxx::_Lock_policy)2> >::deallocate(std::_Sp_counted_ptr_inplace<xgboost::PredictionCacheEntry, std::allocator<xgboost::PredictionCacheEntry>, (__gnu_cxx::_Lock_policy)2>*, unsigned long) /usr/include/c++/10/ext/new_allocator.h:139
    #2 0x55b8281b8514 in std::allocator_traits<std::allocator<std::_Sp_counted_ptr_inplace<xgboost::PredictionCacheEntry, std::allocator<xgboost::PredictionCacheEntry>, (__gnu_cxx::_Lock_policy)2> > >::deallocate(std::allocator<std::_Sp_counted_ptr_inplace<xgboost::PredictionCacheEntry, std::allocator<xgboost::PredictionCacheEntry>, (__gnu_cxx::_Lock_policy)2> >&, std::_Sp_counted_ptr_inplace<xgboost::PredictionCacheEntry, std::allocator<xgboost::PredictionCacheEntry>, (__gnu_cxx::_Lock_policy)2>*, unsigned long) /usr/include/c++/10/bits/alloc_traits.h:492
    #3 0x55b8281b4818 in std::__allocated_ptr<std::allocator<std::_Sp_counted_ptr_inplace<xgboost::PredictionCacheEntry, std::allocator<xgboost::PredictionCacheEntry>, (__gnu_cxx::_Lock_policy)2> > >::~__allocated_ptr() /usr/include/c++/10/bits/allocated_ptr.h:73
    #4 0x55b8281bc991 in std::_Sp_counted_ptr_inplace<xgboost::PredictionCacheEntry, std::allocator<xgboost::PredictionCacheEntry>, (__gnu_cxx::_Lock_policy)2>::_M_destroy() (/workspace/build/testxgboost+0x9a2b991)
    #5 0x55b827138d9c in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() /usr/include/c++/10/bits/shared_ptr_base.h:174
    #6 0x55b82710aaab in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/include/c++/10/bits/shared_ptr_base.h:736
    #7 0x55b8280f9c09 in std::__shared_ptr<xgboost::PredictionCacheEntry, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/include/c++/10/bits/shared_ptr_base.h:1188
    #8 0x55b8280f9c70 in std::shared_ptr<xgboost::PredictionCacheEntry>::~shared_ptr() /usr/include/c++/10/bits/shared_ptr.h:121
    #9 0x55b8280f9e86 in xgboost::PredictionContainer::Cache(std::shared_ptr<xgboost::DMatrix>, xgboost::DeviceOrd) /workspace/include/xgboost/predictor.h:55
    #10 0x55b828148d30 in xgboost::LearnerImpl::Predict(std::shared_ptr<xgboost::DMatrix>, bool, xgboost::HostDeviceVector<float>*, int, int, bool, bool, bool, bool, bool) /workspace/src/learner.cc:1398
    #11 0x55b82992aadd in operator() /workspace/tests/cpp/test_learner.cc:274
    #12 0x55b82997275e in __invoke_impl<void, xgboost::Learner_MultiThreadedPredict_Test::TestBody()::<lambda()> > /usr/include/c++/10/bits/invoke.h:60
    #13 0x55b829972265 in __invoke<xgboost::Learner_MultiThreadedPredict_Test::TestBody()::<lambda()> > /usr/include/c++/10/bits/invoke.h:95
    #14 0x55b829971a6c in _M_invoke<0> /usr/include/c++/10/thread:264
    #15 0x55b8299706b4 in operator() /usr/include/c++/10/thread:271
    #16 0x55b82996bb31 in _M_run /usr/include/c++/10/thread:215
    #17 0x7f6e5c0ed252  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc252)

previously allocated by thread T15161 here:
    #0 0x7f6e5c9571e7 in operator new(unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x55b8281ba2a2 in __gnu_cxx::new_allocator<std::_Sp_counted_ptr_inplace<xgboost::PredictionCacheEntry, std::allocator<xgboost::PredictionCacheEntry>, (__gnu_cxx::_Lock_policy)2> >::allocate(unsigned long, void const*) /usr/include/c++/10/ext/new_allocator.h:121
    #2 0x55b8281b83dc in std::allocator_traits<std::allocator<std::_Sp_counted_ptr_inplace<xgboost::PredictionCacheEntry, std::allocator<xgboost::PredictionCacheEntry>, (__gnu_cxx::_Lock_policy)2> > >::allocate(std::allocator<std::_Sp_counted_ptr_inplace<xgboost::PredictionCacheEntry, std::allocator<xgboost::PredictionCacheEntry>, (__gnu_cxx::_Lock_policy)2> >&, unsigned long) /usr/include/c++/10/bits/alloc_traits.h:460
    #3 0x55b8281b46af in std::__allocated_ptr<std::allocator<std::_Sp_counted_ptr_inplace<xgboost::PredictionCacheEntry, std::allocator<xgboost::PredictionCacheEntry>, (__gnu_cxx::_Lock_policy)2> > > std::__allocate_guarded<std::allocator<std::_Sp_counted_ptr_inplace<xgboost::PredictionCacheEntry, std::allocator<xgboost::PredictionCacheEntry>, (__gnu_cxx::_Lock_policy)2> > >(std::allocator<std::_Sp_counted_ptr_inplace<xgboost::PredictionCacheEntry, std::allocator<xgboost::PredictionCacheEntry>, (__gnu_cxx::_Lock_policy)2> >&) /usr/include/c++/10/bits/allocated_ptr.h:97
    #4 0x55b8281adb47 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<xgboost::PredictionCacheEntry, std::allocator<xgboost::PredictionCacheEntry>>(xgboost::PredictionCacheEntry*&, std::_Sp_alloc_shared_tag<std::allocator<xgboost::PredictionCacheEntry> >) (/workspace/build/testxgboost+0x9a1cb47)
    #5 0x55b8281a4057 in std::__shared_ptr<xgboost::PredictionCacheEntry, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<xgboost::PredictionCacheEntry>>(std::_Sp_alloc_shared_tag<std::allocator<xgboost::PredictionCacheEntry> >) (/workspace/build/testxgboost+0x9a13057)
    #6 0x55b828195c8b in std::shared_ptr<xgboost::PredictionCacheEntry>::shared_ptr<std::allocator<xgboost::PredictionCacheEntry>>(std::_Sp_alloc_shared_tag<std::allocator<xgboost::PredictionCacheEntry> >) (/workspace/build/testxgboost+0x9a04c8b)
    #7 0x55b828180435 in std::shared_ptr<xgboost::PredictionCacheEntry> std::allocate_shared<xgboost::PredictionCacheEntry, std::allocator<xgboost::PredictionCacheEntry>>(std::allocator<xgboost::PredictionCacheEntry> const&) /usr/include/c++/10/bits/shared_ptr.h:862
    #8 0x55b828166374 in std::shared_ptr<xgboost::PredictionCacheEntry> std::make_shared<xgboost::PredictionCacheEntry>() (/workspace/build/testxgboost+0x99d5374)
    #9 0x55b82814f898 in std::shared_ptr<xgboost::PredictionCacheEntry> xgboost::DMatrixCache<xgboost::PredictionCacheEntry>::CacheItem<>(std::shared_ptr<xgboost::DMatrix>) (/workspace/build/testxgboost+0x99be898)
    #10 0x55b8280f9dba in xgboost::PredictionContainer::Cache(std::shared_ptr<xgboost::DMatrix>, xgboost::DeviceOrd) /workspace/include/xgboost/predictor.h:55
    #11 0x55b828148d30 in xgboost::LearnerImpl::Predict(std::shared_ptr<xgboost::DMatrix>, bool, xgboost::HostDeviceVector<float>*, int, int, bool, bool, bool, bool, bool) /workspace/src/learner.cc:1398
    #12 0x55b82992aadd in operator() /workspace/tests/cpp/test_learner.cc:274
    #13 0x55b82997275e in __invoke_impl<void, xgboost::Learner_MultiThreadedPredict_Test::TestBody()::<lambda()> > /usr/include/c++/10/bits/invoke.h:60
    #14 0x55b829972265 in __invoke<xgboost::Learner_MultiThreadedPredict_Test::TestBody()::<lambda()> > /usr/include/c++/10/bits/invoke.h:95
    #15 0x55b829971a6c in _M_invoke<0> /usr/include/c++/10/thread:264
    #16 0x55b8299706b4 in operator() /usr/include/c++/10/thread:271
    #17 0x55b82996bb31 in _M_run /usr/include/c++/10/thread:215
    #18 0x7f6e5c0ed252  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc252)

Thread T15161 created by T0 here:
    #0 0x7f6e5c8f9685 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
    #1 0x7f6e5c0ed328 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc328)
    #2 0x55b82994b5a3 in construct<std::thread, xgboost::Learner_MultiThreadedPredict_Test::TestBody()::<lambda()> > /usr/include/c++/10/ext/new_allocator.h:156
    #3 0x55b829947414 in construct<std::thread, xgboost::Learner_MultiThreadedPredict_Test::TestBody()::<lambda()> > /usr/include/c++/10/bits/alloc_traits.h:512
    #4 0x55b82994179d in emplace_back<xgboost::Learner_MultiThreadedPredict_Test::TestBody()::<lambda()> > /usr/include/c++/10/bits/vector.tcc:115
    #5 0x55b82992c698 in xgboost::Learner_MultiThreadedPredict_Test::TestBody() /workspace/tests/cpp/test_learner.cc:279
    #6 0x55b829d24937 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /workspace/build/googletest-src/googletest/src/gtest.cc:2607
    #7 0x55b829d06af6 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /workspace/build/googletest-src/googletest/src/gtest.cc:2643
    #8 0x55b829c4da6d in testing::Test::Run() /workspace/build/googletest-src/googletest/src/gtest.cc:2682
    #9 0x55b829c51394 in testing::TestInfo::Run() /workspace/build/googletest-src/googletest/src/gtest.cc:2861
    #10 0x55b829c556f9 in testing::TestSuite::Run() /workspace/build/googletest-src/googletest/src/gtest.cc:3015
    #11 0x55b829c9cc5b in testing::internal::UnitTestImpl::RunAllTests() /workspace/build/googletest-src/googletest/src/gtest.cc:5855
    #12 0x55b829d2a878 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /workspace/build/googletest-src/googletest/src/gtest.cc:2607
    #13 0x55b829d0b799 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /workspace/build/googletest-src/googletest/src/gtest.cc:2643
    #14 0x55b829c902f8 in testing::UnitTest::Run() /workspace/build/googletest-src/googletest/src/gtest.cc:5438
    #15 0x55b8299d502b in RUN_ALL_TESTS() /workspace/build/googletest-src/googletest/include/gtest/gtest.h:2490
    #16 0x55b8299d4bb2 in main /workspace/tests/cpp/test_main.cc:19
    #17 0x7f6e5bd08d8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)

SUMMARY: AddressSanitizer: heap-use-after-free /workspace/src/gbm/gbtree.cc:490 in xgboost::gbm::GBTree::PredictBatchImpl(xgboost::DMatrix*, xgboost::PredictionCacheEntry*, bool, int, int) const
Shadow bytes around the buggy address:
  0x0c06811baf80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c06811baf90: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c06811bafa0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c06811bafb0: fa fa fa fa fa fa fa fa fd fd fd fa fa fa fd fd
  0x0c06811bafc0: fd fa fa fa fa fa fa fa fa fa fd fd fd fa fa fa
=>0x0c06811bafd0: 00 00 00 fa fa fa fd fd fd[fd]fa fa fd fd fd fa
  0x0c06811bafe0: fa fa fd fd fd fa fa fa 00 00 00 00 fa fa 00 00
  0x0c06811baff0: 00 fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c06811bb000: fd fd fd fd fa fa fd fd fd fd fa fa fa fa fa fa
  0x0c06811bb010: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c06811bb020: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==3895==ABORTING
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant