Analysis library crash on exit #908
Description
Rarely, in REST production test runs, there are segfaults as the process is terminating. It looks like this is in the ANALYSIS library.
Example output files are in this directory, and the stack trace is given below:
/volatile/halld/home/gxproj4/jobs/ver13-perl/job_info_120400_191
17:24:16.021 [fatal] Segfault detected! 17:24:16.022 [fatal] Hard exit due to segmentation fault! Backtrace: 17:24:16.022 [fatal] 17:24:16.022 [fatal] 0: DAnalysis::DSourceComboTimeHandler::Fill_Histograms() 17:24:16.022 [fatal] hd_root:6076f0 17:24:16.022 [fatal] 17:24:16.022 [fatal] 1: DAnalysis::DSourceComboer::~DSourceComboer() 17:24:16.022 [fatal] hd_root:4127a7 17:24:16.022 [fatal] 17:24:16.022 [fatal] 2: DAnalysisResults_factory::~DAnalysisResults_factory() 17:24:16.022 [fatal] hd_root:431132 17:24:16.022 [fatal] 17:24:16.022 [fatal] 3: DAnalysisResults_factory::~DAnalysisResults_factory() 17:24:16.022 [fatal] hd_root:431799 17:24:16.022 [fatal] 17:24:16.022 [fatal] 4: JFactorySet::~JFactorySet() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1f9997 17:24:16.022 [fatal] 17:24:16.022 [fatal] 5: JFactorySet::~JFactorySet() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1f9a82 17:24:16.022 [fatal] 17:24:16.022 [fatal] 6: JEvent::~JEvent() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1eb6f5 17:24:16.022 [fatal] 17:24:16.022 [fatal] 7: void __gnu_cxx::new_allocator::destroy(JEvent*) 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:22d749 17:24:16.022 [fatal] 17:24:16.022 [fatal] 8: void std::allocator_traits >::destroy(std::allocator&, JEvent*) 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:22d715 17:24:16.022 [fatal] 17:24:16.022 [fatal] 9: std::_Sp_counted_ptr_inplace, (__gnu_cxx::_Lock_policy)2>::_M_dispose() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:22d603 17:24:16.022 [fatal] 17:24:16.022 [fatal] 10: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() 17:24:16.022 [fatal] hd_root:3c811a 17:24:16.022 [fatal] 17:24:16.022 [fatal] 11: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d7fc9 17:24:16.022 [fatal] 17:24:16.022 [fatal] 12: std::__shared_ptr::~__shared_ptr() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1f4850 17:24:16.022 [fatal] 17:24:16.022 [fatal] 13: std::shared_ptr::~shared_ptr() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1f4892 17:24:16.022 [fatal] 17:24:16.022 [fatal] 14: void std::_Destroy >(std::shared_ptr*) 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:22b6d5 17:24:16.022 [fatal] 17:24:16.022 [fatal] 15: void std::_Destroy_aux::__destroy*>(std::shared_ptr*, std::shared_ptr*) 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:22a4c0 17:24:16.022 [fatal] 17:24:16.022 [fatal] 16: void std::_Destroy*>(std::shared_ptr*, std::shared_ptr*) 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:228836 17:24:16.022 [fatal] 17:24:16.022 [fatal] 17: void std::_Destroy*, std::shared_ptr >(std::shared_ptr*, std::shared_ptr*, std::allocator >&) 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:226b4d 17:24:16.022 [fatal] 17:24:16.022 [fatal] 18: std::vector, std::allocator > >::~vector() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:21fa07 17:24:16.022 [fatal] 17:24:16.022 [fatal] 19: JEventPool::~JEventPool() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:22d526 17:24:16.022 [fatal] 17:24:16.022 [fatal] 20: JEventPool::~JEventPool() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:22d54e 17:24:16.022 [fatal] 17:24:16.022 [fatal] 21: JTopologyBuilder::~JTopologyBuilder() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:218519 17:24:16.022 [fatal] 17:24:16.022 [fatal] 22: void __gnu_cxx::new_allocator::destroy(JTopologyBuilder*) 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1eace1 17:24:16.022 [fatal] 17:24:16.022 [fatal] 23: void std::allocator_traits >::destroy(std::allocator&, JTopologyBuilder*) 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1e9861 17:24:16.022 [fatal] 17:24:16.022 [fatal] 24: std::_Sp_counted_ptr_inplace, (__gnu_cxx::_Lock_policy)2>::_M_dispose() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1e883d 17:24:16.022 [fatal] 17:24:16.022 [fatal] 25: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() 17:24:16.022 [fatal] hd_root:3c811a 17:24:16.022 [fatal] 17:24:16.022 [fatal] 26: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d7fc9 17:24:16.022 [fatal] 17:24:16.022 [fatal] 27: std::__shared_ptr::~__shared_ptr() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d580c 17:24:16.022 [fatal] 17:24:16.022 [fatal] 28: std::shared_ptr::~shared_ptr() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d5828 17:24:16.022 [fatal] 17:24:16.022 [fatal] 29: jana::components::JComponent::Service::~Service() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d585a 17:24:16.022 [fatal] 17:24:16.022 [fatal] 30: JExecutionEngine::~JExecutionEngine() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:20bde3 17:24:16.022 [fatal] 17:24:16.022 [fatal] 31: JExecutionEngine::~JExecutionEngine() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:20be0a 17:24:16.022 [fatal] 17:24:16.022 [fatal] 32: std::default_delete::operator()(JExecutionEngine*) const 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1dd23e 17:24:16.022 [fatal] 17:24:16.022 [fatal] 33: std::_Sp_counted_deleter, std::allocator, (__gnu_cxx::_Lock_policy)2>::_M_dispose() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1e8f76 17:24:16.022 [fatal] 17:24:16.022 [fatal] 34: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() 17:24:16.022 [fatal] hd_root:3c811a 17:24:16.022 [fatal] 17:24:16.022 [fatal] 35: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d7fc9 17:24:16.022 [fatal] 17:24:16.022 [fatal] 36: std::__shared_ptr::~__shared_ptr() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d9e90 17:24:16.022 [fatal] 17:24:16.022 [fatal] 37: std::shared_ptr::~shared_ptr() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d9ed2 17:24:16.022 [fatal] 17:24:16.022 [fatal] 38: std::pair >::~pair() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1e6818 17:24:16.022 [fatal] 17:24:16.022 [fatal] 39: void __gnu_cxx::new_allocator > > >::destroy > >(std::pair >*) 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1e6838 17:24:16.022 [fatal] 17:24:16.022 [fatal] 40: void std::allocator_traits > > > >::destroy > >(std::allocator > > >&, std::pair >*) 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1e5459 17:24:16.022 [fatal] 17:24:16.022 [fatal] 41: std::_Rb_tree >, std::_Select1st > >, std::less, std::allocator > > >::_M_destroy_node(std::_Rb_tree_node > >*) 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1e3609 17:24:16.022 [fatal] 17:24:16.022 [fatal] 42: std::_Rb_tree >, std::_Select1st > >, std::less, std::allocator > > >::_M_drop_node(std::_Rb_tree_node > >*) 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1e18c9 17:24:16.022 [fatal] 17:24:16.022 [fatal] 43: std::_Rb_tree >, std::_Select1st > >, std::less, std::allocator > > >::_M_erase(std::_Rb_tree_node > >*) 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1dfca5 17:24:16.022 [fatal] 17:24:16.022 [fatal] 44: std::_Rb_tree >, std::_Select1st > >, std::less, std::allocator > > >::~_Rb_tree() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1dd068 17:24:16.022 [fatal] 17:24:16.022 [fatal] 45: std::map, std::less, std::allocator > > >::~map() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d9098 17:24:16.022 [fatal] 17:24:16.022 [fatal] 46: JServiceLocator::~JServiceLocator() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1dcc7a 17:24:16.022 [fatal] 17:24:16.022 [fatal] 47: std::default_delete::operator()(JServiceLocator*) const 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1dcca0 17:24:16.022 [fatal] 17:24:16.022 [fatal] 48: std::unique_ptr >::~unique_ptr() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d8dc4 17:24:16.022 [fatal] 17:24:16.022 [fatal] 49: JApplication::~JApplication() 17:24:16.022 [fatal] /cvmfs/oasis.opensciencegrid.org/gluex/group/halld/Software/builds/Linux_Alma9-x86_64-gcc11.5.0-cntr/jana/jana_2.4.0/Linux_Alma9-x86_64-gcc11.5.0-cntr/lib/libJANA.so:1d2524 17:24:16.022 [fatal] 17:24:16.022 [fatal] 50: main 17:24:16.022 [fatal] hd_root:3a4505 17:24:16.022 [fatal] 17:24:16.022 [fatal] 51: ??? 17:24:16.022 [fatal] /lib64/libc.so.6:295d0 17:24:16.022 [fatal] 17:24:16.022 [fatal] 52: __libc_start_main 17:24:16.022 [fatal] /lib64/libc.so.6:29680 17:24:16.022 [fatal] 17:24:16.022 [fatal] 53: _start 17:24:16.022 [fatal] hd_root:3b1c85 17:24:16.022 [fatal]