Spine 1.2.28 encountered segfault

[kiwildante]

Describe the bug
i have a few cacti server that collect data for my device, i have server that only have about a hundred devices, and i have server that have a thousand devices, both produce segfault, but the hundred one only produce a few, but in the thousand device produce a lot with no backtrace, a few have a backtrace, i already set nofiles to a larger one, and have mysql max connection larger, but it still produce the same result, i run with 20/10 p/t for the large one, and 3/3 for the hundred.
it affect the polling time to be maximum, cause maybe the poller is waiting the spine to complete but the spine produce a segfault

To Reproduce
Steps to reproduce the behavior:

1. Go to 'logs'
2. Change to cacti_stderr.log
3. Some Spine Error segfault
4. See error

Expected behavior
poller runs normally without segfault

Screenshots
here for the hundred device log

here for the thousand device

here's with backtrace

here's poller information for the thousand one

Server (please complete the following information):

• OS: Ubuntu 16.04.7 LTS
• Version 1.2.28

Compiling (please complete the following information):

• compiler: gcc 5.4.0
• autoconf: autoconf 2.69
• glibc: ldd 2.23
• source: files cacti spine latest

Note: you can find glibc version by running ldd --version

Note: if source is github, please include last commit reference

Additional context

full backtrace log

2024-10-16 12:23:44 - FATAL: Spine Encountered a Segmentation Fault

Generating backtrace...0 line(s)...
*** Error in `/usr/local/spine/bin/spine': double free or corruption (!prev): 0x000000000255aa40 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777f5)[0x7f041833d7f5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8038a)[0x7f041834638a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f041834a58c]
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(+0x56846)[0x7f0418c0c846]
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(+0x281ba)[0x7f0418bde1ba]
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(+0x2ec11)[0x7f0418be4c11]
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(mysql_send_query+0x57)[0x7f0418be1187]
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(mysql_real_query+0x11)[0x7f0418be1201]
/usr/local/spine/bin/spine[0x4057ab]
/usr/local/spine/bin/spine[0x40e479]
/usr/local/spine/bin/spine[0x412ab2]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f04186976ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f04183cd51d]
======= Memory map: ========
00400000-00424000 r-xp 00000000 08:01 922807 /usr/local/spine/bin/spine
00623000-00624000 r--p 00023000 08:01 922807 /usr/local/spine/bin/spine
00624000-00625000 rw-p 00024000 08:01 922807 /usr/local/spine/bin/spine
00625000-0069a000 rw-p 00000000 00:00 0
024d4000-0258a000 rw-p 00000000 00:00 0 [heap]
7f03e4000000-7f03e4039000 rw-p 00000000 00:00 0
7f03e4039000-7f03e8000000 ---p 00000000 00:00 0
7f03e8000000-7f03e8023000 rw-p 00000000 00:00 0
7f03e8023000-7f03ec000000 ---p 00000000 00:00 0
7f03ec000000-7f03ec022000 rw-p 00000000 00:00 0
7f03ec022000-7f03f0000000 ---p 00000000 00:00 0
7f03f0000000-7f03f003f000 rw-p 00000000 00:00 0
7f03f003f000-7f03f4000000 ---p 00000000 00:00 0
7f03f4000000-7f03f4022000 rw-p 00000000 00:00 0
7f03f4022000-7f03f8000000 ---p 00000000 00:00 0
7f03f8000000-7f03f8022000 rw-p 00000000 00:00 0
7f03f8022000-7f03fc000000 ---p 00000000 00:00 0
7f03fdffc000-7f03fdffd000 ---p 00000000 00:00 0
7f03fdffd000-7f03fe7fd000 rw-p 00000000 00:00 0
7f03fe7fd000-7f03fe7fe000 ---p 00000000 00:00 0
7f03fe7fe000-7f03feffe000 rw-p 00000000 00:00 0
7f03feffe000-7f03fefff000 ---p 00000000 00:00 0
7f03fefff000-7f03ff7ff000 rw-p 00000000 00:00 0
7f03ff7ff000-7f03ff800000 ---p 00000000 00:00 0
7f03ff800000-7f0400000000 rw-p 00000000 00:00 0
7f0400000000-7f0400022000 rw-p 00000000 00:00 0
7f0400022000-7f0404000000 ---p 00000000 00:00 0
7f0404000000-7f0404039000 rw-p 00000000 00:00 0
7f0404039000-7f0408000000 ---p 00000000 00:00 0
7f0408000000-7f0408022000 rw-p 00000000 00:00 0
7f0408022000-7f040c000000 ---p 00000000 00:00 0
7f040c000000-7f040c022000 rw-p 00000000 00:00 0
7f040c022000-7f0410000000 ---p 00000000 00:00 0
7f0410000000-7f0410022000 rw-p 00000000 00:00 0
7f0410022000-7f0414000000 ---p 00000000 00:00 0
7f041404b000-7f041404c000 ---p 00000000 00:00 0
7f041404c000-7f041484c000 rw-p 00000000 00:00 0
7f041484c000-7f041484d000 ---p 00000000 00:00 0
7f041484d000-7f041504d000 rw-p 00000000 00:00 0
7f041504d000-7f041504e000 ---p 00000000 00:00 0
7f041504e000-7f041584e000 rw-p 00000000 00:00 0
7f041584e000-7f041584f000 ---p 00000000 00:00 0
7f041584f000-7f041604f000 rw-p 00000000 00:00 0
7f041604f000-7f0416050000 ---p 00000000 00:00 0
7f0416050000-7f0416850000 rw-p 00000000 00:00 0
7f0416850000-7f0416851000 ---p 00000000 00:00 0
7f0416851000-7f0417051000 rw-p 00000000 00:00 0
7f0417051000-7f041705c000 r-xp 00000000 08:01 7340172 /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f041705c000-7f041725b000 ---p 0000b000 08:01 7340172 /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f041725b000-7f041725c000 r--p 0000a000 08:01 7340172 /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f041725c000-7f041725d000 rw-p 0000b000 08:01 7340172 /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f041725d000-7f0417263000 rw-p 00000000 00:00 0
7f0417263000-7f0417279000 r-xp 00000000 08:01 7340307 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f0417279000-7f0417478000 ---p 00016000 08:01 7340307 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f0417478000-7f0417479000 rw-p 00015000 08:01 7340307 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f0417479000-7f04175eb000 r-xp 00000000 08:01 789126 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f04175eb000-7f04177eb000 ---p 00172000 08:01 789126 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f04177eb000-7f04177f5000 r--p 00172000 08:01 789126 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f04177f5000-7f04177f7000 rw-p 0017c000 08:01 789126 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f04177f7000-7f04177fb000 rw-p 00000000 00:00 0
7f04177fb000-7f0417859000 r-xp 00000000 08:01 7340369 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f0417859000-7f0417a59000 ---p 0005e000 08:01 7340369 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f0417a59000-7f0417a5d000 r--p 0005e000 08:01 7340369 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f0417a5d000-7f0417a63000 rw-p 00062000 08:01 7340369 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f0417a63000-7f0417a7c000 r-xp 00000000 08:01 7340235 /lib/x86_64-linux-gnu/libz.so.1.2.8
7f0417a7c000-7f0417c7b000 ---p 00019000 08:01 7340235 /lib/x86_64-linux-gnu/libz.so.1.2.8
7f0417c7b000-7f0417c7c000 r--p 00018000 08:01 7340235 /lib/x86_64-linux-gnu/libz.so.1.2.8
7f0417c7c000-7f0417c7d000 rw-p 00019000 08:01 7340235 /lib/x86_64-linux-gnu/libz.so.1.2.8
7f0417c7d000-7f0417c80000 r-xp 00000000 08:01 7340163 /lib/x86_64-linux-gnu/libdl-2.23.so
7f0417c80000-7f0417e7f000 ---p 00003000 08:01 7340163 /lib/x86_64-linux-gnu/libdl-2.23.so
7f0417e7f000-7f0417e80000 r--p 00002000 08:01 7340163 /lib/x86_64-linux-gnu/libdl-2.23.so
7f0417e80000-7f0417e81000 rw-p 00003000 08:01 7340163 /lib/x86_64-linux-gnu/libdl-2.23.so
7f0417e81000-7f041809c000 r-xp 00000000 08:01 7340094 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f041809c000-7f041829b000 ---p 0021b000 08:01 7340094 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f041829b000-7f04182b7000 r--p 0021a000 08:01 7340094 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f04182b7000-7f04182c3000 rw-p 00236000 08:01 7340094 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f04182c3000-7f04182c6000 rw-p 00000000 00:00 0
7f04182c6000-7f0418486000 r-xp 00000000 08:01 7340200 /lib/x86_64-linux-gnu/libc-2.23.so
7f0418486000-7f0418686000 ---p 001c0000 08:01 7340200 /lib/x86_64-linux-gnu/libc-2.23.so
7f0418686000-7f041868a000 r--p 001c0000 08:01 7340200 /lib/x86_64-linux-gnu/libc-2.23.so
7f041868a000-7f041868c000 rw-p 001c4000 08:01 7340200 /lib/x86_64-linux-gnu/libc-2.23.so
7f041868c000-7f0418690000 rw-p 00000000 00:00 0
7f0418690000-7f04186a8000 r-xp 00000000 08:01 7340101 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f04186a8000-7f04188a7000 ---p 00018000 08:01 7340101 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f04188a7000-7f04188a8000 r--p 00017000 08:01 7340101 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f04188a8000-7f04188a9000 rw-p 00018000 08:01 7340101 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f04188a9000-7f04188ad000 rw-p 00000000 00:00 0
7f04188ad000-7f04189b5000 r-xp 00000000 08:01 7340176 /lib/x86_64-linux-gnu/libm-2.23.so
7f04189b5000-7f0418bb4000 ---p 00108000 08:01 7340176 /lib/x86_64-linux-gnu/libm-2.23.so
7f0418bb4000-7f0418bb5000 r--p 00107000 08:01 7340176 /lib/x86_64-linux-gnu/libm-2.23.so
7f0418bb5000-7f0418bb6000 rw-p 00108000 08:01 7340176 /lib/x86_64-linux-gnu/libm-2.23.so
7f0418bb6000-7f0418ef2000 r-xp 00000000 08:01 788003 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.20
7f0418ef2000-7f04190f1000 ---p 0033c000 08:01 788003 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.20
7f04190f1000-7f04190f5000 r--p 0033b000 08:01 788003 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.20
7f04190f5000-7f0419167000 rw-p 0033f000 08:01 788003 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.20
7f0419167000-7f041916d000 rw-p 00000000 00:00 0
7f041916d000-7f0419210000 r-xp 00000000 08:01 787890 /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3
7f0419210000-7f0419410000 ---p 000a3000 08:01 787890 /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3
7f0419410000-7f0419411000 r--p 000a3000 08:01 787890 /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3
7f0419411000-7f0419413000 rw-p 000a4000 08:01 787890 /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3
7f0419413000-7f0419449000 rw-p 00000000 00:00 0
7f0419449000-7f041946f000 r-xp 00000000 08:01 7340173 /lib/x86_64-linux-gnu/ld-2.23.so
7f04194c2000-7f041965a000 r--p 00000000 08:01 788398 /usr/lib/locale/locale-archive
7f041965a000-7f0419663000 rw-p 00000000 00:00 0
7f041966d000-7f041966e000 rw-p 00000000 00:00 0
7f041966e000-7f041966f000 r--p 00025000 08:01 7340173 /lib/x86_64-linux-gnu/ld-2.23.so
7f041966f000-7f0419670000 rw-p 00026000 08:01 7340173 /lib/x86_64-linux-gnu/ld-2.23.so
7f0419670000-7f0419671000 rw-p 00000000 00:00 0
7ffc62213000-7ffc62314000 rw-p 00000000 00:00 0 [stack]
7ffc623cd000-7ffc623cf000 r--p 00000000 00:00 0 [vvar]
7ffc623cf000-7ffc623d1000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
2024-10-16 12:30:27 - FATAL: Spine Interrupted by Abort Signal

[TheWitness]

You likely ran out of connections. Look at the connections logic on the technical support page.

[TheWitness]

Also, for 1000 devices, I would normally use 2 processes and maybe 20 threads.

[kiwildante]

i already sent the screenshot for the connection logic, i set the max connection to 2000 and the min req is 400, so i guess it's not the connection problem,
allright i'll try to change it to 2 process and 20 threads and see if it still produce the segfault

[kiwildante]

i already set the process to 2 and threads to 20, but it makes the segfault occur more often from before

[TheWitness]

What's you log level?

[TheWitness]

What operating system and version?

[kiwildante]

i'm setting it to low right now,
i'm using Ubuntu 16.04.7 LTS

[kiwildante]

in the poller setting i set the invalid data logging to detailed

[TheWitness]

Did you get a core file? Can you do a simple backtrace of it one that provides line numbers and files?

[kiwildante]

i'm sorry, i already tried to get the core files, but it didn't record anything, i'll try it again later

[kiwildante]

i can only provide you with this cacti log backtrace that happens in one of my 100 devices server, it uses 3 process and 3 thread

2024-10-28 11:40:20 - FATAL: Spine Interrupted by Abort Signal -- *** Error in `/usr/local/spine/bin/spine': double free or corruption (!prev): 0x0000000000917260 *** ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x777f5)[0x7f66c85007f5] /lib/x86_64-linux-gnu/libc.so.6(+0x8038a)[0x7f66c850938a] /lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f66c850d58c] /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(+0x56846)[0x7f66c8dcf846] /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(+0x281ba)[0x7f66c8da11ba] /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(+0x2ec11)[0x7f66c8da7c11] /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(mysql_send_query+0x57)[0x7f66c8da4187] /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20(mysql_real_query+0x11)[0x7f66c8da4201] /usr/local/spine/bin/spine[0x4055fb] /usr/local/spine/bin/spine[0x40e565] /usr/local/spine/bin/spine[0x412422] /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f66c885a6ba] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f66c85904dd] ======= Memory map: ======== 00400000-00422000 r-xp 00000000 09:00 25692760 /usr/local/spine/bin/spine 00622000-00623000 r--p 00022000 09:00 25692760 /usr/local/spine/bin/spine 00623000-00624000 rw-p 00023000 09:00 25692760 /usr/local/spine/bin/spine 00624000-00699000 rw-p 00000000 00:00 0 008c5000-0093c000 rw-p 00000000 00:00 0 [heap] 7f66b0000000-7f66b0023000 rw-p 00000000 00:00 0 7f66b0023000-7f66b4000000 ---p 00000000 00:00 0 7f66b8000000-7f66b803a000 rw-p 00000000 00:00 0 7f66b803a000-7f66bc000000 ---p 00000000 00:00 0 7f66bc000000-7f66bc040000 rw-p 00000000 00:00 0 7f66bc040000-7f66c0000000 ---p 00000000 00:00 0 7f66c0000000-7f66c003a000 rw-p 00000000 00:00 0 7f66c003a000-7f66c4000000 ---p 00000000 00:00 0 7f66c5210000-7f66c5211000 ---p 00000000 00:00 0 7f66c5211000-7f66c5a11000 rw-p 00000000 00:00 0 7f66c5a11000-7f66c5a12000 ---p 00000000 00:00 0 7f66c5a12000-7f66c6212000 rw-p 00000000 00:00 0 7f66c6212000-7f66c6213000 ---p 00000000 00:00 0 7f66c6213000-7f66c6a13000 rw-p 00000000 00:00 0 7f66c6a13000-7f66c6a14000 ---p 00000000 00:00 0 7f66c6a14000-7f66c7214000 rw-p 00000000 00:00 0 7f66c7214000-7f66c721f000 r-xp 00000000 09:00 17568645 /lib/x86_64-linux-gnu/libnss_files-2.23.so 7f66c721f000-7f66c741e000 ---p 0000b000 09:00 17568645 /lib/x86_64-linux-gnu/libnss_files-2.23.so 7f66c741e000-7f66c741f000 r--p 0000a000 09:00 17568645 /lib/x86_64-linux-gnu/libnss_files-2.23.so 7f66c741f000-7f66c7420000 rw-p 0000b000 09:00 17568645 /lib/x86_64-linux-gnu/libnss_files-2.23.so 7f66c7420000-7f66c7426000 rw-p 00000000 00:00 0 7f66c7426000-7f66c743c000 r-xp 00000000 09:00 17564175 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f66c743c000-7f66c763b000 ---p 00016000 09:00 17564175 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f66c763b000-7f66c763c000 rw-p 00015000 09:00 17564175 /lib/x86_64-linux-gnu/libgcc_s.so.1 7f66c763c000-7f66c77ae000 r-xp 00000000 09:00 24905766 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21 7f66c77ae000-7f66c79ae000 ---p 00172000 09:00 24905766 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21 7f66c79ae000-7f66c79b8000 r--p 00172000 09:00 24905766 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21 7f66c79b8000-7f66c79ba000 rw-p 0017c000 09:00 24905766 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21 7f66c79ba000-7f66c79be000 rw-p 00000000 00:00 0 7f66c79be000-7f66c7a1c000 r-xp 00000000 09:00 17564254 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f66c7a1c000-7f66c7c1c000 ---p 0005e000 09:00 17564254 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f66c7c1c000-7f66c7c20000 r--p 0005e000 09:00 17564254 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f66c7c20000-7f66c7c27000 rw-p 00062000 09:00 17564254 /lib/x86_64-linux-gnu/libssl.so.1.0.0 7f66c7c27000-7f66c7c40000 r-xp 00000000 09:00 17564128 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f66c7c40000-7f66c7e3f000 ---p 00019000 09:00 17564128 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f66c7e3f000-7f66c7e40000 r--p 00018000 09:00 17564128 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f66c7e40000-7f66c7e41000 rw-p 00019000 09:00 17564128 /lib/x86_64-linux-gnu/libz.so.1.2.8 7f66c7e41000-7f66c7e44000 r-xp 00000000 09:00 17568630 /lib/x86_64-linux-gnu/libdl-2.23.so 7f66c7e44000-7f66c8043000 ---p 00003000 09:00 17568630 /lib/x86_64-linux-gnu/libdl-2.23.so 7f66c8043000-7f66c8044000 r--p 00002000 09:00 17568630 /lib/x86_64-linux-gnu/libdl-2.23.so 7f66c8044000-7f66c8045000 rw-p 00003000 09:00 17568630 /lib/x86_64-linux-gnu/libdl-2.23.so 7f66c8045000-7f66c825f000 r-xp 00000000 09:00 17564159 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f66c825f000-7f66c845e000 ---p 0021a000 09:00 17564159 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f66c845e000-7f66c847a000 r--p 00219000 09:00 17564159 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f66c847a000-7f66c8486000 rw-p 00235000 09:00 17564159 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 7f66c8486000-7f66c8489000 rw-p 00000000 00:00 0 7f66c8489000-7f66c8649000 r-xp 00000000 09:00 17568632 /lib/x86_64-linux-gnu/libc-2.23.so 7f66c8649000-7f66c8849000 ---p 001c0000 09:00 17568632 /lib/x86_64-linux-gnu/libc-2.23.so 7f66c8849000-7f66c884d000 r--p 001c0000 09:00 17568632 /lib/x86_64-linux-gnu/libc-2.23.so 7f66c884d000-7f66c884f000 rw-p 001c4000 09:00 17568632 /lib/x86_64-linux-gnu/libc-2.23.so 7f66c884f000-7f66c8853000 rw-p 00000000 00:00 0 7f66c8853000-7f66c886b000 r-xp 00000000 09:00 17568633 /lib/x86_64-linux-gnu/libpthread-2.23.so 7f66c886b000-7f66c8a6a000 ---p 00018000 09:00 17568633 /lib/x86_64-linux-gnu/libpthread-2.23.so 7f66c8a6a000-7f66c8a6b000 r--p 00017000 09:00 17568633 /lib/x86_64-linux-gnu/libpthread-2.23.so 7f66c8a6b000-7f66c8a6c000 rw-p 00018000 09:00 17568633 /lib/x86_64-linux-gnu/libpthread-2.23.so 7f66c8a6c000-7f66c8a70000 rw-p 00000000 00:00 0 7f66c8a70000-7f66c8b78000 r-xp 00000000 09:00 17568636 /lib/x86_64-linux-gnu/libm-2.23.so 7f66c8b78000-7f66c8d77000 ---p 00108000 09:00 17568636 /lib/x86_64-linux-gnu/libm-2.23.so 7f66c8d77000-7f66c8d78000 r--p 00107000 09:00 17568636 /lib/x86_64-linux-gnu/libm-2.23.so 7f66c8d78000-7f66c8d79000 rw-p 00108000 09:00 17568636 /lib/x86_64-linux-gnu/libm-2.23.so 7f66c8d79000-7f66c90b5000 r-xp 00000000 09:00 24912247 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.20 7f66c90b5000-7f66c92b4000 ---p 0033c000 09:00 24912247 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.20 7f66c92b4000-7f66c92b8000 r--p 0033b000 09:00 24912247 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.20 7f66c92b8000-7f66c932a000 rw-p 0033f000 09:00 24912247 /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20.3.20 7f66c932a000-7f66c9330000 rw-p 00000000 00:00 0 7f66c9330000-7f66c93d3000 r-xp 00000000 09:00 24906006 /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3 7f66c93d3000-7f66c95d3000 ---p 000a3000 09:00 24906006 /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3 7f66c95d3000-7f66c95d4000 r--p 000a3000 09:00 24906006 /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3 7f66c95d4000-7f66c95d6000 rw-p 000a4000 09:00 24906006 /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3 7f66c95d6000-7f66c960c000 rw-p 00000000 00:00 0 7f66c960c000-7f66c9632000 r-xp 00000000 09:00 17568643 /lib/x86_64-linux-gnu/ld-2.23.so 7f66c9686000-7f66c981e000 r--p 00000000 09:00 24908536 /usr/lib/locale/locale-archive 7f66c981e000-7f66c9827000 rw-p 00000000 00:00 0 7f66c9830000-7f66c9831000 rw-p 00000000 00:00 0 7f66c9831000-7f66c9832000 r--p 00025000 09:00 17568643 /lib/x86_64-linux-gnu/ld-2.23.so 7f66c9832000-7f66c9833000 rw-p 00026000 09:00 17568643 /lib/x86_64-linux-gnu/ld-2.23.so 7f66c9833000-7f66c9834000 rw-p 00000000 00:00 0 7ffe15453000-7ffe15552000 rw-p 00000000 00:00 0 [stack] 7ffe155f7000-7ffe155fa000 r--p 00000000 00:00 0 [vvar] 7ffe155fa000-7ffe155fc000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

[TheWitness]

Where are you saving core files? Can you enable the capture of the core file and then backtrace the core file? If now. Go to your build directory, copy the spine.conf file into that build location. And then run spine this way:

cd /spine/build/directory
gdb ./spine
run -V 3 -S
# wait for crash
bt
# capture output
quit

[TheWitness]

@kiwildante, any update?