[CRASH] Compression module memory corruption #2602
Description
OpenSIPS version you are running
version: opensips 3.1.3 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
main.c compiled on 19:32:34 Aug 12 2021 with gcc 8
This is with nightly 3.1 (right before 3.1.4 was tagged). I have been chasing a segfault for quite some time now in our production environment and haven't been able to reproduce in our lab / dev environments. The crash is very intermittent (a month between crashes) but always triggers a similar set of logs:
[120] CRITICAL:core:build_res_buf_from_sip_res:
>>> len mismatch : calculated 541, written 569
It seems you have hit a programming bug.
Please help us make OpenSIPS better by reporting it at https://github.com/OpenSIPS/opensips/issues
Jun 24 14:39:39 [117] CRITICAL:core:fm_status: different free frag. count: 10!=9 for hash 33
Jun 24 14:39:39 [117] CRITICAL:core:fm_status: different free frag. count: 0!=1 for hash 54
Jun 24 14:39:39 [118] CRITICAL:core:fm_status: different free frag. count: 4!=3 for hash 33
Jun 24 14:39:39 [118] CRITICAL:core:fm_status: different free frag. count: 1!=2 for hash 53
Jun 24 14:39:39 [116] CRITICAL:core:fm_status: different free frag. count: 4!=3 for hash 33
Jun 24 14:39:39 [116] CRITICAL:core:fm_status: different free frag. count: 42!=43 for hash 49
The logs seem to indicate some memory corruption, so I ran with Q_MALLOC + DBG and within a few calls OpenSIPS crashed with SIGABRT and seems the allocator detected memory issue with the compression module:
CRITICAL:core:qm_debug_frag: qm_*: prev. fragm. tail overwritten(c0c0c0c0c0c0c00a, abcdefedabcdefed)[0x7f75c2e64068:0x7f75c2e64098] (wrap_realloc, compression_helpers.c:374)!
We only use the mc_compact method, so it seems that the mc_compact_cb function is likely the culprit.
Here is the core dump that was produced (purged of sensitive info). I can provide the full dump if needed via email.
Let me know if you need any further info!
To Reproduce
The crash is random and I haven't been able to reproduce.
OS/environment information
- Operating System: debian 10
- OpenSIPS installation: build from nightly 3.1
- other relevant information: