 Double free or corruption (fasttop) in libvncclient.so.1 (LibVNCServer 0.9.15, OpenSSL 1.1.1w)

If you'd like to put out an incentive for fixing this bug, you can do so at https://issuehunt.io/r/LibVNC/libvncserver

Describe the bug

A double free or corruption error occurs in `libvncclient.so.1` when using LibVNCServer built with a custom OpenSSL 1.1.1w installation.  This leads to the client disconnecting shortly after connecting.

To Reproduce

Build and install OpenSSL 1.1.1w from source to a custom location (/usr/local/openssl-1.1.1w).
Build LibVNCServer 0.9.15 from source using the provided RPM spec file, which configures CMake to use the custom OpenSSL installation.
Install the resulting LibVNCServer RPM.
Use a LibVNCClient (e.g., through guacd) to connect to a VNC server.
Observe that the connection drops after a short period.
Expected Behavior

The LibVNCClient should maintain a stable connection to the VNC server.

Logs/Backtraces

![Image](https://github.com/user-attachments/assets/a51a65cb-660d-41d3-84bc-6965d957fc60)

and the log text:
``` log
guacd[5448]: INFO:	User "@b72076de-e069-4b82-9145-738c74bbbc17" joined connection "$e7a6be12-0511-4bcf-91d5-e5019d693dc5" (1 users now present)
guacd[5448]: DEBUG:	Client is using protocol version "VERSION_1_5_0"
guacd[5448]: INFO:	min gap size: 20971520
guacd[5448]: INFO:	rec session:/opt/lsblj/records/videos/202502/20//192.168.40.160--20250220164536881
guacd[5448]: DEBUG:	guac_vnc_log_clipboard log
guacd[5448]: TRACE:	Received nop instruction
guacd[5448]: TRACE:	Received nop instruction
guacd[5448]: DEBUG:	guac_vnc_log_clipboard log
*** Error in `/opt/lsblj/server/sbin/guacd': double free or corruption (fasttop): 0x000000000261fe20 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81329)[0x7fa010705329]
/opt/libvnc/lib64/libvncclient.so.1(HandleCursorShape+0x686)[0x7fa0083c300b]
/opt/libvnc/lib64/libvncclient.so.1(HandleRFBServerMessage+0x32c)[0x7fa0083c99fb]
/opt/lsblj/server/lib/libguac-client-vnc.so(guac_vnc_client_thread+0x3ac)[0x7fa0085fa8ac]
/lib64/libpthread.so.0(+0x7ea5)[0x7fa01218eea5]
/lib64/libc.so.6(clone+0x6d)[0x7fa0107829fd]
======= Memory map: ========
00400000-00407000 r-xp 00000000 fd:00 134489901                          /opt/lsblj/server/sbin/guacd
00606000-00607000 r--p 00006000 fd:00 134489901                          /opt/lsblj/server/sbin/guacd
00607000-00608000 rw-p 00007000 fd:00 134489901                          /opt/lsblj/server/sbin/guacd
01de4000-02603000 rw-p 00000000 00:00 0                                  [heap]
02603000-026a0000 rw-p 00000000 00:00 0                                  [heap]
7f9fe8000000-7f9fe8021000 rw-p 00000000 00:00 0 
7f9fe8021000-7f9fec000000 ---p 00000000 00:00 0 
7f9ff0000000-7f9ff0025000 rw-p 00000000 00:00 0 
7f9ff0025000-7f9ff4000000 ---p 00000000 00:00 0 
7f9ff7201000-7f9ff7803000 rw-p 00000000 00:00 0 
7f9ff7803000-7f9ff7804000 ---p 00000000 00:00 0 
7f9ff7804000-7f9ff8004000 rw-p 00000000 00:00 0 
7f9ff8004000-7f9ff8005000 ---p 00000000 00:00 0 
7f9ff8005000-7f9ff8c83000 rw-p 00000000 00:00 0 
7f9ff8c83000-7f9ff8cc6000 r-xp 00000000 fd:00 362270                     /usr/lib64/libFLAC.so.8.3.0
7f9ff8cc6000-7f9ff8ec6000 ---p 00043000 fd:00 362270                     /usr/lib64/libFLAC.so.8.3.0
7f9ff8ec6000-7f9ff8ec7000 r--p 00043000 fd:00 362270                     /usr/lib64/libFLAC.so.8.3.0
7f9ff8ec7000-7f9ff8ec8000 rw-p 00044000 fd:00 362270                     /usr/lib64/libFLAC.so.8.3.0
7f9ff8ec8000-7f9ff8ed3000 r-xp 00000000 fd:00 44339                      /usr/lib64/libgsm.so.1.0.12
7f9ff8ed3000-7f9ff90d2000 ---p 0000b000 fd:00 44339                      /usr/lib64/libgsm.so.1.0.12
7f9ff90d2000-7f9ff90d3000 r--p 0000a000 fd:00 44339                      /usr/lib64/libgsm.so.1.0.12
7f9ff90d3000-7f9ff90d4000 rw-p 0000b000 fd:00 44339                      /usr/lib64/libgsm.so.1.0.12
7f9ff90d4000-7f9ff90eb000 r-xp 00000000 fd:00 11613                      /usr/lib64/libnsl-2.17.so
7f9ff90eb000-7f9ff92ea000 ---p 00017000 fd:00 11613                      /usr/lib64/libnsl-2.17.so
7f9ff92ea000-7f9ff92eb000 r--p 00016000 fd:00 11613                      /usr/lib64/libnsl-2.17.so
7f9ff92eb000-7f9ff92ec000 rw-p 00017000 fd:00 11613                      /usr/lib64/libnsl-2.17.so
7f9ff92ec000-7f9ff92ee000 rw-p 00000000 00:00 0 
7f9ff92ee000-7f9ff92fd000 r-xp 00000000 fd:00 44212                      /usr/lib64/libXi.so.6.1.0
7f9ff92fd000-7f9ff94fc000 ---p 0000f000 fd:00 44212                      /usr/lib64/libXi.so.6.1.0
7f9ff94fc000-7f9ff94fd000 r--p 0000e000 fd:00 44212                      /usr/lib64/libXi.so.6.1.0
7f9ff94fd000-7f9ff94fe000 rw-p 0000f000 fd:00 44212                      /usr/lib64/libXi.so.6.1.0
7f9ff94fe000-7f9ff9515000 r-xp 00000000 fd:00 13689                      /usr/lib64/libelf-0.176.so
7f9ff9515000-7f9ff9714000 ---p 00017000 fd:00 13689                      /usr/lib64/libelf-0.176.so
7f9ff9714000-7f9ff9715000 r--p 00016000 fd:00 13689                      /usr/lib64/libelf-0.176.so
7f9ff9715000-7f9ff9716000 rw-p 00017000 fd:00 13689                      /usr/lib64/libelf-0.176.so
7f9ff9716000-7f9ff971a000 r-xp 00000000 fd:00 13714                      /usr/lib64/libattr.so.1.1.0
7f9ff971a000-7f9ff9919000 ---p 00004000 fd:00 13714                      /usr/lib64/libattr.so.1.1.0
7f9ff9919000-7f9ff991a000 r--p 00003000 fd:00 13714                      /usr/lib64/libattr.so.1.1.0
7f9ff991a000-7f9ff991b000 rw-p 00004000 fd:00 13714                      /usr/lib64/libattr.so.1.1.0
7f9ff991b000-7f9ff9920000 r-xp 00000000 fd:00 1107336                    /usr/lib64/libasyncns.so.0.3.1
7f9ff9920000-7f9ff9b1f000 ---p 00005000 fd:00 1107336                    /usr/lib64/libasyncns.so.0.3.1
7f9ff9b1f000-7f9ff9b20000 r--p 00004000 fd:00 1107336                    /usr/lib64/libasyncns.so.0.3.1
7f9ff9b20000-7f9ff9b21000 rw-p 00005000 fd:00 1107336                    /usr/lib64/libasyncns.so.0.3.1
7f9ff9b21000-7f9ff9b79000 r-xp 00000000 fd:00 362272                     /usr/lib64/libsndfile.so.1.0.25
7f9ff9b79000-7f9ff9d79000 ---p 00058000 fd:00 362272                     /usr/lib64/libsndfile.so.1.0.25
7f9ff9d79000-7f9ff9d7b000 r--p 00058000 fd:00 362272                     /usr/lib64/libsndfile.so.1.0.25
7f9ff9d7b000-7f9ff9d7c000 rw-p 0005a000 fd:00 362272                     /usr/lib64/libsndfile.so.1.0.25
7f9ff9d7c000-7f9ff9d80000 rw-p 00000000 00:00 0 
7f9ff9d80000-7f9ff9d89000 r-xp 00000000 fd:00 13728                      /usr/lib64/libwrap.so.0.7.6
7f9ff9d89000-7f9ff9f88000 ---p 00009000 fd:00 13728                      /usr/lib64/libwrap.so.0.7.6
7f9ff9f88000-7f9ff9f89000 r--p 00008000 fd:00 13728                      /usr/lib64/libwrap.so.0.7.6
7f9ff9f89000-7f9ff9f8a000 rw-p 00009000 fd:00 13728                      /usr/lib64/libwrap.so.0.7.6
7f9ff9f8a000-7f9ff9f8b000 rw-p 00000000 00:00 0 
7f9ff9f8b000-7f9ff9f90000 r-xp 00000000 fd:00 44216                      /usr/lib64/libXtst.so.6.1.0
7f9ff9f90000-7f9ffa18f000 ---p 00005000 fd:00 44216                      /usr/lib64/libXtst.so.6.1.0
7f9ffa18f000-7f9ffa190000 r--p 00004000 fd:00 44216                      /usr/lib64/libXtst.so.6.1.0
7f9ffa190000-7f9ffa191000 rw-p 00005000 fd:00 44216                      /usr/lib64/libXtst.so.6.1.0
7f9ffa191000-7f9ffa198000 r-xp 00000000 fd:00 42347                      /usr/lib64/libSM.so.6.0.1
7f9ffa198000-7f9ffa397000 ---p 00007000 fd:00 42347                      /usr/lib64/libSM.so.6.0.1
7f9ffa397000-7f9ffa398000 r--p 00006000 fd:00 42347                      /usr/lib64/libSM.so.6.0.1
7f9ffa398000-7f9ffa399000 rw-p 00007000 fd:00 42347                      /usr/lib64/libSM.so.6.0.1
7f9ffa399000-7f9ffa3b0000 r-xp 00000000 fd:00 13747                      /usr/lib64/libICE.so.6.3.0
7f9ffa3b0000-7f9ffa5af000 ---p 00017000 fd:00 13747                      /usr/lib64/libICE.so.6.3.0
7f9ffa5af000-7f9ffa5b0000 r--p 00016000 fd:00 13747                      /usr/lib64/libICE.so.6.3.0
7f9ffa5b0000-7f9ffa5b1000 rw-p 00017000 fd:00 13747                      /usr/lib64/libICE.so.6.3.0
7f9ffa5b1000-7f9ffa5b5000 rw-p 00000000 00:00 0 
7f9ffa5b5000-7f9ffa5b6000 r-xp 00000000 fd:00 44199                      /usr/lib64/libX11-xcb.so.1.0.0guacd[5137]: INFO:	Connection "$e7a6be12-0511-4bcf-91d5-e5019d693dc5" removed.


```

Your environment (please complete the following information):

OS and version: CentOS 7.9
Compiler and version: 4.8.5
LibVNCServer version: 0.9.15
OpenSSL version: 1.1.1w (custom build)
Additional context

OpenSSL 1.1.1w is installed in a non-standard location: /usr/local/openssl-1.1.1w.

The LibVNCServer RPM spec file is configured to use this custom OpenSSL installation via CMake flags:
``` spec
%prep
%setup -q -n libvncserver-LibVNCServer-%{version}

%build
# 使用 CMake 构建，添加你的编译选项
mkdir -p build
cd build
cmake -DCMAKE_C_FLAGS="-I/usr/local/openssl1.1.1w/include" \
      -DCMAKE_CXX_FLAGS="-I/usr/local/openssl1.1.1w/include" \
      -DCMAKE_EXE_LINKER_FLAGS="-L/usr/local/openssl1.1.1w/lib" \
      -DCMAKE_SHARED_LINKER_FLAGS="-L/usr/local/openssl1.1.1w/lib" \
      -DOPENSSL_ROOT_DIR=/usr/local/openssl1.1.1w \
      -DOPENSSL_LIBRARIES=/usr/local/openssl1.1.1w/lib \
      -DOPENSSL_INCLUDE_DIR=/usr/local/openssl1.1.1w/include \
      -DCMAKE_INSTALL_PREFIX=/opt/libvnc \
      -DWITH_GNUTLS=OFF \
      -DWITH_OPENSSL=ON \
      -DWITH_FFMPEG=OFF \
      -DWITH_GCRYPT=OFF \
      ..
make %{?_smp_mflags}

%install
# 安装到指定目录
rm -rf $RPM_BUILD_ROOT
cd build
make install DESTDIR=$RPM_BUILD_ROOT

%files
/opt/libvnc/include
/opt/libvnc/lib64
```

- The issue is observed when using guacd to establish the VNC connection, but it might also affect other LibVNCClient implementations.
- Crucially, when reverting to LibVNCServer version 0.9.14, the issue does not occur. This suggests the problem was introduced in version 0.9.15


I'm sorry, I'm not a C language developer, so I can't provide better information




Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Double free or corruption (fasttop) in libvncclient.so.1 (LibVNCServer 0.9.15, OpenSSL 1.1.1w) #652

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Uh oh!

Double free or corruption (fasttop) in libvncclient.so.1 (LibVNCServer 0.9.15, OpenSSL 1.1.1w) #652

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions