[CSIT-1964] 3nb-spr, 3n-snr: Wireguardhw tests are likely to crash

[vvalderrv]

Description

This is similar to CSIT-1886, but there are notable differences. CSIT-1886 affects mainly large scale tests, this also affects small scale tests (but hw only). Some failures are similar to CSIT-1886 (NDR fails due to some tunnels reporting peer error), but some failures suggest VPP crash (which does not happen with CSIT-1886) by failing VPP PID check, although no actual core file is seen so far.

More investigation is needed to prove this is really an independent failure, not just some infrequent known issue occurring more frequently in 3nb-spr wireguardhw tests.

[0] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2406-3nb-spr/67/log.html.gz#s1-s1-s1-s3-s5

Assignee

Vratko Polak

Reporter

Vratko Polak

Comments

• vrpolak (Fri, 15 Nov 2024 10:20:23 +0000): In another run, VPP did not crash, but stats [2] show "Peer error" on main thread and "Keypair error" on workers, so I suspect the cause is the same, just a different part of memory got corrupted.

[2] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2410-3n-snr/14/log.html.gz#s1-s1-s1-s3-s2-t6-k3-k7-k1-k1-k1-k8-k14-k1-k1-k1-k1

• vrpolak (Fri, 15 Nov 2024 10:13:30 +0000):

  Still present in rls2410, but this time I see core [1].

Here is the relevant part:

#7

#8 0x00007fff34fd9e2a in EVP_PKEY_get_raw_public_key () from /lib/x86_64-linux-gnu/libcrypto.so.3

#9 0x00007fff2e43607b in curve25519_gen_public () at /w/workspace/vpp-merge-2410-ubuntu2204-x86_64/src/plugins/wireguard/wireguard_key.c:74

#10 0x00007fff2e4379dc in noise_create_initiation () at /w/workspace/vpp-merge-2410-ubuntu2204-x86_64/src/plugins/wireguard/wireguard_noise.c:140

#11 0x00007fff2e439d94 in wg_send_handshake (vm=vm@entry=0x7fff35e00700, peer=0x7fff3a621af8, is_retry=) at /w/workspace/vpp-merge-2410-ubuntu2204-x86_64/src/plugins/wireguard/wireguard_send.c:175

#12 0x00007fff2e43f968 in wg_expired_new_handshake (vm=0x7fff35e00700, peer=) at /w/workspace/vpp-merge-2410-ubuntu2204-x86_64/src/plugins/wireguard/wireguard_timer.c:187

[1] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2410-3n-snr/13/log.html.gz#s1-s1-s1-s3-s2-t1-k3-k4-k1

Original issue: https://jira.fd.io/browse/CSIT-1964

[vvalderrv]

In another run, VPP did not crash, but stats [2] show "Peer error" on main thread and "Keypair error" on workers, so I suspect the cause is the same, just a different part of memory got corrupted.

[2] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2410-3n-snr/14/log.html.gz#s1-s1-s1-s3-s2-t6-k3-k7-k1-k1-k1-k8-k14-k1-k1-k1-k1

[vvalderrv]

Still present in rls2410, but this time I see core [1].
Here is the relevant part:

#7 <signal handler called>
#8 0x00007fff34fd9e2a in EVP_PKEY_get_raw_public_key () from /lib/x86_64-linux-gnu/libcrypto.so.3
#9 0x00007fff2e43607b in curve25519_gen_public (<snip>) at /w/workspace/vpp-merge-2410-ubuntu2204-x86_64/src/plugins/wireguard/wireguard_key.c:74
#10 0x00007fff2e4379dc in noise_create_initiation (<snip>) at /w/workspace/vpp-merge-2410-ubuntu2204-x86_64/src/plugins/wireguard/wireguard_noise.c:140
#11 0x00007fff2e439d94 in wg_send_handshake (vm=vm@entry=0x7fff35e00700, peer=0x7fff3a621af8, is_retry=<optimized out>) at /w/workspace/vpp-merge-2410-ubuntu2204-x86_64/src/plugins/wireguard/wireguard_send.c:175
#12 0x00007fff2e43f968 in wg_expired_new_handshake (vm=0x7fff35e00700, peer=<optimized out>) at /w/workspace/vpp-merge-2410-ubuntu2204-x86_64/src/plugins/wireguard/wireguard_timer.c:187

[1] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2410-3n-snr/13/log.html.gz#s1-s1-s1-s3-s2-t1-k3-k4-k1

[vrpolakatcisco]

Tests are still failing [3] in rls2502, although mostly without core.

[3] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2502-3n-snr/35/log.html.gz#s1-s1-s1-s3-s10-t2-k3-k4-k1