[VPP-2118] 3n spr: Unusable performance of ipsec tests with SHA_256_128

[vvalderrv]

Description

In contrast, tests with SHA_512_256 are not affected.

The impact depends on testbed. Performance is halved [3] on 3n-icx, on 3nb-spr the performance is so poor PDR would [2] be below 9kpps.

According to bisect run [1], the cause is 40047 [0].

[0] 40047: crypto-openssl: refactor openssl API usage | https://gerrit.fd.io/r/c/vpp/+/40047

[1] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/vpp-csit-bisect-master-ubuntu2204-x86_64-3n-icx/11/

[2] https://logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-ndrpdr-weekly-master-3nb-spr/58/log.html.gz#s1-s1-s1-s1-s16-t1-k2-k10-k14

[3] https://csit.fd.io/comparisons/#eNqNkEFuwyAQRU_jbCIqICbOJosm3laqrF7AhXGEYmM6YKvtImcv2E6pqy4qIcGf_4b5GgctSA_qmBWnjBcIDSAYCeGd7R63c9WBX_QLDsE6b6di7NS9WaFquKPhGq2NYuaDMwImE1vHc7onPH-gnGD4rXaQcG0arBO9M0TLd8IovQC3DA6MyjeirLqmFtnjd24Rw7BzlKJc_AY7pz9_IYIdTivK-w-7Zp6qKg15LqsFL6ZzH93ZGv-xtUDVHfifi0iLmZGxbgf4e097gpLfKLkIkYuCFkLdXlke2SnMxvTYHWNoUW76wbca0C36CwfggEQ

Assignee

Unassigned

Reporter

Vratko Polak

Comments

• vrpolak (Mon, 16 Dec 2024 12:26:47 +0000): And here is "show run" [7] from a recent run.

[7] https://logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3nb-spr/314/log.html.gz#s1-s1-s1-s1-s22-t1-k2-k10-k9-k10-k1-k1-k1-k12

• vrpolak (Mon, 16 Dec 2024 12:24:48 +0000): Trending link [6] to show MRR performance for ethip4ipsec10000tnlsw-udp-ar-ip4base-int-aes256cbc-hmac256sha (1c 1518B, dpdk plugin).

[6] https://csit.fd.io/trending/#eNrllk1ugzAQhU9DN9VI2MQlmy6a5h6VMdNilR_H4yTQ09chSCYSrLqxCguweCP703swNrnO4gdh_ZqIQ5IfEp7r0t-S7O3ZPy7GQNaCrB3s0i_khvV1zlIoTfkNyg7GdcAE2xfAFKCrtNlpQ6hY6i_X1nSFc2lAWvBKIQlBtw4kEhcvqlBQNVL5IVUSGmtv6_L327rl2T1ABMVUQ1DW0EK5tChD_Z04qA5pts6f-MOkn1Y2SPoHw8yjRaFCec9nonoEcoOZqZMt-XGsWAtIqx4882gD7lmqThFFtAK3xZBK4GLygWcqrowW2DYXEbV25gKP6jdaZttcRI4q303uDV_wmLaiJbKtxSOBjP9Mp26v-vyC0kFT9yKKjNbxthZUMToR65lhle5fxiSOT21nm_EI7sdUdVdwVsuaple_cO9ZVg

• vrpolak (Fri, 15 Nov 2024 12:33:04 +0000): Abysmal performace [5] leading to test failures is still present in rls2410.

[5] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2410-3na-spr/43/log.html.gz#s1-s1-s1-s1-s3-t1-k3-k7-k1-k1-k1-k8-k10-k1-k1-k1-k12

• vrpolak (Mon, 29 Jul 2024 12:24:10 +0000): I opened VPP-2120 specifically for test failures (as opposed to just poor performance), but on second though there is no real need to have two tickets for the same issue. I will just update title of this ticket and close VPP-2120 as a duplicate.

Example failure [4], from rls2406.

[4] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2406-3na-spr/57/log.html.gz#s1-s1-s1-s1-s3-t1-k3-k7-k1-k1-k1-k8-k10-k1-k1-k1-k12

Original issue: https://jira.fd.io/browse/VPP-2118

[vvalderrv]

And here is "show run" [7] from a recent run.

[7] https://logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-3nb-spr/314/log.html.gz#s1-s1-s1-s1-s22-t1-k2-k10-k9-k10-k1-k1-k1-k12

[vvalderrv]

Trending link [6] to show MRR performance for ethip4ipsec10000tnlsw-udp-ar-ip4base-int-aes256cbc-hmac256sha (1c 1518B, dpdk plugin).

[6] https://csit.fd.io/trending/#eNrllk1ugzAQhU9DN9VI2MQlmy6a5h6VMdNilR_H4yTQ09chSCYSrLqxCguweCP703swNrnO4gdh_ZqIQ5IfEp7r0t-S7O3ZPy7GQNaCrB3s0i_khvV1zlIoTfkNyg7GdcAE2xfAFKCrtNlpQ6hY6i_X1nSFc2lAWvBKIQlBtw4kEhcvqlBQNVL5IVUSGmtv6_L327rl2T1ABMVUQ1DW0EK5tChD_Z04qA5pts6f-MOkn1Y2SPoHw8yjRaFCec9nonoEcoOZqZMt-XGsWAtIqx4882gD7lmqThFFtAK3xZBK4GLygWcqrowW2DYXEbV25gKP6jdaZttcRI4q303uDV_wmLaiJbKtxSOBjP9Mp26v-vyC0kFT9yKKjNbxthZUMToR65lhle5fxiSOT21nm_EI7sdUdVdwVsuaple_cO9ZVg

[vvalderrv]

Abysmal performace [5] leading to test failures is still present in rls2410.

[5] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2410-3na-spr/43/log.html.gz#s1-s1-s1-s1-s3-t1-k3-k7-k1-k1-k1-k8-k10-k1-k1-k1-k12

[vvalderrv]

I opened VPP-2120 specifically for test failures (as opposed to just poor performance), but on second though there is no real need to have two tickets for the same issue. I will just update title of this ticket and close VPP-2120 as a duplicate.

Example failure [4], from rls2406.

[4] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2406-3na-spr/57/log.html.gz#s1-s1-s1-s1-s3-t1-k3-k7-k1-k1-k1-k8-k10-k1-k1-k1-k12

[vrpolakatcisco]

Welcome to github.
In fd.io lab we now also have Emerald Rapids testbeds. 3n-emr is having some infra issues, butsing ad-hoc unidirectional ipsec suite (encryption only) I can confirm [8] that the performance issue is also present on 2n-emr.
I realized that the graph node variants used on SPR and EMR are probably the same as on ICX (unless there are some conditional paths I do not see), so this is probably not an issue of VPP source nor compiler, but directly of Intel CPUs, because .02 instructions per cycle (without any big cache misses) is not an acceptable CPU performance.

[8] https://logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-verify-master-2n-emr/7/log.html.gz#s1-s1-s1-s1-s1-t1-k2-k10-k9-k10-k1-k1-k1-k14

[vvalderrv]

Jira Date Created: Thu, 27 Jun 2024 14:05:58 +0000
Jira Date Updated: Mon, 16 Dec 2024 12:26:47 +0000