[CSIT-1960] 2n-zn2: AVF xxv710 sometimes loses one direction of traffic, mostly with geneve

[vvalderrv]

Description

This was previously mentioned as a secondary symptom of CSIT-1800. But that ticket is mostly about VPP crashes (not seen recently), while this symptom (NDRPDR failure but no crash) still can [0] happen, albeit rarely. Interestingly, all tests within the suite failed the same way.

No clear relation to number of tunnels is visible yet.

More investigation needed for proper VPP bug-report.

[0] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2406-2n-zn2/54/log.html.gz#s1-s1-s1-s3-s1-t1-k2-k9-k14

Assignee

Vratko Polak

Reporter

Vratko Polak

Comments

• vrpolak (Wed, 13 Nov 2024 13:27:04 +0000): > "l3 mac mismatch" errors leading to punt.

First time I see this symptom outside geneve (still on zen2). It happened [3] in ip4base test, so perhaps it is possible in all AVF xxv710 tests on zn2. Changing title.

[3] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2410-2n-zn2/37/log.html.gz#s1-s1-s1-s2-s10-t1-k3-k7-k1-k1-k1-k8-k14-k1-k1-k1-k1

• vrpolak (Wed, 13 Nov 2024 12:52:57 +0000): In rls2410, this happened [2] for 16Tun Geneve suite.

[2] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2410-2n-zn2/22/log.html.gz#s1-s1-s1-s3-s1-t1-k2-k9-k14

• vrpolak (Wed, 14 Aug 2024 12:04:47 +0000): Very rare. Last time seen in trending as MRR regression (also for 16tnl). Telemetry [1] shows "l3 mac mismatch" errors leading to punt.

[1] https://logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/955/log.html.gz#s1-s1-s1-s3-s1-t1-k2-k9-k9-k14-k1-k1-k1-k1

Original issue: https://jira.fd.io/browse/CSIT-1960

[vvalderrv]

> "l3 mac mismatch" errors leading to punt.

First time I see this symptom outside geneve (still on zen2). It happened [3] in ip4base test, so perhaps it is possible in all AVF xxv710 tests on zn2. Changing title.

[3] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2410-2n-zn2/37/log.html.gz#s1-s1-s1-s2-s10-t1-k3-k7-k1-k1-k1-k8-k14-k1-k1-k1-k1

[vvalderrv]

In rls2410, this happened [2] for 16Tun Geneve suite.

[2] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-iterative-2410-2n-zn2/22/log.html.gz#s1-s1-s1-s3-s1-t1-k2-k9-k14

[vvalderrv]

Very rare. Last time seen in trending as MRR regression (also for 16tnl). Telemetry [1] shows "l3 mac mismatch" errors leading to punt.

[1] https://logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-mrr-daily-master-2n-zn2/955/log.html.gz#s1-s1-s1-s3-s1-t1-k2-k9-k9-k14-k1-k1-k1-k1

[vrpolakatcisco]

Looking at trending [4], I can conclude that this is either fixed, or at least the frequency is greatly reduced. I will reopen if I see this happening again.

[4] https://csit.fd.io/trending/#eNrtltuKwjAQhp-me7MMbMbUXu3Fqu8hsR1toU1DktbD05tKZczCKrugLOJFjv_PzOSDhDjfWlo6qj-TdJZkswSzqghdMvl6D0NvDKCGg0bAdENoxG7XZ-IDVL-GykiYyhWIHMiXwyq0lXIEjbVDDJwPMYrORwFZMeWelZ_SsF1ZUuwPuVjy5C6SxNWwa21VQ646EFtD_azngQVLIo_j-725UMcjZouT42_g8DnB4d3ByecEJ-8Jbuk7ral2367sOHaF2ZCmnkBMg_HRXM_F3eB7vdh_edEj7PjC_qBnIsIuX9h__8ikizfd2ub0LwhzV7Zb8LZStRu3jh8DSXo