[VPP-2121] sw_interface_add_del_address: avf process node failed to reply in 5 seconds

[vvalderrv]

Description

An example core [0] from CSIT test on VPP debug (+asan) image. So far this issue is only seen in IPsec policy tests (which do not usually use AVF plugin in trending).

The issue reminds me of similar failure (old, logs no longer available) that got fixed by [1] (but my attempt at analogous fix [2] did not help).

[0] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/vpp-csit-verify-perf-master-ubuntu2204-x86_64-3n-snr/49/csit_current/0/log.html.gz#s1-s1-s1-s1-s1-t1-k3-k4-k1

[1] 40024: dev: mark API handlers as thread safe | https://gerrit.fd.io/r/c/vpp/+/40024

[2] https://gerrit.fd.io/r/c/vpp/+/41343

Assignee

Unassigned

Reporter

Vratko Polak

Comments

• vrpolak (Tue, 19 Nov 2024 12:53:44 +0000): As expected, as this got fixed only in plugins/dev_iavf, the CSIT tests agains plugins/avf are still failing [5] in rls2410.

[5] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-coverage-2410-3n-icx/72/log.html.gz#s1-s1-s1-s1-s4-t1-k3-k4-k1

• vrpolak (Fri, 23 Aug 2024 15:03:33 +0000): Proposed fix: [4].

[4] 41343: avf: avoid AVF deadlock | https://gerrit.fd.io/r/c/vpp/+/41343

• vrpolak (Wed, 21 Aug 2024 14:08:04 +0000): After some digging, now I am convinced this is the same issue I have been trying to fix [3] two years ago.

[3] 37083: avf: tolerate socket events in avf_process_request | https://gerrit.fd.io/r/c/vpp/+/37083

• vrpolak (Wed, 7 Aug 2024 11:10:57 +0000): So far this seems quite deterministic, and the trigger happens be avoided on CSIT side by waiting for API reply before sending new message. Not sure if the difference is casual or just a timing accident.

Original issue: https://jira.fd.io/browse/VPP-2121

[vvalderrv]

As expected, as this got fixed only in plugins/dev_iavf, the CSIT tests agains plugins/avf are still failing [5] in rls2410.

[5] https://s3-logs.fd.io/vex-yul-rot-jenkins-1/csit-vpp-perf-report-coverage-2410-3n-icx/72/log.html.gz#s1-s1-s1-s1-s4-t1-k3-k4-k1

[vvalderrv]

Proposed fix: [4].

[4] 41343: avf: avoid AVF deadlock | https://gerrit.fd.io/r/c/vpp/+/41343

[vvalderrv]

After some digging, now I am convinced this is the same issue I have been trying to fix [3] two years ago.

[3] 37083: avf: tolerate socket events in avf_process_request | https://gerrit.fd.io/r/c/vpp/+/37083

[vvalderrv]

So far this seems quite deterministic, and the trigger happens be avoided on CSIT side by waiting for API reply before sending new message. Not sure if the difference is casual or just a timing accident.

[vvalderrv]

Jira Date Created: Wed, 31 Jul 2024 11:49:24 +0000
Jira Date Updated: Tue, 19 Nov 2024 12:53:44 +0000