Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[VPP-1398] VPP crashes when deleting tapv2 interface #2862

Open
vvalderrv opened this issue Feb 1, 2025 · 5 comments
Open

[VPP-1398] VPP crashes when deleting tapv2 interface #2862

vvalderrv opened this issue Feb 1, 2025 · 5 comments

Comments

@vvalderrv
Copy link
Contributor

Description

I have run into this issue where VPP crashes when VPP ML2 agent tries to delete the tapv2 interface during cleanup.

 

Here's the VPP logs:

 

Aug 01 15:27:24 overcloud-controller-0.opnfvlf.org vnet[437543]: acl_plugin: ACL_FA_CLEANER_DELETE_BY_SW_IF_INDEX bitmap: 0, clear_all: 0

Aug 01 15:27:24 overcloud-controller-0.opnfvlf.org vnet[437543]: acl_plugin: ACL_FA_CLEANER: thread 0, pending clear bitmap: 0

Aug 01 15:27:24 overcloud-controller-0.opnfvlf.org vnet[437543]: acl_plugin: ACL_FA_CLEANER: thread 4294967295, pending clear bitmap: 0

Aug 01 15:27:24 overcloud-controller-0.opnfvlf.org vnet[437543]: acl_plugin: CLEANER mains len: 2 per-worker len: 2

Aug 01 15:27:24 overcloud-controller-0.opnfvlf.org vnet[437543]: acl_plugin: ACL_FA_NODE_CLEAN: cleaning done

Aug 01 15:27:26 overcloud-controller-0.opnfvlf.org vnet[437543]: acl_plugin: ACL_FA_CLEANER_DELETE_BY_SW_IF_INDEX bitmap: 0, clear_all: 0

Aug 01 15:27:26 overcloud-controller-0.opnfvlf.org vnet[437543]: acl_plugin: ACL_FA_CLEANER: thread 0, pending clear bitmap: 0

Aug 01 15:27:26 overcloud-controller-0.opnfvlf.org vnet[437543]: acl_plugin: ACL_FA_CLEANER: thread 4294967295, pending clear bitmap: 0

Aug 01 15:27:26 overcloud-controller-0.opnfvlf.org vnet[437543]: acl_plugin: CLEANER mains len: 2 per-worker len: 2

Aug 01 15:27:26 overcloud-controller-0.opnfvlf.org vnet[437543]: acl_plugin: ACL_FA_NODE_CLEAN: cleaning done

Aug 01 15:27:27 overcloud-controller-0.opnfvlf.org vnet[437543]: received signal SIGSEGV, PC 0x7f4370ce9c9a, faulting address 0xcc

Aug 01 15:27:27 overcloud-controller-0.opnfvlf.org vnet[437543]: #0  0x00007f43716186a5 0x7f43716186a5

Aug 01 15:27:27 overcloud-controller-0.opnfvlf.org vnet[437543]: #1  0x00007f436f9ca6d0 0x7f436f9ca6d0

Aug 01 15:27:27 overcloud-controller-0.opnfvlf.org vnet[437543]: #2  0x00007f4370ce9c9a 0x7f4370ce9c9a

Aug 01 15:27:27 overcloud-controller-0.opnfvlf.org vnet[437543]: #3  0x00007f432dc1ef7c dpdk_buffer_free_avx2 + 0xbdc

Aug 01 15:27:27 overcloud-controller-0.opnfvlf.org vnet[437543]: #4  0x00007f43710afb92 virtio_free_used_desc + 0x92

Aug 01 15:27:27 overcloud-controller-0.opnfvlf.org vnet[437543]: #5  0x00007f43710d3eab virtio_vring_free + 0x33b

Aug 01 15:27:27 overcloud-controller-0.opnfvlf.org vnet[437543]: #6  0x00007f43710d7b19 tap_delete_if + 0x119

Aug 01 15:27:27 overcloud-controller-0.opnfvlf.org vnet[437543]: #7  0x00007f43710d85e6 0x7f43710d85e6

Aug 01 15:27:27 overcloud-controller-0.opnfvlf.org systemd[1]: vpp.service: main process exited, code=killed, status=6/ABRT

 

 

 

Here's the gdb backtrace:

 

Program received signal SIGSEGV, Segmentation fault.

0x00007ffff7061c9a in replication_recycle_callback (vm=0x7ffff7bacf80 <vlib_global_main>, fl=0x7fffb569b700) at /usr/src/debug/vpp-18.07/src/vnet/replication.c:181

181           feature_node_index = ctx->recycle_node_index;

(gdb) bt

#0  0x00007ffff7061c9a in replication_recycle_callback (vm=0x7ffff7bacf80 <vlib_global_main>, fl=0x7fffb569b700) at /usr/src/debug/vpp-18.07/src/vnet/replication.c:181

#1  0x00007fffb3f96f7c in vlib_buffer_free_inline (follow_buffer_next=1, n_buffers=, buffers=, vm=)

    at /w/workspace/vpp-merge-1807-centos7/build-root/rpmbuild/vpp-18.07/build-data/../src/plugins/dpdk/buffer.c:388

#2  dpdk_buffer_free_avx2 (vm=, buffers=, n_buffers=)

    at /w/workspace/vpp-merge-1807-centos7/build-root/rpmbuild/vpp-18.07/build-data/../src/plugins/dpdk/buffer.c:398

#3  0x00007ffff7427b92 in vlib_buffer_free (n_buffers=1, buffers=, vm=0x7ffff7bacf80 <vlib_global_main>) at /usr/src/debug/vpp-18.07/src/vlib/buffer_funcs.h:544

#4  virtio_free_used_desc (vm=vm@entry=0x7ffff7bacf80 <vlib_global_main>, vring=vring@entry=0x7fffb7336fc0) at /usr/src/debug/vpp-18.07/src/vnet/devices/virtio/device.c:115

#5  0x00007ffff744beab in virtio_vring_free (vm=vm@entry=0x7ffff7bacf80 <vlib_global_main>, vif=vif@entry=0x7fffb7337a80, idx=)

    at /usr/src/debug/vpp-18.07/src/vnet/devices/virtio/virtio.c:165

#6  0x00007ffff744fb19 in tap_delete_if (vm=0x7ffff7bacf80 <vlib_global_main>, sw_if_index=sw_if_index@entry=4) at /usr/src/debug/vpp-18.07/src/vnet/devices/tap/tap.c:447

#7  0x00007ffff74505e6 in vl_api_tap_delete_v2_t_handler (mp=0x30172084) at /usr/src/debug/vpp-18.07/src/vnet/devices/tap/tapv2_api.c:166

#8  0x00007ffff7bb61d3 in vl_msg_api_handler_with_vm_node (am=am@entry=0x7ffff7dda000 <api_main>, the_msg=0x30172084, vm=vm@entry=0x7ffff7bacf80 <vlib_global_main>,

    node=node@entry=0x7fffb5943000) at /usr/src/debug/vpp-18.07/src/vlibapi/api_shared.c:508

#9  0x00007ffff7bbdcb5 in void_mem_api_handle_msg_i (am=, q=, node=0x7fffb5943000, vm=0x7ffff7bacf80 <vlib_global_main>)

    at /usr/src/debug/vpp-18.07/src/vlibmemory/memory_api.c:687

#10 vl_mem_api_handle_msg_main (vm=vm@entry=0x7ffff7bacf80 <vlib_global_main>, node=node@entry=0x7fffb5943000) at /usr/src/debug/vpp-18.07/src/vlibmemory/memory_api.c:697

#11 0x00007ffff7bcce1c in vl_api_clnt_process (vm=, node=0x7fffb5943000, f=) at /usr/src/debug/vpp-18.07/src/vlibmemory/vlib_api.c:349

#12 0x00007ffff7956836 in vlib_process_bootstrap (_a=) at /usr/src/debug/vpp-18.07/src/vlib/main.c:1231

#13 0x00007ffff6483068 in clib_calljmp () at /usr/src/debug/vpp-18.07/src/vppinfra/longjmp.S:110

#14 0x00007fffb5b4ce30 in ?? ()

#15 0x00007ffff7957b29 in vlib_process_startup (f=0x0, p=0x7fffb5943000, vm=0x7ffff7bacf80 <vlib_global_main>) at /usr/src/debug/vpp-18.07/src/vlib/main.c:1253

#16 dispatch_process (vm=0x7ffff7bacf80 <vlib_global_main>, p=0x7fffb5943000, last_time_stamp=9708908937542267, f=0x0) at /usr/src/debug/vpp-18.07/src/vlib/main.c:1298

#17 0x0000000000000000 in ?? ()

(gdb)

Assignee

Mohsin Kazmi

Reporter

Onong Tayeng

Comments

  • sykazmi (Wed, 20 Feb 2019 12:54:08 +0000):

    API custom dump trace:

https://pastebin.com/PSdMynkQ

  • ot (Tue, 19 Feb 2019 15:11:12 +0000): Yes, I still see the issue with 19.01.
  • jhahn (Sun, 17 Feb 2019 23:08:31 +0000): Onong Tayeng Is this still an issue in 19.01?
  • ot (Thu, 16 Aug 2018 17:14:06 +0000): VPP version = 18.07 RC1
  • ot (Thu, 16 Aug 2018 17:13:33 +0000): The API trace which reproduces the issue.

Original issue: https://jira.fd.io/browse/VPP-1398
@vvalderrv
Copy link
Contributor Author

API custom dump trace:
https://pastebin.com/PSdMynkQ

@vvalderrv
Copy link
Contributor Author

Yes, I still see the issue with 19.01.

@vvalderrv
Copy link
Contributor Author

Onong Tayeng Is this still an issue in 19.01?

@vvalderrv
Copy link
Contributor Author

VPP version = 18.07 RC1

@vvalderrv
Copy link
Contributor Author

The API trace which reproduces the issue. 

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@vvalderrv