Navigation Failure for LTE devices

[asantos-vk]

Your Environment

• 1.8:
• Affected Component: Access Gateway
• Affected Subcomponent: AGW Datapath
• Deployment Environment: bare-metal

Describe the Issue

I am troubleshooting a navigation issue, and here are the outputs. In short, packets arrive on GTP_BR0 but do not appear on SGi interface.

To Reproduce

After a period of operation, the devices stop navigating. The UE shows a signal, but the device can't pass data. One way to reproduce is resetting the gtpu_sys_2152 interface.

Expected behavior

Device is connected and passing data.

UL/DL STATS

root@access-gateway:/home/magma# dp_probe_cli.py -i 724900000000008 -d UL stats
IMSI: 724900000000008, IP: 192.168.128.13
UL stats: n_packets=1287, n_bytes=97239
root@access-gateway:/home/magma# dp_probe_cli.py -i 724900000000008 -d UL stats
IMSI: 724900000000008, IP: 192.168.128.13
UL stats: n_packets=1302, n_bytes=98147
root@access-gateway:/home/magma# dp_probe_cli.py -i 724900000000008 -d DL stats
IMSI: 724900000000008, IP: 192.168.128.13
LOCAL: n_packets=0, n_bytes=0
mtr0: n_packets=0, n_bytes=0
root@access-gateway:/home/magma#

TCPDUMP GTPU_SYS WITH DST 142.250.79.206

root@access-gateway:/home/magma# tcpdump -eni gtpu_sys_2152 dst 142.250.79.206
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on gtpu_sys_2152, link-type RAW (Raw IP), capture size 262144 bytes
10:32:47.435036 ip: 192.168.128.13 > 142.250.79.206: ICMP echo request, id 69, seq 1, length 8
10:32:48.455133 ip: 192.168.128.13 > 142.250.79.206: ICMP echo request, id 69, seq 2, length 8
10:32:49.475068 ip: 192.168.128.13.47450 > 142.250.79.206.7: Flags [S], seq 2878851933, win 65535, options [mss 1300,sackOK,TS val 159437123 ecr 0,nop,wscale 9], length 0
10:32:50.505082 ip: 192.168.128.13.47450 > 142.250.79.206.7: Flags [S], seq 2878851933, win 65535, options [mss 1300,sackOK,TS val 159438145 ecr 0,nop,wscale 9], length 0
10:32:52.535047 ip: 192.168.128.13 > 142.250.79.206: ICMP echo request, id 70, seq 1, length 8
10:32:53.645077 ip: 192.168.128.13 > 142.250.79.206: ICMP echo request, id 70, seq 2, length 8
10:32:54.568173 ip: 192.168.128.13.47454 > 142.250.79.206.7: Flags [S], seq 2541092784, win 65535, options [mss 1300,sackOK,TS val 159442212 ecr 0,nop,wscale 9], length 0
10:32:55.585104 ip: 192.168.128.13.47454 > 142.250.79.206.7: Flags [S], seq 2541092784, win 65535, options [mss 1300,sackOK,TS val 159443234 ecr 0,nop,wscale 9], length 0
10:32:57.615059 ip: 192.168.128.13 > 142.250.79.206: ICMP echo request, id 71, seq 1, length 8
^C
9 packets captured
9 packets received by filter
0 packets dropped by kernel

OVS-VSCTL SHOW

root@access-gateway:/home/magma# ovs-vsctl show
bc6f8bd1-fae8-48b3-b5bb-d36cf9b9ef8c
Manager "ptcp:6640"
Bridge gtp_br0
Controller "tcp:127.0.0.1:6633"
is_connected: true
Controller "tcp:127.0.0.1:6654"
is_connected: true
fail_mode: secure
Port patch-up
Interface patch-up
type: patch
options: {peer=patch-agw}
Port mtr0
Interface mtr0
type: internal
Port ipfix0
Interface ipfix0
type: internal
Port li_port
Interface li_port
type: internal
Port gtp0
Interface gtp0
type: gtpu
options: {key=flow, remote_ip=flow}
Port g_6402000a
Interface g_6402000a
type: gtpu
options: {csum="false", key=flow, remote_ip="10.0.2.100"}
bfd_status: {diagnostic="Neighbor Signaled Session Down", flap_count="0", forwarding="true", remote_diagnostic="No Diagnostic", remote_state=up, state=up}
Port gtp_br0
Interface gtp_br0
type: internal
Port proxy_port
Interface proxy_port
Bridge uplink_br0
Port dhcp0
Interface dhcp0
type: internal
Port uplink_br0
Interface uplink_br0
type: internal
Port patch-agw
Interface patch-agw
type: patch
options: {peer=patch-up}
ovs_version: "2.15.4"
root@access-gateway:/home/magma#

[asantos-vk]

updating:
ovs flows keeps being modified while UE tries to pass data. When running watch -n0 ovs-ofctl -O OpenFlow13 dump-flows gtp_br0 table=0 we can see that the flows are not persisting.

[asantos-vk]

Also, there are some logs that ovsdb connection is being dropped.
root@magma-agw:/home/magma# tail -f /var/log/openvswitch/ovsdb-server.log
2024-04-02T19:56:12.728Z|00444|reconnect|WARN|unix#840: connection dropped (Connection reset by peer)
2024-04-02T19:56:12.761Z|00445|reconnect|WARN|unix#842: connection dropped (Connection reset by peer)
2024-04-02T19:56:12.777Z|00446|reconnect|WARN|unix#843: connection dropped (Connection reset by peer)
2024-04-02T19:56:12.794Z|00447|reconnect|WARN|unix#844: connection dropped (Connection reset by peer)
2024-04-02T19:56:12.814Z|00448|reconnect|WARN|unix#845: connection dropped (Connection reset by peer)
2024-04-02T19:56:12.828Z|00449|reconnect|WARN|unix#846: connection dropped (Connection reset by peer)
2024-04-02T19:56:12.844Z|00450|reconnect|WARN|unix#847: connection dropped (Connection reset by peer)
2024-04-02T19:56:12.861Z|00451|reconnect|WARN|unix#848: connection dropped (Connection reset by peer)
2024-04-02T19:56:12.880Z|00452|reconnect|WARN|unix#849: connection dropped (Connection reset by peer)
2024-04-02T19:56:12.896Z|00453|reconnect|WARN|unix#850: connection dropped (Connection reset by peer)
^C