API SSL error with some ipv6 addresses

[tootai]

Hi list,

I use Telegram bot on a Debian12 ipv6 server running librenms and Transport being Telegram. I also wrote a script to send messages to a group. Since those last days I didn't receive any alert from Librenms and I discover that curl fail with error 28 SSL connection timeout when testing from Librenms. An error occurs too after 2 minutes when I run curl from command line like

curl https://api.telegram.org/bot/sendMessage?chat_id=&text=Is%20not,%20functional%20yet
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:02:01 --:--:-- 0
curl: (35) Recv failure: Connection reset by peer

Running curl -4 ... and the job is done. api.telegram.org is resolved in ipv4 and ipv6, nc -v <api.telegram.org ipv6> 443 succeeded as well as a mtr to the IP.

I discovered that with another ipv6 address, from the same server, it's OK ! What's going on here ? The failing ipv6 address being from the IP range 2001:67c:1298::/48.

Thanks for your support

[levlam]

For me api.telegram.org resolves to 2001:67c:4e8:f004::9, which isn't in the mentioned network.

[tootai]

Exactly, but why I didn't get full answer with IPv6 belonging to the above /48 range ?

[levlam]

Why should you receive an answer from the 2001:67c:1298::/48 IP address, if api.telegram.org resolves to another address?

[tootai]

OK, seems I didn't explain clearly: my IP is from 2001:67c:1298::/48 range. My other IP is from 2a01:4f8:210::/48 range. Connecting to api using the IP from this second range is OK but when I use the other IP from the first range I see some traffic with tcpdump, outgoing and incoming, which means both servers are connected and exchanging datas but nothing else, it finish with a timeout in terminal mode or error 28 SSL inside Librenms.

Hope you now understand the problematic ;)

[levlam]

This looks like an Internet connectivity issue, which happens much more often with IPv6 networks. You can try to use tcptraceroute to find the cause of the issue and report it to your Internet provider.

[tootai]

I don't think so: from the same IP range other computer I have the same problem. Also this server is ONLY ipv6 and running 24h/24h (backups, web server, emails aso) so I would not put 1 cent of the connectivity. Server is in a DC with other of our servers, no one is showing a problem. And remember, Librenms is running on it watching our servers in various countries, not any alert about such a failure.

[tootai]

mtr report 17ms avg time with 0% loss for 100 packets sent

[levlam]

mtr uses ICMPv6 packets, so its results are often irrelevant when investigating TCP connectivity issues.

[tootai]

Please, dont't try to explain me how mtr/icmp/... works. Generally we all use ICMP/ICMPv6 to check connectivity. If you feel out of idea, just tell it.

Remember with one IP range it's OK the other one not, both using the same Internet provider till a certain point.

[levlam]

If site works from one network and doesn't work from another, then this is very likely to be a network connectivity issue. The used protocol for data transfer is of utmost importance in the case of such network issues. For api.telegram.org there is no difference from which IPv6 address the request is received.

[tootai]

I opened a case on peering partner. I ran a tcptraceroute6 and got for port 80

root@zone-s:/usr/local/bin# tcptraceroute6 api.telegram.org traceroute to api.telegram.org (2001:67c:4e8:f004::9) from 2001:67c:1298::ff04, port 80, from port 17001, 30 hops max, 60 bytes packets 1 2a01:4f8:120:1104::70:1 (2a01:4f8:120:1104::70:1) 0.103 ms 0.146 ms 0.115 ms 2 fd99:a:b:98:67::1 (fd99:a:b:98:67::1) 3.552 ms 3.555 ms 3.306 ms 3 2a00:6340:1000:2024::2 (2a00:6340:1000:2024::2) 7.566 ms 6.779 ms 6.660 ms 4 2a00:6340:1000:2024::1 (2a00:6340:1000:2024::1) 6.857 ms 6.779 ms 6.559 ms 5 C35362-410.C35362-413.telegram.org (2001:7f8::f259:0:2) 17.739 ms 13.356 ms 15.435 ms 6 2001:67c:4e8:801a:1:9102:1:1 (2001:67c:4e8:801a:1:9102:1:1) 13.292 ms 15.342 ms 13.189 ms 7 2001:67c:4e8:801a:1:801e:1:2 (2001:67c:4e8:801a:1:801e:1:2) 15.329 ms 13.406 ms 13.152 ms 8 2001:67c:4e8:eeee:801e:4:0:2632 (2001:67c:4e8:eeee:801e:4:0:2632) 13.457 ms 13.291 ms 13.314 ms 9 2001:67c:4e8:f004::9 (2001:67c:4e8:f004::9) 12.911 ms [open] 12.881 ms 13.014 ms

and for port 443

root@zone-s:/usr/local/bin# tcptraceroute6 api.telegram.org 443 traceroute to api.telegram.org (2001:67c:4e8:f004::9) from 2001:67c:1298::ff04, port 443, from port 16883, 30 hops max, 60 bytes packets 1 2a01:4f8:120:1104::70:1 (2a01:4f8:120:1104::70:1) 0.119 ms 0.159 ms 0.213 ms 2 fd99:a:b:98:67::1 (fd99:a:b:98:67::1) 3.155 ms 3.157 ms 2.949 ms 3 2a00:6340:1000:2024::2 (2a00:6340:1000:2024::2) 6.859 ms 6.714 ms 6.757 ms 4 * * * 5 * * * 6 C35362-410.C35362-413.telegram.org (2001:7f8::f259:0:2) 13.879 ms 14.173 ms 13.704 ms 7 2001:67c:4e8:801a:1:9102:1:1 (2001:67c:4e8:801a:1:9102:1:1) 13.683 ms 13.744 ms 13.570 ms 8 2001:67c:4e8:801a:1:801e:1:2 (2001:67c:4e8:801a:1:801e:1:2) 13.531 ms 13.381 ms 13.570 ms 9 2001:67c:4e8:eeee:801e:4:0:2632 (2001:67c:4e8:eeee:801e:4:0:2632) 13.501 ms 13.552 ms 13.301 ms 10 2001:67c:4e8:f004::9 (2001:67c:4e8:f004::9) 12.885 ms [open] 12.742 ms 12.926 ms

In verbose mode
`* Host api.telegram.org:443 was resolved.

• IPv6: 2001:67c:4e8:f004::9
• IPv4: 149.154.167.220
• Trying [2001:67c:4e8:f004::9]:443...
• Connected to api.telegram.org (2001:67c:4e8:f004::9) port 443
• ALPN: curl offers h2,http/1.1
• TLSv1.3 (OUT), TLS handshake, Client hello (1):
• CAfile: /etc/ssl/certs/ca-certificates.crt
• CApath: /etc/ssl/certs
• Recv failure: Connection reset by peer
• OpenSSL SSL_connect: Connection reset by peer in connection to api.telegram.org:443
• Closing connection
  curl: (35) Recv failure: Connection reset by peer`

Nothing more

So like icmpv6, tcp6 seems OK but still in error. Banging my head :)

[levlam]

Could you also try mtr --tcp -P 443?

[levlam]

Could you also try from some other IP address from the 2001:67c:1298::0/48 network other than 2001:67c:1298::ff04?

[tootai]

Same with all 2001:67c:1298::/64 range (I use this /64 and 2001:67c:1298:10::/64 from this /48 range)

root@zone-s:/tmp# mtr --tcp -P 443 --report-wide api.telegram.org
Start: 2024-03-20T15:39:21+0100 [4/1258]
HOST: zone-s Loss% Snt Last Avg Best Wrst StDev
1.|-- 2a01:4f8:120:1104::70:1 0.0% 10 0.4 0.3 0.2 0.5 0.1
2.|-- fd99:a:b:98:67::1 0.0% 10 3.2 3.4 3.2 4.8 0.5
3.|-- 2a00:6340:1000:2024::2 0.0% 10 6.8 7.0 6.7 7.7 0.3
4.|-- 2a00:6340:1000:2024::1 50.0% 10 7.2 7.5 7.2 8.3 0.5
5.|-- C35362-410.C35362-413.telegram.org 60.0% 10 14.5 17.6 13.7 27.9 6.9
6.|-- C35362-410.C35362-413.telegram.org 0.0% 10 15.2 14.7 13.6 19.2 1.7
2001:67c:4e8:801a:1:9102:1:1
2001:67c:4e8:7003:1:9102:1:1
2001:67c:4e8:801b:1:9102:1:1
7.|-- 2001:67c:4e8:801a:1:801e:1:2 0.0% 10 14.3 14.0 13.7 14.3 0.2
2001:67c:4e8:801a:1:9102:1:1
2001:67c:4e8:7003:1:9102:1:1
2001:67c:4e8:801b:1:801f:1:2
2001:67c:4e8:801a:1:801f:1:2
2001:67c:4e8:7003:1:801e:1:2
2001:67c:4e8:801b:1:9102:1:1
8.|-- 2001:67c:4e8:eeee:801f:4:0:2631 0.0% 10 13.8 13.8 13.5 14.1 0.2
2001:67c:4e8:eeee:801e:4:0:2632
2001:67c:4e8:7003:1:801e:1:2
2001:67c:4e8:eeee:801e:4:0:2634
2001:67c:4e8:eeee:801f:4:0:2634
2001:67c:4e8:801a:1:801f:1:2
2001:67c:4e8:eeee:801f:4:0:2633
2001:67c:4e8:801b:1:801e:1:2
2001:67c:4e8:801b:1:801f:1:2
9.|-- 2001:67c:4e8:eeee:801f:4:0:2633 0.0% 10 13.8 13.8 13.4 14.1 0.2
2001:67c:4e8:eeee:801f:4:0:2635
2001:67c:4e8:eeee:801e:4:0:2636
2001:67c:4e8:eeee:801e:4:0:2631
2001:67c:4e8:eeee:801f:4:0:2631
2001:67c:4e8:eeee:801e:4:0:2634
2001:67c:4e8:f004::9
2001:67c:4e8:eeee:801f:4:0:2634
10.|-- 2001:67c:4e8:f004::9 0.0% 8 13.7 13.6 13.3 14.2 0.3

I downloaded with curl from various repos including debian-netinstall.iso, everything like expected.

[tootai]

First packets output of tcpdump from failing IP:

11:33:41.430289` ens1 Out IP6 2001:67c:1298::ff04.60280 > 2001:67c:4e8:f004::9.https: Flags [S], seq 2294335465, win 64800, options [mss 1440,sackOK,TS val 2421837041 ecr 0,nop,wscale 7], length 0

11:33:41.443297 ens1 In IP6 2001:67c:4e8:f004::9.https > 2001:67c:1298::ff04.60280: Flags [S.], seq 3879809308, ack 2294335466, win 28560, options [mss 1440,sackOK,TS val 3435357583 ecr 2421837041,nop,wscale 10], length 0
11:33:41.443327 ens1 Out IP6 2001:67c:1298::ff04.60280 > 2001:67c:4e8:f004::9.https: Flags [.], ack 1, win 507, options [nop,nop,TS val 2421837054 ecr 3435357583], length 0
11:33:41.446243 ens1 Out IP6 2001:67c:1298::ff04.60280 > 2001:67c:4e8:f004::9.https: Flags [P.], seq 1:518, ack 1, win 507, options [nop,nop,TS val 2421837057 ecr 3435357583], length 517
11:33:41.459314 ens1 In IP6 2001:67c:4e8:f004::9.https > 2001:67c:1298::ff04.60280: Flags [.], ack 518, win 29, options [nop,nop,TS val 3435357587 ecr 2421837057], length 0
11:33:41.495913 ens1 In IP6 2001:67c:4e8:f004::9.https > 2001:67c:1298::ff04.60280: Flags [P.], seq 5525:5698, ack 518, win 29, options [nop,nop,TS val 3435357597 ecr 2421837057], length 173
11:33:41.495932 ens1 Out IP6 2001:67c:1298::ff04.60280 > 2001:67c:4e8:f004::9.https: Flags [.], ack 1, win 507, options [nop,nop,TS val 2421837107 ecr 3435357587,nop,nop,sack 1 {5525:5698}], length 0

As you can notice, the before last line is a seq:5525:5698, ack 518 followed by a new ack1. Now the output of a working IP:

11:43:41.211636 ens1 Out IP6 2a01:4f8:120:1104::70:12.49804 > 2001:67c:4e8:f004::9.443: Flags [S], seq 3848103319, win 64800, options [mss 1440,sackOK,TS val 2764072909 ecr 0,nop,wscale 7], length 0
11:43:41.222328 ens1 In IP6 2001:67c:4e8:f004::9.443 > 2a01:4f8:120:1104::70:12.49804: Flags [S.], seq 3343337882, ack 3848103320, win 28560, options [mss 1440,sackOK,TS val 4200978678 ecr 2764072909,nop,wscale 10], length 0
11:43:41.222377 ens1 Out IP6 2a01:4f8:120:1104::70:12.49804 > 2001:67c:4e8:f004::9.443: Flags [.], ack 1, win 507, options [nop,nop,TS val 2764072919 ecr 4200978678], length 0
11:43:41.225508 ens1 Out IP6 2a01:4f8:120:1104::70:12.49804 > 2001:67c:4e8:f004::9.443: Flags [P.], seq 1:518, ack 1, win 507, options [nop,nop,TS val 2764072922 ecr 4200978678], length 517
11:43:41.236092 ens1 In IP6 2001:67c:4e8:f004::9.443 > 2a01:4f8:120:1104::70:12.49804: Flags [.], ack 518, win 29, options [nop,nop,TS val 4200978681 ecr 2764072922], length 0
11:43:41.237072 ens1 In IP6 2001:67c:4e8:f004::9.443 > 2a01:4f8:120:1104::70:12.49804: Flags [.], seq 1:1429, ack 518, win 29, options [nop,nop,TS val 4200978682 ecr 2764072922], length 1428
11:43:41.237082 ens1 Out IP6 2a01:4f8:120:1104::70:12.49804 > 2001:67c:4e8:f004::9.443: Flags [.], ack 1429, win 501, options [nop,nop,TS val 2764072934 ecr 4200978682], length 0

Here the before last line is seq1:1429 followed by the ack 1429 line which does the sequences continuing as they should.

Question now is why in the first case I receive a false seq number with P flag ?