🐛Cloudflared on truenas doesn't work

[PunkFleet]

Version is v1.2.5

I checked the logs after the update and realized it wasn't working. It was able to deploy successfully and normally, but was unable to establish a connection. I put truenas after openwrt v23.05 bypass gateway and I initially thought it was an openwrt issue, but it worked fine until I updated cloudflarev1.2.0 -> v 1.2.5.

2024-12-23 07:19:54.971616+00:002024-12-23T07:19:54Z INF Starting tunnel tunnelID=d652c760-3ccf-45fd-b637-8be5cbf34702
2024-12-23 07:19:54.971668+00:002024-12-23T07:19:54Z INF Version 2024.12.2 (Checksum cb61fcb41380efff4e2ef0a1eeea23bcddff360e8703f99dbb5c24da1de937b9)
2024-12-23 07:19:54.971674+00:002024-12-23T07:19:54Z INF GOOS: linux, GOVersion: go1.22.5-devel-cf, GoArch: amd64
2024-12-23 07:19:54.971678+00:002024-12-23T07:19:54Z INF Settings: map[no-autoupdate:true]
2024-12-23 07:19:54.971683+00:002024-12-23T07:19:54Z INF Environmental variables map[TUNNEL_TOKEN:*****]
2024-12-23 07:19:54.971755+00:002024-12-23T07:19:54Z INF Generated Connector ID: d8b5a889-8a16-402f-933b-389f3847f1b1
2024-12-23 07:19:54.975843+00:002024-12-23T07:19:54Z INF Initial protocol quic
2024-12-23 07:19:54.980569+00:002024-12-23T07:19:54Z INF ICMP proxy will use 172.16.5.2 as source for IPv4
2024-12-23 07:19:54.980621+00:002024-12-23T07:19:54Z INF ICMP proxy will use ::1 in zone lo as source for IPv6
2024-12-23 07:19:54.985284+00:002024-12-23T07:19:54Z INF ICMP proxy will use 172.16.5.2 as source for IPv4
2024-12-23 07:19:54.985376+00:002024-12-23T07:19:54Z INF ICMP proxy will use ::1 in zone lo as source for IPv6
2024-12-23 07:19:54.985490+00:002024-12-23T07:19:54Z INF Starting metrics server on 127.0.0.1:20241/metrics
2024-12-23 07:19:54.987556+00:002024/12/23 07:19:54 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
2024-12-23 07:19:59.990348+00:002024-12-23T07:19:59Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.192.107
2024-12-23 07:19:59.990481+00:002024-12-23T07:19:59Z INF Retrying connection in up to 2s connIndex=0 event=0 ip=198.41.192.107
2024-12-23 07:20:05.207616+00:002024-12-23T07:20:05Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.192.167
2024-12-23 07:20:05.207753+00:002024-12-23T07:20:05Z INF Retrying connection in up to 4s connIndex=0 event=0 ip=198.41.192.167
2024-12-23 07:20:11.457522+00:002024-12-23T07:20:11Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.192.47
2024-12-23 07:20:11.457658+00:002024-12-23T07:20:11Z INF Retrying connection in up to 8s connIndex=0 event=0 ip=198.41.192.47
2024-12-23 07:20:18.642862+00:002024-12-23T07:20:18Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.73
2024-12-23 07:20:18.642895+00:002024-12-23T07:20:18Z INF Retrying connection in up to 16s connIndex=0 event=0 ip=198.41.200.73
2024-12-23 07:20:30.141814+00:002024-12-23T07:20:30Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.192.107
2024-12-23 07:20:30.141911+00:002024-12-23T07:20:30Z INF Retrying connection in up to 32s connIndex=0 event=0 ip=198.41.192.107
2024-12-23 07:20:39.778089+00:002024-12-23T07:20:39Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.192.227
2024-12-23 07:20:39.778224+00:002024-12-23T07:20:39Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.192.227
2024-12-23 07:20:56.092303+00:002024-12-23T07:20:56Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.43
2024-12-23 07:20:56.092417+00:002024-12-23T07:20:56Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.43
2024-12-23 07:21:27.198045+00:002024-12-23T07:21:27Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.13
2024-12-23 07:21:27.198194+00:002024-12-23T07:21:27Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.200.13
2024-12-23 07:21:52.816659+00:002024-12-23T07:21:52Z ERR Failed to dial a quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.192.67
2024-12-23 07:21:52.816772+00:002024-12-23T07:21:52Z INF Retrying connection in up to 1m4s connIndex=0 event=0 ip=198.41.192.67
2024-12-23 07:21:55.996144+00:002024-12-23T07:21:55Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with `quic` protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/
2024-12-23 07:21:55.996305+00:00If you are using private routing to this Tunnel, then ICMP, UDP (and Private DNS Resolution) will not work unless your cloudflared can connect with Cloudflare Network with `quic`. connIndex=0 event=0 ip=198.41.192.67
2024-12-23 07:21:55.996327+00:002024-12-23T07:21:55Z INF Switching to fallback protocol http2 connIndex=0 event=0 ip=198.41.192.67
2024-12-23 07:22:10.998905+00:002024-12-23T07:22:10Z ERR Unable to establish connection with Cloudflare edge error="DialContext error: dial tcp 198.41.200.43:7844: i/o timeout" connIndex=0 event=0 ip=198.41.200.43
2024-12-23 07:22:10.999001+00:002024-12-23T07:22:10Z ERR Serve tunnel error error="DialContext error: dial tcp 198.41.200.43:7844: i/o timeout" connIndex=0 event=0 ip=198.41.200.43
2024-12-23 07:22:10.999021+00:002024-12-23T07:22:10Z INF Retrying connection in up to 1s connIndex=0 event=0 ip=198.41.200.43
2024-12-23 07:22:27.799234+00:002024-12-23T07:22:27Z ERR Unable to establish connection with Cloudflare edge error="DialContext error: dial tcp 198.41.192.67:7844: i/o timeout" connIndex=0 event=0 ip=198.41.192.67
2024-12-23 07:22:27.799353+00:002024-12-23T07:22:27Z ERR Serve tunnel error error="DialContext error: dial tcp 198.41.192.67:7844: i/o timeout" connIndex=0 event=0 ip=198.41.192.67
2024-12-23 07:22:27.799406+00:002024-12-23T07:22:27Z INF Retrying connection in up to 4s connIndex=0 event=0 ip=198.41.192.67
2024-12-23 07:22:46.423718+00:002024-12-23T07:22:46Z ERR Unable to establish connection with Cloudflare edge error="DialContext error: dial tcp 198.41.200.43:7844: i/o timeout" connIndex=0 event=0 ip=198.41.200.43
2024-12-23 07:22:46.423858+00:002024-12-23T07:22:46Z ERR Serve tunnel error error="DialContext error: dial tcp 198.41.200.43:7844: i/o timeout" connIndex=0 event=0 ip=198.41.200.43
2024-12-23 07:22:46.423879+00:002024-12-23T07:22:46Z INF Retrying connection in up to 8s connIndex=0 event=0 ip=198.41.200.43
2024-12-23 07:23:08.074934+00:002024-12-23T07:23:08Z ERR Unable to establish connection with Cloudflare edge error="DialContext error: dial tcp 198.41.200.23:7844: i/o timeout" connIndex=0 event=0 ip=198.41.200.23
2024-12-23 07:23:08.075063+00:002024-12-23T07:23:08Z ERR Serve tunnel error error="DialContext error: dial tcp 198.41.200.23:7844: i/o timeout" connIndex=0 event=0 ip=198.41.200.23
2024-12-23 07:23:08.075077+00:002024-12-23T07:23:08Z INF Retrying connection in up to 16s connIndex=0 event=0 ip=198.41.200.23
2024-12-23 07:23:35.764188+00:002024-12-23T07:23:35Z ERR Unable to establish connection with Cloudflare edge error="DialContext error: dial tcp 198.41.192.27:7844: i/o timeout" connIndex=0 event=0 ip=198.41.192.27
2024-12-23 07:23:35.764323+00:002024-12-23T07:23:35Z ERR Serve tunnel error error="DialContext error: dial tcp 198.41.192.27:7844: i/o timeout" connIndex=0 event=0 ip=198.41.192.27
2024-12-23 07:23:35.764349+00:002024-12-23T07:23:35Z INF Retrying connection in up to 32s connIndex=0 event=0 ip=198.41.192.27
2024-12-23 07:24:16.396715+00:002024-12-23T07:24:16Z ERR Unable to establish connection with Cloudflare edge error="DialContext error: dial tcp 198.41.192.47:7844: i/o timeout" connIndex=0 event=0 ip=198.41.192.47
2024-12-23 07:24:16.396815+00:002024-12-23T07:24:16Z ERR Serve tunnel error error="DialContext error: dial tcp 198.41.192.47:7844: i/o timeout" connIndex=0 event=0 ip=198.41.192.47

[otisdog8]

I'm getting a similar error on k3s. Notably:

• Tested with tcpdump & hubble (cilium observability thing) - it doesn't look like any packets are getting sent to any 198.0.0.0/8 ip addresses, even though the log messages specify 198.0.0.0/8 ips.
• I don't think it's a cluster configuration issue, since when I exec -it into a different pod, I can manually send tcp and udp packets to 198.41.192.47:7844, and these do show up on tcpdump and in hubble.
• using protocol: http2 works; I see packets in tcpdump and hubble picks up on the flows
• I went all the way back to 2024.05.0 and those images didn't work

I didn't build a debug image to check if the packets were getting lost at the container before they got forwarded through the host interface, but it's basically that or a cloudflared issue.