-
Notifications
You must be signed in to change notification settings - Fork 905
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🐛"stream #### canceled by remote with error code 0" connIndex=0 event=1 ingressRule=0 originService=" Started roughly 7/19 #1300
Comments
Please feel free to comment with rough time issue started, cloudflared version used, way tunnel is deployed (Docker, VM, Baremetal, etc) I admittedly am not able to directly do 1 on 1 troubleshooting for everyone but I've let people know internally that multiple people started reporting this error recently and linked the Github bug so it will be helpful for gathering information. |
Issue started for me around 20JUL24 00:21 MT is when one of my monitors first detected an issue. I have Cloudflare Tunnels deployed using Docker image cloudflare/cloudflared:latest. Host: Ubuntu 22.04.3. Logs:
|
Issue started for us ~4 days ago - 20.07.24. We are using multiple Docker based tunnels distributed across various OSes from Ubuntu to RPi OS. Most of them are 2024.6.1 but some systems are 2024.6.0. Issue is that after a 1-2 days of uptime the tunnel starts disconnecting intermittently - some calls to the backend work and some fail with error code 524 (Timeout). We have noticed that when this happens out of 10 HTTP calls 3 fail. The workaround is to restart the tunnel. We have confirmed that its not a network issue on our side. It also happens to multiple, independent systems that are located in different datacenters. We have observed the issue happening at Hetzner Germany, OVH France and our own DC in Bulgaria on the same day - not the same time. Those providers have direct peering with Cloudflare and have no reported network outages during that time. Logs:
We are also seeing:
|
Our production and staging environment went down across two K8S clusters and eight Docker Swarm clusters in three different psychical locations the 19th of July and again this morning (24th of July). Restarting our cloudflared system services and cloudflared containers helped. The tunnel metrics @DevinCarr @jcsf This needs to be addressed immidiately. I have also opened an enterprise support ticket, to make sure this gets some traction. I don't think this is a problem with the cloudflared binary, since we see it accross a wide array of cloudflared versions. |
same here across multiple tunnels. |
I utilize a Cloudflare tunnel on my home lab and I initially ran into this issue while on vacation last week. Had to VPN into my home network to restart the tunnel. I'm running a tunnel on Unraid utilizing this Docker repo: https://github.com/AriaGomes/Unraid-Cloudflared-Tunnel. I'm receiving the same error messages as everyone else. |
We are investigating on our side we will let you know once we have more information. Sorry for not having more information to provide right now. |
Started Monday for us using multiple docker tunnels on version 2024.06.[0/1] |
Same issue here, running cloudflared in docker with following config:
|
Same here, it started when the larger cloudflare update was rolling out into different datacenters earlier this month. The phenomenon looks like this: |
Same here. |
Looks better now. Tunnels were rerouted to other datacenters. No issues again. |
I have When I check the logs, I just see a sea of stream closed with error 0 messages. A reboot fixes it for a few hours, then it starts up again. |
Exact same issue. Seems this is hitting a LOT of people suddenly and probably thousands more who don't even know it's happening. This ticket should be changed to Priority "HIGH". |
We're seeing a similar issue. 2024.6.1 for all clients for us. Started 7/22/24 for us. |
My tunnel went first time down on 12th and after that it's been working couple of days at a time. Yesterday it only worked couple of hours after restarting so I tested reverting to an older version. Now almost a day with version 2024.4.1 I haven't (yet) seen any issues. Before these issues tunnel had been working without any problems way over a year. |
Our connectors are set to use http2, anyone having this issue using quic? |
Yes I only use quic and have this issue. |
Same here, HA 12.4, Cloudflared version: 5.1.15 |
Running a NixOS 23.11 VM with version We noticed issues as early as July 17 but possibly even a few days earlier. |
I've also noticed this issue. Is affecting the stability of my tunnels with no obvious sign of issues, I wish there was a health check so that I could restart my tunnel when I have an issue like this. |
@dledfordcf, any news from the inside team(s)? |
At this time, the impact should no longer be visible. We had made a change on the edge that caused a small amount of QUIC packets to be routed and dropped for some cloudflared tunnel connections. This is the reason why many of your cloudflared logs mentioned timeouts and remote/local closing the stream connections. This change has been rolled back and your tunnels should go back to normal without any change on your part. However, please keep in mind that you may still occasionally see the error message in your cloudflared logs:
Thank you for your patience as we investigated this. |
I'm still seeing this problem. I realise that you said we'll still sometimes see it. Does this really count as a fix then? At least I guess I'll have to wait and see. |
Same for me. The problem is still here on every hour |
per your last comment @DevinCarr, is it possible to make it silent/only shown on deeper log levels? I dont think this is shown at general level on web servers such as nginx (eyeball early disconnects, etc..) please correct me if i'm wrong. |
Hi.
Anything I can provide for further investigation? |
Hi, Im also getting this issue intermittently and have had to stop using the tunnel as a result:
Happy to run any further tests if people need |
I have these errors constantly. No one is reporting not being able to connect but there have been some weird things with pages loading improperly from a few people.
|
I see this issue on 1 of 12 tunnels I operate. Any fix in sight? |
Hey there. We are having the exact same issue, and it seems to be related to how Cloudflare handles peer discovery. Cloudflared stills picks proxy locations that can be monitored as "rerouted" from Cloudflare status page. I updated our analysis here: #1360 (comment) |
im still having this issue |
the same issue here. Need restart the service to back normal, after some hours, issue back again. :( |
Describe the bug
Opening this as a centralized place for this issue.
To Reproduce
Was unable to directly reproduce on my own tunnel, but the from what I have gathered from others:
The issue happens at random, rebooting tunnel resolves the issue temporarily, but it will resurface.
Unifying fact seems to be version 2024.6.1 and issues starting around 7/19
If it's an issue with Cloudflare Tunnel:
4. Tunnel ID : Multiple
5. cloudflared config:
Expected behavior
Tunnel would connect to edge and work
Environment and versions
Logs and errors
I dont have any of my own logs for this
Additional context
No tunnel updates were released when this started as 2024.6.1 has been out around a month, but there is a large grouping of people with this starting around the same time of 7/19/2024
If anyone from the Cloudflare team checks this bug as well, feel free to hit me up internally.
The text was updated successfully, but these errors were encountered: