-
Notifications
You must be signed in to change notification settings - Fork 989
Description
Describe the bug
When connecting to nginx with https using http2 the following error appears in the logs intermittently and results in client facing 502 errors;
Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
To Reproduce
Steps to reproduce the behavior:
- Install nginx container (such as nginx:alpine) and cloudflared:latest
- Use an nginx config file with http2 on (Using a short timeout or request limit triggers GOAWAY frames faster);
server {
listen 443 ssl default_server;
http2 on;
keepalive_timeout 1s;
keepalive_requests 1;
}
- Use a cloudflared config file with http2 on;
ingress:
- service: https://nginx:443
originRequest:
http2Origin: true
- Start sending requests through cloudflared and view logs. You will see the error;
Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
- Setting
http2Origin: false
resolves the issue.
Expected behavior
Cloudflared should see the GOAWAY frame and retry the request.
Environment and versions
- Cloudflared Version: 2024.6.0
Additional context
Appears to be an issue with http.Request
missing a no-op GetBody()
method.
The GOAWAY frame (type=0x7) is used to initiate graceful shutdown of a connection by a server. Nginx sends GOAWAY frames when a keepalive connection needs to end (e.g. has hit keepalive_timeout
or keepalive_requests
limits). This happens regularly with default nginx http2 configs.