Skip to content

🐛 GOAWAY frames not handled correctly with http2 #1273

@zemaj

Description

@zemaj

Describe the bug
When connecting to nginx with https using http2 the following error appears in the logs intermittently and results in client facing 502 errors;

Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error

To Reproduce
Steps to reproduce the behavior:

  1. Install nginx container (such as nginx:alpine) and cloudflared:latest
  2. Use an nginx config file with http2 on (Using a short timeout or request limit triggers GOAWAY frames faster);
server {
    listen      443 ssl default_server;
    http2       on;
    keepalive_timeout           1s;
    keepalive_requests         1;
}
  1. Use a cloudflared config file with http2 on;
    ingress:
      - service: https://nginx:443
        originRequest:
          http2Origin: true
  1. Start sending requests through cloudflared and view logs. You will see the error;
Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error
  1. Setting http2Origin: false resolves the issue.

Expected behavior
Cloudflared should see the GOAWAY frame and retry the request.

Environment and versions

  • Cloudflared Version: 2024.6.0

Additional context
Appears to be an issue with http.Request missing a no-op GetBody() method.

The GOAWAY frame (type=0x7) is used to initiate graceful shutdown of a connection by a server. Nginx sends GOAWAY frames when a keepalive connection needs to end (e.g. has hit keepalive_timeout or keepalive_requests limits). This happens regularly with default nginx http2 configs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Priority: NormalMinor issue impacting one or more usersType: BugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions