Exposing hubble-ui behind Contour/Envoy ingress leads to unsupported HTTP/2 protocol preface requests

[demikl]

Hi.

Using helm chart 1.12.7

I've exposed Hubble-UI through an ingress managed by Contour. I've exposed the path /api/ through a service configured with h2c protocol since these requests are HTTP/2 based.

Unfortunately, this does not work since, as far as I understand, configuring Contour to use gRPC requests leads to its reverse-proxy, envoy, to check if the upstream server (hubble-ui is this case) is talking HTTP/2 protocol with these requests:

100.83.80.69 - - [24/Feb/2023:16:45:17 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" "-"
100.83.80.69 - - [24/Feb/2023:16:45:19 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" "-"
100.83.80.69 - - [24/Feb/2023:16:45:25 +0000] "PRI * HTTP/2.0" 400 157 "-" "-" "-"
...

Hubble-UI does not like those requests and envoy is failing to start forwarding requests from /api/ to hubble-ui. A solution might be to stop envoy from sending those "probes" but it looks like it is not yet implemented. On the other hand, it looks like any HTTP/2 server should implement a response to the PRI verb.

Here is the ingress I'm using :

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hubble-ui
  namespace: kube-system
spec:
  rules:
  - host: hubble.somewhere.com
    http:
      paths:
      - backend:
          service:
            name: hubble-ui
            port:
              name: http
        path: /
        pathType: Prefix
  - host: hubble.somewhere.com
    http:
      paths:
      - backend:
          service:
            name: hubble-ui-grpc
            port:
              name: http
        path: /api/
        pathType: Prefix

along with those two services:

---
apiVersion: v1
kind: Service
metadata:
  name: hubble-ui
  namespace: kube-system
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8081
  selector:
    k8s-app: hubble-ui

---
apiVersion: v1
kind: Service
metadata:
  annotations:
    projectcontour.io/upstream-protocol.h2c: http
  name: hubble-ui-grpc
  namespace: kube-system
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8081
  selector:
    k8s-app: hubble-ui

[demikl]

Not sure if these logs would help, but here are the debug logs from the envoy reverse-proxy managed by Contour.

You might notice that the request is received as HTTP plain-text, that's because this envoy is exposed behind an AWS NLB that is handling the TLS termination.

IMHO the most interesting log line is:

invalid http2: Remote peer returned unexpected data while we expected SETTINGS frame.  Perhaps, peer does not support HTTP/2 properly.

[2023-02-24 17:39:27.588][15][debug][http] [source/common/http/conn_manager_impl.cc:306] [C45] new stream
[2023-02-24 17:39:27.588][15][debug][http] [source/common/http/conn_manager_impl.cc:930] [C45][S11027684788373223109] request headers complete (end_stream=false):
':method', 'POST'
':path', '/api/ui.UI/GetControlStream'
':authority', 'hubble-ui.<REDACTED>'
':scheme', 'https'
'user-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/110.0'
'accept', 'application/grpc-web-text'
'accept-language', 'en-US,fr;q=0.8,fr-FR;q=0.5,en;q=0.3'
'accept-encoding', 'gzip, deflate, br'
'content-type', 'application/grpc-web-text'
'x-user-agent', 'grpc-web-javascript/0.1'
'x-grpc-web', '1'
'content-length', '8'
'origin', 'https://hubble-ui.<REDACTED>'
'referer', 'https://hubble-ui.<REDACTED>/'
'sec-fetch-dest', 'empty'
'sec-fetch-mode', 'cors'
'sec-fetch-site', 'same-origin'
'te', 'trailers'
'cookie', 'crowd.token_key=<REDACTED>'

[2023-02-24 17:39:27.588][15][debug][connection] [./source/common/network/connection_impl.h:92] [C45] current connecting state: false
[2023-02-24 17:39:27.588][15][debug][router] [source/common/router/router.cc:470] [C45][S11027684788373223109] cluster 'kube-system/hubble-ui/80/f4f94965ec' match for URL '/api/ui.UI/GetControlStream'
[2023-02-24 17:39:27.588][15][debug][router] [source/common/router/router.cc:678] [C45][S11027684788373223109] router decoding headers:
':method', 'POST'
':path', '/api/ui.UI/GetControlStream'
':authority', 'hubble-ui.<REDACTED>'
':scheme', 'https'
'user-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/110.0'
'accept', 'application/grpc-web-text'
'accept-language', 'en-US,fr;q=0.8,fr-FR;q=0.5,en;q=0.3'
'accept-encoding', 'gzip, deflate, br'
'content-type', 'application/grpc'
'x-user-agent', 'grpc-web-javascript/0.1'
'x-grpc-web', '1'
'origin', 'https://hubble-ui.<REDACTED>'
'referer', 'https://hubble-ui.<REDACTED>/'
'sec-fetch-dest', 'empty'
'sec-fetch-mode', 'cors'
'sec-fetch-site', 'same-origin'
'te', 'trailers'
'cookie', 'crowd.token_key=<REDACTED>'
'x-forwarded-for', '10.206.48.5'
'x-forwarded-proto', 'http'
'x-envoy-internal', 'true'
'x-request-id', 'b496003b-9c47-4021-b10b-76a577401b4a'
'grpc-accept-encoding', 'identity'
'x-envoy-expected-rq-timeout-ms', '15000'
'x-request-start', 't=1677260367.588'

[2023-02-24 17:39:27.588][15][debug][pool] [source/common/http/conn_pool_base.cc:78] queueing stream due to no available connections (ready=0 busy=0 connecting=0)
[2023-02-24 17:39:27.588][15][debug][pool] [source/common/conn_pool/conn_pool_base.cc:290] trying to create new connection
[2023-02-24 17:39:27.588][15][debug][pool] [source/common/conn_pool/conn_pool_base.cc:145] creating a new connection (connecting=0)
[2023-02-24 17:39:27.588][15][debug][http2] [source/common/http/http2/codec_impl.cc:1794] [C148] updating connection-level initial window size to 268435456
[2023-02-24 17:39:27.588][15][debug][connection] [./source/common/network/connection_impl.h:92] [C148] current connecting state: true
[2023-02-24 17:39:27.588][15][debug][client] [source/common/http/codec_client.cc:57] [C148] connecting
[2023-02-24 17:39:27.588][15][debug][connection] [source/common/network/connection_impl.cc:924] [C148] connecting to 100.83.84.142:8081
[2023-02-24 17:39:27.588][15][debug][connection] [source/common/network/connection_impl.cc:943] [C148] connection in progress
[2023-02-24 17:39:27.589][15][debug][connection] [source/common/network/connection_impl.cc:683] [C148] connected
[2023-02-24 17:39:27.589][15][debug][client] [source/common/http/codec_client.cc:88] [C148] connected
[2023-02-24 17:39:27.589][15][debug][pool] [source/common/conn_pool/conn_pool_base.cc:327] [C148] attaching to next stream
[2023-02-24 17:39:27.589][15][debug][pool] [source/common/conn_pool/conn_pool_base.cc:181] [C148] creating stream
[2023-02-24 17:39:27.589][15][debug][router] [source/common/router/upstream_request.cc:564] [C45][S11027684788373223109] pool ready
[2023-02-24 17:39:27.589][15][debug][http] [source/common/http/conn_manager_impl.cc:913] [C45][S11027684788373223109] request end stream
[2023-02-24 17:39:27.589][15][debug][client] [source/common/http/codec_client.cc:139] [C148] encode complete
[2023-02-24 17:39:27.590][15][debug][http2] [source/common/http/http2/codec_impl.cc:1379] [C148] invalid http2: Remote peer returned unexpected data while we expected SETTINGS frame.  Perhaps, peer does not support HTTP/2 properly.
[2023-02-24 17:39:27.590][15][debug][http2] [source/common/http/http2/codec_impl.cc:1345] [C148] sent goaway code=1
[2023-02-24 17:39:27.590][15][debug][client] [source/common/http/codec_client.cc:168] [C148] Error dispatching received data: The user callback function failed
[2023-02-24 17:39:27.590][15][debug][connection] [source/common/network/connection_impl.cc:139] [C148] closing data_to_write=34 type=1
[2023-02-24 17:39:27.590][15][debug][connection] [source/common/network/connection_impl.cc:250] [C148] closing socket: 1
[2023-02-24 17:39:27.591][15][debug][client] [source/common/http/codec_client.cc:107] [C148] disconnect. resetting 1 pending requests
[2023-02-24 17:39:27.591][15][debug][client] [source/common/http/codec_client.cc:156] [C148] request reset
[2023-02-24 17:39:27.591][15][debug][pool] [source/common/conn_pool/conn_pool_base.cc:214] [C148] destroying stream: 0 remaining
[2023-02-24 17:39:27.591][15][debug][router] [source/common/router/router.cc:1212] [C45][S11027684788373223109] upstream reset: reset reason: protocol error, transport failure reason: 
[2023-02-24 17:39:27.591][15][debug][http] [source/common/http/filter_manager.cc:905] [C45][S11027684788373223109] Sending local reply with details upstream_reset_before_response_started{protocol_error}
[2023-02-24 17:39:27.591][15][debug][http] [source/common/http/conn_manager_impl.cc:1551] [C45][S11027684788373223109] encoding headers via codec (end_stream=false):
':status', '502'
'content-length', '0'
'content-type', 'application/grpc-web-text+proto'
'grpc-message', 'upstream connect error or disconnect/reset before headers. reset reason: protocol error'
'grpc-status', '14'
'date', 'Fri, 24 Feb 2023 17:39:27 GMT'
'server', 'envoy'

[2023-02-24 17:39:27.591][15][debug][http2] [source/common/http/http2/codec_impl.cc:1479] [C45] stream 61 closed: 0
[2023-02-24 17:39:27.591][15][debug][http2] [source/common/http/http2/codec_impl.cc:1532] [C45] Recouping 0 bytes of flow control window for stream 61.
[2023-02-24 17:39:27.591][15][debug][pool] [source/common/conn_pool/conn_pool_base.cc:483] [C148] client disconnected, failure reason: 
[2023-02-24 17:39:27.591][15][debug][pool] [source/common/conn_pool/conn_pool_base.cc:453] invoking idle callbacks - is_draining_for_deletion_=false
[2023-02-24 17:39:27.591][15][debug][connection] [source/common/network/connection_impl.cc:651] [C148] remote close

[alvaroaleman]

Fix for this is in #590, can this issue be closed?