Config reverse proxy

[Mefevre]

Hello community,

I set up a Teleport cluster, added nodes and applications.
The cluster is located behind an Apache2 reverse proxy.

So far everything is going well. I am able to connect to the cluster through the reverse proxy and connect to the applications and nodes.

The problem comes from the Teleport client (tsh.exe).
I can't connect to the cluster when I use the reverse proxy. When I tap directly on the cluster, no problem, everything works normally.

Do you have any idea where the problem could be coming from?

Error send by client tsh

ERROR: connection error: desc = "transport: Error while dialing: failed to dial: failed to switch Protocols 502"

Config of apache2

<VirtualHost *:443>
    ServerName bastion.mondomaine.fr

    ErrorLog ${APACHE_LOG_DIR}/error.bastion.log
    CustomLog ${APACHE_LOG_DIR}/access.bastion.log combined

    SSLEngine on
    SSLProtocol TLSv1.3 TLSv1.2
    SSLCertificateFile /etc/letsencrypt/live/mondomaine.fr/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/mondomaine.fr/privkey.pem

    # On fait du proxy vers un autre serveur en https
    SSLProxyEngine On
    SSLProxyCheckPeerCN Off
    SSLProxyCheckPeerName Off
    SSLProxyCheckPeerExpire off
    SSLProxyVerify none

    # On transmet notre certificat au serveur distant (crt + key combiné)
    #  SSLProxyMachineCertificateFile "/etc/ssl/certs/https/client.pem"
    ProxyPass /v1/ wss://192.168.1.235/v1/
    ProxyPass / https://192.168.1.235/
    ProxyPassReverse / https://192.168.1.235/
    ProxyPreserveHost On
</VirtualHost>

Config cluster teleport

version: v3
teleport:
  nodename: bastion
  data_dir: /var/lib/teleport
  log:
    output: stderr
    severity: DEBUG
    format:
      output: text
  ca_pin: ""
  diag_addr: ""
auth_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3025
  cluster_name: bastion.mondomain.fr
  proxy_listener_mode: multiplex
  authentication:
    second_factor: off
ssh_service:
  enabled: "yes"
  commands:
  - name: hostname
    command: [hostname]
    period: 1m0s
proxy_service:
  enabled: "yes"
  web_listen_addr: 0.0.0.0:443
  public_addr: bastion.mondomain.fr:443
  https_keypairs:
  - key_file: /var/lib/teleport/mondomain.fr.key
    cert_file: /var/lib/teleport/mondomain.fr.crt
  https_keypairs_reload_interval: 0s
  acme: {}

Log in proxy

37.65.47.40 - - [12/Feb/2024:14:45:10 +0100] "POST /v1/webapi/ssh/certs HTTP/1.1" 200 13078 "-" "Go-http-client/1.1"

==> /var/log/apache2/error.bastion.log <==
[Mon Feb 12 14:45:10.593179 2024] [proxy:error] [pid 99063:tid 140279294174784] [client 37.65.47.40:47621] AH00898: Unexpected Upgrade: alpn-ping (expecting n/a) returned by /webapi/connectionupgrade

==> /var/log/apache2/access.bastion.log <==
37.65.47.40 - - [12/Feb/2024:14:45:10 +0100] "GET /webapi/connectionupgrade HTTP/1.1" 502 5088 "-" "Go-http-client/1.1"

==> /var/log/apache2/error.bastion.log <==
[Mon Feb 12 14:45:10.601255 2024] [proxy:error] [pid 99063:tid 140279285782080] [client 37.65.47.40:49406] AH00898: Unexpected Upgrade: alpn-ping (expecting n/a) returned by /webapi/connectionupgrade

[Mefevre]

version teleport : Teleport v15.0.1 git:v15.0.1-0-gd347510 go1.21.6
version tsh : Teleport v15.0.1 git: go1.21.6

[webvictim]

I'm not familiar with how to do this using Apache - it's much easier using something like nginx (guide here: #26445) or Caddy.

If you insist on using Apache, try this: https://serverfault.com/a/1123715

[Mefevre]

thank you for your return @webvictim !
I have already come across this suggestion. I tested it, but without success for me.

I want to stay on Apache2 because there is a lot of configuration in place. and that switching to nginx would take a lot of time.

[webvictim]

I got this working after a bit of trial and error. Here are my configs from an Ubuntu 22.04 server:

/etc/apache2/sites-available/001-teleport.conf:

<VirtualHost *:443>
    ServerName teleport.example.com

    ErrorLog /var/log/apache2/teleport.error.log
    CustomLog /var/log/apache2/teleport.access.log combined

    SSLEngine on
    SSLProxyEngine on
    SSLProtocol TLSv1.3 TLSv1.2
    SSLCertificateFile /etc/letsencrypt/live/teleport.example.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/teleport.example.com/privkey.pem

    # Replace localhost:3080 with the IP:port of your Teleport server if needed
    # The upgrade=ANY line is needed to pass custom connection upgrades used by Teleport
    ProxyPass / https://localhost:3080/ upgrade=ANY
    ProxyPassReverse / https://localhost:3080/
    ProxyPreserveHost on
    ProxyRequests off
</VirtualHost>

/etc/teleport.yaml:

version: v3
teleport:
  nodename: ip-172-31-32-65
  data_dir: /var/lib/teleport
  log:
    output: stderr
    severity: DEBUG
    format:
      output: text
auth_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3025
  proxy_listener_mode: multiplex
  cluster_name: teleport.example.com
  authentication:
    second_factor: on
    webauthn:
      rp_id: teleport.example.com
ssh_service:
  enabled: "yes"
  commands:
  - name: hostname
    command: [hostname]
    period: 1m0s
proxy_service:
  enabled: "yes"
  # listen internally on port 3080
  web_listen_addr: 0.0.0.0:3080
  # public address is Apache port 443
  public_addr: teleport.example.com:443
  # trust X-Forwarded-For headers sent by reverse proxy (for correct remote IPs in Teleport logs)
  trust_x_forwarded_for: true

Enable modules mod_proxy mod_ssl mod_rewrite and site 001-teleport, then restart Apache:

sudo a2enmod proxy ssl rewrite
sudo a2ensite 001-teleport
sudo systemctl restart apache2

[webvictim]

This solution is also published here: #38426

[jayjay3108]

Thank you very much for dropping your yaml. This helped me to finally set up webauthn.

[Mefevre]

In your tests, have you tested the connection with a tsh client ?

[webvictim]

Good point, I have not - I'll give it a try later. The config for arbitrary non-websocket connection upgrades is likely to be considerably harder to write!

I know that the alpn-ping test from the TLS routing troubleshooting doesn't work with this config so tsh will probably not, but there is support coming for solely using websockets soon which should help.

[webvictim]

OK, turns out this is actually very easy as long as you're using Apache 2.4.47 or higher.

Just replace your ProxyPass directive with this:

ProxyPass / https://localhost:3080/ upgrade=ANY

The upgrade=ANY line means that the proxy will pass through any connection upgrades to the backend. You can then also remove all the mod_rewrite stuff which makes the config nicer.

[Mefevre]

@webvictim
I confirm that the last information you sent me is the correct one.
This solved my problem with the tsh client

Great solution

ProxyPass / https://localhost:3080/ upgrade=ANY