[BUG] REVERSE_PROXY_WS not working in KUBERNETES_MODE

[ANBAL534]

What happened?

Websockets are not working in kubernetes mode of bunkerweb. Calls towards webockets endpoints gets a 426 Upgrade Required.

I noticed this when setting up a home assistant server in my local kubernetes cluster running on k3s in x86_64 servers (good old trusty proliants)

It is reproducible on baremetal, in a simple k3d for testing and it is not reproducible on a regular docker compose deployment of bunkerweb.

How to reproduce?

I managed to reproduce with versions 1.5.6, 1.5.5, 1.5.4, I have not tested with older versions.

You can use k3d to create a simple testing kubernetes cluster, remember to disable traefik:

1. k3d cluster create -p "80:80@loadbalancer" -p "443:443@loadbalancer" -s 1 --k3s-arg "--disable=traefik@server:*" --wait
2. My configurations for the tests: manifests.zip
3. Apply bunkerweb: kubectl apply -f bunkerweb.yaml
4. Apply test websocket echo server: kubectl apply -f wstest.yaml
5. Use any tool to test the websocket (I used websocat and curl):
   4.1. websocat ws://wstest.127.0.0.1.nip.io/
   4.2. curl -i -N -H 'Connection: "Upgrade"' -H "Upgrade: websocket" -H "Host: wstest.127.0.0.1.nip.io" http://wstest.127.0.0.1.nip.io/

To compare with the regular docker compose deployment, you can docker compose up my provided docker-compose.yaml file by only changing the proxy host target IP to where the websocket server is running (ex.: your local PC IP)

Then execute the websocket server locally with docker:
docker run -p 3000:3000 elegantmonkeys/websockets-demo

And test with websocat:
websocat ws://127.0.0.1:3000

You will be able to see that the websocket server from k8s will return a 426 where the one going through regular docker compose works as intended.

Configuration file(s) (yaml or .env)

[manifests.zip](https://github.com/bunkerity/bunkerweb/files/15142800/manifests.zip)
The bunkerweb and wstest manifests for the kubernetes test and docker compose file to compare the functionality with a working one.

Relevant log output

N/A

BunkerWeb version

1.5.6

What integration are you using?

Kubernetes

Linux distribution (if applicable)

Almalinux 9.3

Removed private data

• I have removed all private data from the configuration file and the logs

Code of Conduct

• I agree to follow this project's Code of Conduct

[ANBAL534]

Keeping investigating I found that the scheduler is not updating correctly the NGINX config files:
In the non-autoconf docker compose deployment the configuration for the domain is correct:

But in the kubernetes mode deployment, it is not updated:

[TheophileDiot]

Hi @ANBAL534, thank you for opening this issue. Do you have any scheduler/bunkerweb logs that could help us investigate as well on this ? Thank you

[ANBAL534]

logs.log
Here there are the logs for all bunkerweb components deployed in the test cluster I made following my steps in the original post.

In the DB the setting is picked correctly, but doesnt translates in the the correct nginx configuration:

[fl0ppy-d1sk]

Hello @ANBAL534,

Can you try to rename your annotation bunkerweb.io/wstest.127.0.0.1.nip.io_REVERSE_PROXY_WS to bunkerweb.io/wstest.127.0.0.1.nip.io_REVERSE_PROXY_WS_1 ?

[fl0ppy-d1sk]

Closing this as it should do the trick. Don't hesitate to open a new issue if needed.

[ANBAL534]

Indeed, that solved the issue.

But in the documentation it didnt specified the necessity of such modification to the annotation.

Are there other settings that requires this?