converting nginx ingress to istio

[Baitanik]

we need istio as the gateway for metalk8s.. we already have installed istio and changed some of the things..
dex we changed to run on http...
wanted to know is there any example configuration by which we can easily change nginx to istio gateway, virtual service.
i see metalk8s-ui - running on http.
dex is running on https .
kubernertes-api running on https..

dex i changed to run on http... for kubernetes api (/api/kubernetes), I am not getting which places to change to make is run on http...

we want normal istio scenario .. where https/tls terminates on istio ingressgateway.(or sidecar level). and inside the cluster backend applications ,it is http only..
any idea, how to achieve this ? could someone please help

[NicolasT]

Hello,

Right now there are no plans to support Istio natively in MetalK8s or replace nginx-ingress by it (this is where the 'opinionated' from our tagline comes in 😉 ). However, it should definitely be possible to remove nginx-ingress (after deployment) and install Istio in a MetalK8s cluster, as apparently you did.

I assume you're looking to use the MetalK8s UI, and hence need to expose the Kubernetes API so the UI webapp can communicate with it. I'm a bit surprised with Istio you need all services that get exposed (through an Ingress object) to be plain HTTP services. Can't you proxy to a HTTPS service as well? 🤔

If indeed this is not possible, for kube-apiserver you'd need to let it listen without TLS on some port by using its --insecure-port flag. However, be advised kube-apiserver runs in the host network namespace (it's a static Pod managed by kubelet directly), so if you use --insecure-port without limiting access to e.g., 127.0.0.1, the non-TLS access would be available to the network.

By the way, you'll need to do the same for the Salt API service.

---

I'm not an Istio expert, but I think instead of changing all services to serve plain HTTP traffic, I'd be surprised there's no way to achieve proxying of the ingress controller to HTTPS endpoints... With nginx-ingress we had to apply some specific annotations to the Ingress objects to get this done: if you look at the metalk8s-ui-proxies-https Ingress object in the metalk8s-ui namespace, you'll see we set the nginx.ingress.kubernetes.io/backend-protocol annotation to HTTPS to proxy to kubernetes-api and salt-api. Similarly, on the metalk8s-ui-proxies-http Ingress in the same namespace, this annotation is set to HTTP to proxy the Prometheus and AlertManager APIs.

If Istio and its ingress controller would have similar functionality, you could add the relevant annotations to the existing Ingress objects, and change the kubernetes.io/ingress.class annotation to whichever is applicable for your Istio setup.

[Baitanik]

we are trying both options actually.. but straight forward I have not found a way like in nginx ingress we have one annotation..
There is something called destination rule in istio.. but getting some 503 error while doing that.. do trying to debug.. but have not got the reason for failure...

[Baitanik]

Tried to make the kubernetes-server api http port...
but now this call ...
https://10.105.58.133:8443/api/kubernetes/api/v1/namespaces?includeUninitialized=&pretty=&continue=&fieldSelector=metadata.name%3Dkube-system returning ...

kind: "Status", apiVersion: "v1", metadata: {}, status: "Failure",…}
apiVersion: "v1"
code: 404
details: {}
kind: "Status"
message: "the server could not find the requested resource"
metadata: {}
reason: "NotFound"
status: "Failure"

any idea. what could be the reson?
i tried with virtual service both prefix and regex
prefix: /api/kubernetes
regex: /api/kubernetes(/|$)(.*)
both it is giving same error..

[Baitanik]

the after giving rewrite in vs , that particular error is gone..
now in
for this call I am getting ::

paths: ["/api", "/api/v1", "/apis", "/apis/", "/apis/admissionregistration.k8s.io",…]
[0 … 99]
0: "/api"
1: "/api/v1"
2: "/apis"
3: "/apis/"
4: "/apis/admissionregistration.k8s.io"
5: "/apis/admissionregistration.k8s.io/v1"
6: "/apis/admissionregistration.k
.... 

Now for /graph , i dont see any ingress rule... probably it was taking from nginx regex...
but for istio vs, directly it is not taking... need to create another vs....

could you please tell
https://10.105.58.133:8443/graph
will be routed to which destination 'prometheus-operator-grafana' ?

[NicolasT]

I believe the /graph is added by Prometheus when you access the API URL from a browser, so /graph is not supposed to work (or be accessed). Basically, your proxy should proxy /api/prometheus(/|$)(.*) to the prometheus-api Service (in the metalk8s-ui namespace), which is a 'trick' we use because an Ingress can only refer to a Service in the same namespace, but we run Prometheus in metalk8s-monitoring. As such, the prometheus-api Service in the metalk8s-ui namespace is a Service of type ExternalName, whose externalName is set to prometheus-operator-prometheus.metalk8s-monitoring.svc.cluster.local, effectively allowing an Ingress in metalk8s-ui to proxy traffic to a Service in metalk8s-monitoring.

So, it appears you need some route/path rewriting for Prometheus API access as well.

[Baitanik]

Thanks @NicolasT ..
Yeah.. /graph was wrong... after adding rewrite in istio, it is working...
so right now /api/kubernertes and /api/prometheus , /grafana... those are working...
for /api/salt i have made the same changes, but not able to test..
from GUI , clicking on which tab or button .. it will make call to /api/salt ?

[NicolasT]

Sorry missed your message.

From memory, only provisioning a new volume interacts with the SaltStack API from the UI. However, I believe at login a token is minted as well, so things may already work. Take a look at the SaltAPI container logs (it's a container running in the salt-master-... Pod in kube-system).

[NicolasT]

@Baitanik To better understand how people in the community use MetalK8s, would you mind chiming in at #3085 to provide some intel about your use-case? Thanks!

[Baitanik]

Yeah sure @NicolasT .. will do that..
just an update now it is working.. we successfully replaced nginx now..
/api/salt was getting called at the time of login only.. did not notice that..

[Baitanik]

Incase some one ends up here.. trying to find steps to converting nginx ingress to istio...
i have tried to make the points where i was kind of stuck..
https://baitanikt.blogspot.com/2021/02/replace-nginx-ingress-with-istio.html