[QUESTION] Access Services From Host Without Mapping Ports to LoadBalancer

[kslacroix]

TL;DR What is the ip of the cluster as seen from the host? I'm trying to reproduce what you would get from minikube ip.

Hi, I'm migrating my local dev environment from minikube to k3d and I'm trying to get something like the minikube ip but for k3d. Here is my use case.

I'm trying to access services inside the cluster from my host using an ingress nginx controller. I use dnsmasq to map subdomains to an ip address (i.e. *.test.example.com) on macos. This is essentially like updating /etc/hosts except I don't need to specify every possible subdomain, which is handy since I need to run about 20 microservices in my cluster, with different subdomains for each. For reference, I followed this tutorial. This works well, I'm sure that's not where my problem lies.

I map *.test.example.com to 127.0.0.1, and add this to my k3d config.yml

ports:
  - port: 80:80
    nodeFilters:
      - loadbalancer
  - port: 0.0.0.0:443:443
    nodeFilters:
      - loadbalancer

I then add an ingress rule to a service x with host x.test.example.com. I can easily access service x from my host machine. But when I try to reach the service from another pod in the cluster I get the following error curl: (6) Couldn't resolve host 'x.test.example.com'. I'm guessing that's because there are no dns entries for my domain in the cluster, and even if there was, it would map to my localhost and not the cluster itself.

I need to use the same address inside and outside the cluster, making everything go through ingress which makes interpod communication use https instead of http (I can't use services meshes because our prod environment is still on docker-swarm... I know, lame).

With minikube, I can simply map *.test.example.com to the result of minikube ip. This removes the need to map ports from cluster to host, freeing ports 80 and 443 on localhost.

With k3d, I tried creating a cluster without the port mapping but with the loadBalancer enabled. This way, I thought I could run this docker container inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' k3d-local-serverlb to get the ip of the loadBalancer, and use this with dnsmasq. But it simply doesn't work.

I also tried with docker network inspect -f '{{(index .IPAM.Config 0).Gateway}}' k3d-local to get the docker network gateway address. But this also fails.

[iwilltry42]

Hi @kslacroix , thanks for starting this discussion!
What's your system? In docker-for-desktop (Win & Mac), the container IPs are not routable since the containers are running inside a VM (same as in Minikube btw.). minikube ip (when minikube is running in VM mode, which I think is/was the default) returns the IP of the VM.
On docker for desktop you can get it from docker's magic host record host.docker.internal if I'm not wrong.
However, then there wouldn't be anything listening there on 443/80, as the loadbalancer and K3s, etc. are running inside containers inside that VM.
I guess with some extra magix (routes, etc.) you could route traffic to the containers (via the IP you got from the docker commands already) via the Docker VM IP 🤔

Going via Ingress for service-to-service communication is quite unusual for real, but anyway, you'd need to add your Ingress names to CoreDNS or make CoreDNS forward DNS queries to your host system so that Pods can resolve the names.

[kslacroix]

Hi, I ended up not using ingress on my local machine because I can't get https anyway (which is the main reason why we use ingress for interpod communication). But I'm still struggling to communicate with my host machine—where my database is located. Is host.k3d.internals supposed to point to my host or to the vm? When I curl from inside the cluster, I get my default nginx backend in response... I thought this used to resolve to the host. Any idea?

Thank you for your response, I also realized that I would need to mess with configMaps for kube-dns and CoreDNS, but it simply wasn't worth it. It's already pretty unusual, I don't need to push that way.

[iwilltry42]

Hey, sorry for the late reply.
On Docker for Desktop, host.k3d.internal resolves to the Docker Gateway, similar to host.docker.internal.
Note though, that host.docker.internal is a different IP on your host and inside docker.

So it should definitely be possible to communicate with the database on your host system.
Did you connect to the correct port then? Because it feels weird that you got a response from some nginx? 🤔

[kslacroix]

I actually received this response because I have an nginx ingress controller (which I installed instead of traefik) on my cluster. I think the fact that host.k3d.internal resolved to the Gatway made the requests to the host actually loopback to the cluster itself. I explained how I fixed it in this thread.

host.k3d.internal actually didn't point to host.docker.internals. I changed the config map myself and now it works (haven't tried your fix yet)