[QUESTION/HELP] "Exposing Services" Guide does not work for macOS users #610

ss89 · 2021-05-15T15:13:32Z

ss89
May 15, 2021

Question / Where do you need Help?

I've been trying out the Ingress and NodePort service exposing guide from here: https://k3d.io/usage/guides/exposing_services/

However when i try to do the curl, i do get this result for both:
curl: (52) Empty reply from server

Ingress Guide:

❯ k3d cluster create --api-port 6550 -p "8081:80@loadbalancer" --agents 2
INFO[0000] Prep: Network
INFO[0004] Created network 'k3d-k3s-default' (7e2037005a5c3b9ac07e4aa041ee22552fa33983c029932311aea10ca68a0fea)
INFO[0004] Created volume 'k3d-k3s-default-images'
INFO[0005] Creating node 'k3d-k3s-default-server-0'
INFO[0005] Creating node 'k3d-k3s-default-agent-0'
INFO[0005] Creating node 'k3d-k3s-default-agent-1'
INFO[0005] Creating LoadBalancer 'k3d-k3s-default-serverlb'
INFO[0005] Starting cluster 'k3s-default'
INFO[0005] Starting servers...
INFO[0005] Starting Node 'k3d-k3s-default-server-0'
INFO[0016] Starting agents...
INFO[0016] Starting Node 'k3d-k3s-default-agent-0'
INFO[0026] Starting Node 'k3d-k3s-default-agent-1'
INFO[0036] Starting helpers...
INFO[0036] Starting Node 'k3d-k3s-default-serverlb'
INFO[0039] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
INFO[0045] Successfully added host record to /etc/hosts in 4/4 nodes and to the CoreDNS ConfigMap
INFO[0045] Cluster 'k3s-default' created successfully!
INFO[0045] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false
INFO[0045] You can now use it like this:
kubectl config use-context k3d-k3s-default
kubectl cluster-info
❯ kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
❯ kubectl create service clusterip nginx --tcp=80:80
service/nginx created
❯ kubectl apply -f nginx-ingress.yml
ingress.networking.k8s.io/nginx created
❯ curl localhost:8081/
curl: (52) Empty reply from server
❯ docker ps
CONTAINER ID   IMAGE                      COMMAND                  CREATED              STATUS              PORTS                                                           NAMES
32bfb40bf97b   rancher/k3d-proxy:v4.4.3   "/bin/sh -c nginx-pr…"   About a minute ago   Up 45 seconds       0.0.0.0:8081->80/tcp, :::8081->80/tcp, 0.0.0.0:6550->6443/tcp   k3d-k3s-default-serverlb
b2a597a53065   rancher/k3s:latest         "/bin/k3s agent"         About a minute ago   Up 56 seconds                                                                       k3d-k3s-default-agent-1
2d9a6f12d642   rancher/k3s:latest         "/bin/k3s agent"         About a minute ago   Up About a minute                                                                   k3d-k3s-default-agent-0
0850a3710c66   rancher/k3s:latest         "/bin/k3s server --t…"   About a minute ago   Up About a minute                                                                   k3d-k3s-default-server-0

NodePort Guide:

❯ k3d cluster create mycluster -p "8082:30080@agent[0]" --agents 2
INFO[0000] Prep: Network
INFO[0003] Created network 'k3d-mycluster' (86e83f433c95f5767d6de5752cbe0ad082df21793baec4a46ea6bc914ce2c653)
INFO[0003] Created volume 'k3d-mycluster-images'
INFO[0004] Creating node 'k3d-mycluster-server-0'
INFO[0004] Creating node 'k3d-mycluster-agent-0'
INFO[0004] Creating node 'k3d-mycluster-agent-1'
INFO[0005] Creating LoadBalancer 'k3d-mycluster-serverlb'
INFO[0005] Starting cluster 'mycluster'
INFO[0005] Starting servers...
INFO[0005] Starting Node 'k3d-mycluster-server-0'
INFO[0015] Starting agents...
INFO[0015] Starting Node 'k3d-mycluster-agent-0'
INFO[0026] Starting Node 'k3d-mycluster-agent-1'
INFO[0035] Starting helpers...
INFO[0035] Starting Node 'k3d-mycluster-serverlb'
INFO[0037] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
INFO[0044] Successfully added host record to /etc/hosts in 4/4 nodes and to the CoreDNS ConfigMap
INFO[0044] Cluster 'mycluster' created successfully!
INFO[0044] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false
INFO[0044] You can now use it like this:
kubectl config use-context k3d-mycluster
kubectl cluster-info
❯ kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
❯ kubectl apply -f nginx-service.yml
service/nginx created
❯ curl localhost:8082/
curl: (52) Empty reply from server
❯ docker ps
CONTAINER ID   IMAGE                      COMMAND                  CREATED              STATUS          PORTS                                         NAMES
19f1322c0ad5   rancher/k3d-proxy:v4.4.3   "/bin/sh -c nginx-pr…"   About a minute ago   Up 28 seconds   80/tcp, 0.0.0.0:53030->6443/tcp               k3d-mycluster-serverlb
ef2917227c29   rancher/k3s:latest         "/bin/k3s agent"         About a minute ago   Up 39 seconds                                                 k3d-mycluster-agent-1
05529d09730d   rancher/k3s:latest         "/bin/k3s agent"         About a minute ago   Up 49 seconds   0.0.0.0:8082->30080/tcp, :::8082->30080/tcp   k3d-mycluster-agent-0
11244303329c   rancher/k3s:latest         "/bin/k3s server --t…"   About a minute ago   Up 59 seconds                                                 k3d-mycluster-server-0

k3d version:

❯ k3d --version
k3d version v4.4.3
k3s version latest (default)

Scope of your Question

Can the guide be updated to include instructions for mac users?

Answered by iwilltry42

May 17, 2021

UPDATE: fixed by docker system reset (see comments)

Hi @ss89 , thanks for starting this discussion!
Actually, those instructions are OS-agnostic, so there shouldn't be any difference in how port-forwarding works in Docker for Mac.
Can you verify, that nginx is actually running there after you've created the deployment (kubectl get pods)?
The commands you typed look correct, but it seems that it may just not be running yet.

View full answer

iwilltry42 · 2021-05-17T08:53:24Z

iwilltry42
May 17, 2021
Maintainer

UPDATE: fixed by docker system reset (see comments)

Hi @ss89 , thanks for starting this discussion!
Actually, those instructions are OS-agnostic, so there shouldn't be any difference in how port-forwarding works in Docker for Mac.
Can you verify, that nginx is actually running there after you've created the deployment (kubectl get pods)?
The commands you typed look correct, but it seems that it may just not be running yet.

8 replies

ss89 May 17, 2021
Author

Here's the info that you requested. Let me know if i can add anything.

❯ curl localhost:8081 -vvv
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8081 (#0)
> GET / HTTP/1.1
> Host: localhost:8081
> User-Agent: curl/7.64.1
> Accept: */*
>
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server
* Closing connection 0

❯ kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
nginx-6799fc88d8-7jq52   1/1     Running   0          77s

❯ kubectl get services

NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.43.0.1       <none>        443/TCP   2m6s
nginx        ClusterIP   10.43.210.120   <none>        80/TCP    90s

❯ kubectl get ingress
NAME    CLASS    HOSTS   ADDRESS                            PORTS   AGE
nginx   <none>   *       172.20.0.2,172.20.0.3,172.20.0.4   80      105s

❯ docker logs k3d-k3s-default-serverlb
2021-05-17T14:47:20Z k3d-k3s-default-serverlb confd[10]: INFO Backend set to env
2021-05-17T14:47:20Z k3d-k3s-default-serverlb confd[10]: INFO Starting confd
2021-05-17T14:47:20Z k3d-k3s-default-serverlb confd[10]: INFO Backend source(s) set to
2021-05-17T14:47:20Z k3d-k3s-default-serverlb confd[10]: INFO /etc/nginx/nginx.conf has md5sum 9c9065b8e74f4b01f6eed5a7af1141b6 should be fe7d42b4dcd13d37fb5a0a18bf02cc0f
2021-05-17T14:47:20Z k3d-k3s-default-serverlb confd[10]: INFO Target config /etc/nginx/nginx.conf out of sync
2021-05-17T14:47:20Z k3d-k3s-default-serverlb confd[10]: INFO Target config /etc/nginx/nginx.conf has been updated
===== Initial nginx configuration =====
error_log stderr notice;

worker_processes auto;
events {
  multi_accept on;
  use epoll;
  worker_connections 1026;
}

stream {

  #######
  # TCP #
  #######
  upstream server_nodes_6443 {
    server k3d-k3s-default-server-0:6443 max_fails=1 fail_timeout=10s;
  }

  server {
    listen        6443;
    proxy_pass    server_nodes_6443;
    proxy_timeout 600;
    proxy_connect_timeout 2s;
  }
  upstream server_nodes_80 {
    server k3d-k3s-default-server-0:80 max_fails=1 fail_timeout=10s;
  }

  server {
    listen        80;
    proxy_pass    server_nodes_80;
    proxy_timeout 600;
    proxy_connect_timeout 2s;
  }

  #######
  # UDP #
  #######
}
=======================================
2021/05/17 14:47:20 [notice] 16#16: using the "epoll" event method
2021/05/17 14:47:20 [notice] 16#16: nginx/1.19.10
2021/05/17 14:47:20 [notice] 16#16: built by gcc 10.2.1 20201203 (Alpine 10.2.1_pre1)
2021/05/17 14:47:20 [notice] 16#16: OS: Linux 5.10.25-linuxkit
2021/05/17 14:47:20 [notice] 16#16: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2021/05/17 14:47:20 [notice] 16#16: start worker processes
2021/05/17 14:47:20 [notice] 16#16: start worker process 17
2021/05/17 14:47:20 [notice] 16#16: start worker process 18
2021/05/17 14:47:20 [notice] 16#16: start worker process 19
2021/05/17 14:47:20 [notice] 16#16: start worker process 20
2021/05/17 14:48:11 [error] 17#17: *107 connect() failed (111: Connection refused) while connecting to upstream, client: 172.20.0.1, server: 0.0.0.0:80, upstream: "172.20.0.2:80", bytes from/to client:0/0, bytes from/to upstream:0/0

ss89 May 17, 2021
Author

Ok this seems interesting: After resetting my docker for mac completely and retrying the exact same steps it worked. Strange thing...

iwilltry42 May 17, 2021
Maintainer

Here's the info that you requested. Let me know if i can add anything.

Great, so from this we know, that everything up until cluster entry works 👍
Then next thing would be to check traefik and the servicelb pods in the kube-system namespace, as they seem to have a problem.

After resetting my docker for mac completely and retrying the exact same steps it worked

It could be that your Docker VM ran out of disk space and thus k3s failed to pull the images required for the servicelb and/or traefik to run, which in turn lead to your issue 🤔
When you do a full reset of docker, that also prunes the overlayfs and the image store, so that could very well be (and we've seen this before) 🤔

ss89 May 17, 2021
Author

Pretty sure i had enough free space because it was telling me it used only 13 of my assigned 64gb. Anyhow i'm happy that it works now and that i could start tinkering with it now. 👍🏻
thanks for your help!

sumeetpatil Jun 26, 2023

I had the same issue. Restarting docker worked for me.

emeraldhieu · 2023-04-18T10:28:38Z

emeraldhieu
Apr 18, 2023

I have the same issue. Restarting Docker Desktop doesn't work for me.

2023/04/18 08:42:56 [error] 49#49: *43 connect() failed (111: Connection refused) while connecting to upstream, client: 172.20.0.1, server: 0.0.0.0:50001, upstream: "172.20.0.3:50001", bytes from/to client:0/0, bytes from/to upstream:0/0

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[QUESTION/HELP] "Exposing Services" Guide does not work for macOS users #610

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 8 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

[QUESTION/HELP] "Exposing Services" Guide does not work for macOS users #610

ss89 May 15, 2021

Question / Where do you need Help?

Scope of your Question

Replies: 2 comments · 8 replies

iwilltry42 May 17, 2021 Maintainer

ss89 May 17, 2021 Author

ss89 May 17, 2021 Author

iwilltry42 May 17, 2021 Maintainer

ss89 May 17, 2021 Author

sumeetpatil Jun 26, 2023

emeraldhieu Apr 18, 2023

ss89
May 15, 2021

Replies: 2 comments 8 replies

iwilltry42
May 17, 2021
Maintainer

ss89 May 17, 2021
Author

ss89 May 17, 2021
Author

iwilltry42 May 17, 2021
Maintainer

ss89 May 17, 2021
Author

emeraldhieu
Apr 18, 2023