Help with resolving an unhealthy workspace agent

[Conor-Behard333]

I have coder deployed on a k8s cluster, I'm using an istio ingress gateway and have a domain pointing to the external IP and a virtual service to direct it to the coder pod matching port 443 to 80 essentially acting as a reverse proxy. The access url for coder is the same as the virtual service host.

During the deployment I pass in the certs needed to validate requests

When I go to https://custom-coder-domain/health everything is working (url is reachable, database, websockets and workspace proxy are working)

When I create a workspace, the kubernetes pod is created. However, after a few minutes of running the agent becomes unhealthy. None of the metrics (cpu usage, RAM usage, etc.) are visible. Also when I shell into either the coder pods or the workspace pod and try to curl https://custom-coder-domain
I get:

* Host custom-coder-domain:443 was resolved. 
* IPv6: (none) 
* IPv4: external IP of istio ingress gateway 
* Trying external IP of istio ingress gateway:443... 
* Connected to custom-coder-domain (external IP of istio ingress gateway) port 443 
* ALPN: curl offers h2,http/1.1 
* TLSv1.3 (OUT), TLS handshake, Client hello (1): 
* CAfile: /etc/ssl/certs/ca-certificates.crt 
* CApath: /etc/ssl/certs 
* TLSv1.3 (IN), TLS handshake, Server hello (2): 
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): 
* TLSv1.3 (IN), TLS handshake, Certificate (11): 
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate 
* Closing connection curl: (60) SSL certificate problem: unable to get local issuer certificate More details here: https://curl.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above.

Not sure if this is an issue with coder trying to access itself via the virtual service.

Would appreciate any help in resolving this issue

[matifali]

cc: @ericpaulsen if you have seen this before.

[Conor-Behard333]

After a bit more digging I've found that in the logs for the created workspace the error is happening when the pods does:
curl -fsSL --compressed https://custom-coder-domain/bin/coder-linux-amd64 -o coder

looked at the pods ssl folder and its doesn't have the needed certs to be able to verify the request. I assume the solution would be to make sure those certs have been added to the pod at its creation. Is there a way to do this?

[matifali]

Yes, you need to have the certs available in the pods image.

[Conor-Behard333]

I'm using the codercom/enterprise-base:ubuntu image, is there a way to either edit the init script or add a script to run "sudo update ca-certificates" as i've mounted the certs into the pod but want to run update ca-certs automatically

[matifali]

You can update the entrypoint in your template.

[Conor-Behard333]

are there any docs on this?

[matifali]

This depends on the template and how it is running the Coder agent init_script. for example on the Kubernetes template, you can modify the command to first to run sudo update-ca-certificates.

coder/examples/templates/kubernetes/main.tf

Line 273 in 5337a70

command = ["sh", "-c", coder_agent.main.init_script]

to

command: ["sh", "-c", "sudo update-ca-certificates && ${coder_agent.main.init_script}"]