TLS Handshake Failure

[jdrew82]

I've searched high and low for any answers regarding this issue and have come up with basically nothing so I'm opening this issue. I've setup a k3s cluster using this repo as a starting point. During my work on adding additional apps and functionality to the cluster I've now ran into an issue where the cluster is unusable. When attempting to reconcile with flux I get the following error:

flux reconcile source git home-kubernetes --verbose
► annotating GitRepository home-kubernetes in flux-system namespace
:heavy_check_mark: GitRepository annotated
◎ waiting for GitRepository reconciliation
✗ GitRepository reconciliation failed: 'failed to checkout and determine revision: unable to clone 'https://github.com/jdrew82/homelab.git': Get "https://github.com/jdrew82/homelab.git/info/refs?service=git-upload-pack": remote error: tls: handshake failure'

Looking in the Flux source-controller pod I see the following:

{"level":"error","ts":"2023-09-24T20:53:01.662Z","msg":"failed to checkout and determine revision: unable to clone 'https://github.com/jdrew82/homelab.git': Get \"https://github.com/jdrew82/homelab.git/info/refs?service=git-upload-pack\": remote error: tls: handshake failure","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository","GitRepository":{"name":"home-kubernetes","namespace":"flux-system"},"namespace":"flux-system","name":"home-kubernetes","reconcileID":"490dd6a7-f5e6-4aec-a1e6-c083af425469","error":"failed to checkout and determine revision: unable to clone 'https://github.com/jdrew82/homelab.git': Get \"https://github.com/jdrew82/homelab.git/info/refs?service=git-upload-pack\": remote error: tls: handshake failure"}
{"level":"error","ts":"2023-09-24T20:53:01.673Z","msg":"Reconciler error","controller":"gitrepository","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"GitRepository","GitRepository":{"name":"home-kubernetes","namespace":"flux-system"},"namespace":"flux-system","name":"home-kubernetes","reconcileID":"490dd6a7-f5e6-4aec-a1e6-c083af425469","error":"failed to checkout and determine revision: unable to clone 'https://github.com/jdrew82/homelab.git': Get \"https://github.com/jdrew82/homelab.git/info/refs?service=git-upload-pack\": remote error: tls: handshake failure"}

From my investigation it appears that ALL TLS connections are seeing this same issue. When I attempt to curl to my Git repo from a netconnect container for troubleshooting I see the following:

kubectl run tmp-shell --rm -i --tty --image nicolaka/netshoot -- /bin/bash
If you don't see a command prompt, try pressing enter.
tmp-shell:~# curl https://github.com/jdrew82/homelab.git/info/refs?service=git-upload-pack\
> ^C
tmp-shell:~# curl https://github.com/jdrew82/homelab.git/info/refs?service=git-upload-pack
curl: (35) OpenSSL/3.1.0: error:0A000410:SSL routines::sslv3 alert handshake failure
tmp-shell:~# curl -Ivk https://github.com/jdrew82/homelab.git/info/refs?service=git-upload-pack
*   Trying 104.21.91.88:443...
* Connected to github.com (104.21.91.88) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, handshake failure (552):
* OpenSSL/3.1.0: error:0A000410:SSL routines::sslv3 alert handshake failure
* Closing connection 0
curl: (35) OpenSSL/3.1.0: error:0A000410:SSL routines::sslv3 alert handshake failure
tmp-shell:~# curl -Ivk https://github.com/jdrew82/homelab.git
*   Trying 104.21.91.88:443...
* Connected to github.com (104.21.91.88) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, handshake failure (552):
* OpenSSL/3.1.0: error:0A000410:SSL routines::sslv3 alert handshake failure
* Closing connection 0
curl: (35) OpenSSL/3.1.0: error:0A000410:SSL routines::sslv3 alert handshake failure

It looks to me like there is some proxy between the containers and the outside world that is handling the TLS connection and breaking it but I'm unable to determine specifically what is being used for egress in the cluster and where the potential break is happening. I'm aware that Cilium is used for the mesh networking in the cluster and I'm assuming that k8s-gateway is also used in some manner but nothing appears to be wrong there. Any assistance/guidance on where I should look or what I should do to troubleshoot this issue would be greatly appreciated.

[onedr0p]

Did anything change in your setup from when it wasn't working to now? Anything external to the cluster?

I'm on vacation so I probably won't be of too much help for another week or two.

[jdrew82]

Did anything change in your setup from when it wasn't working to now? Anything external to the cluster?

I'm on vacation so I probably won't be of too much help for another week or two.

Nope, my external network is fine. It seems to be limited specifically to the cluster itself. No worries on helping right now. Appreciate that you responded on your vacation. Go enjoy! This can wait.

[onedr0p]

Were you able to get anywhere with this or still having the same issues? Maybe try to set DNS on your nodes to directly speak with a upstream server instead of a local DNS server to debug.

[jdrew82]

I have not had a chance to work on this due to work. I'm not sure what the DNS would have to do with this particular issue. The issue appears to be that there is some proxy or something on the outbound traffic path within the cluster that is missing certificates or something like that as it's a TLS handshake failure. As I'm still learning k8s and this particular setup that you created here (thank you for that btw), I'm a bit stumped upon what the particular traffic flow is for outbound. I know there are NGINX ingress controllers and Cilium is the service mesh but that's about it.

[onedr0p]

I mention DNS because if your local DNS server is rewriting outbound requests (on purpose or unintentionally) to somewhere else that could be the issue or the issue appears as if you are getting man in the middled via a proxy of some sort. It might be wise to try and reduce your network complexity and continue to test.

I have converted this into a discussion as I am not sure this is an issue with the template, but maybe more your local network.

[jdrew82]

I get what you're saying but I still don't think it's a DNS issue. Like I said previously, my local machines work fine, this is only happening within the cluster itself. Just to cover all bases though, here's the resolv.conf that is identical on all three of my nodes:

cat /etc/resolv.conf
search {$SECRET_DOMAIN}
nameserver 1.1.1.1
nameserver 1.0.0.1
nameserver 8.8.8.8

*note: SECRET_DOMAIN var is actual domain, I just choose to use the var form here to match what the templates use.

[jdrew82]

So just to be clear, it sounds like you're saying that there is no outbound proxy in the cluster and you feel it's something external to the cluster?

[onedr0p]

That is correct. I'm also fairly confident about that because I've yet seen anyone else report this issue.

[onedr0p]

@jdrew82 If you are still curious about using this project I think this might solve your problem, basically you need to remove the search domain from /etc/resolv.conf

#635 (comment)

[jdrew82]

Thanks for the info. I ended up blowing away the cluster for now. When I rebuild it I'll look here if I'm still having issues.