-
-
Notifications
You must be signed in to change notification settings - Fork 394
Description
Would you like to work on this feature?
yes
What problem are you trying to solve?
When a cluster rotates its root CA, the kubelet updates /var/run/secrets/kubernetes.io/serviceaccount/ca.crt atomically in every pod's projected service-account volume. But Config::incluster() reads that
file exactly once, into Config.root_cert: Vec<Vec<u8>>, and bakes the bytes into a rustls::RootCertStore when the client is built. After rotation, every new TLS handshake to the apiserver fails verification. The process has to restart.
Existing HTTP/2 connections keep working (they already handshook), so a running watch stream doesn't notice anything is wrong. Then it reconnects, and can't.
On clusters with short-lived CA roots, this effectively caps pod lifetime at CA lifetime: any pod that outlives the CA it started under will fail its next reconnect.
Describe the solution you'd like
Handle root CA rotation using the same mechanism that #768 introduced to handle token rotation: poll the CA file for changes and add a rustls::ServerCertVerifier that supports this.
I have a working patch here that I'd be happy to contribute if you'd accept a contribution for this: main...chrnorm:kube:incluster-ca-reload
Describe alternatives you've considered
As a workaround for this, I've implemented a wrapper library with my own ConfigExt::rustls_https_connector. I'd love to upstream an improvement for this though so that other folks can benefit.
Documentation, Adoption, Migration Strategy
No user action needed. Client::try_default() in a pod starts picking up CA rotation on its own. The field doc comment covers the behaviour; a sentence in the in-cluster config docs wouldn't hurt though.
The one wrinkle: Config isn't #[non_exhaustive], so adding a field breaks anyone who constructs it with a struct literal and no ..Default::default(). It looks like fields have been added in minor bumps before though. Is that fine here, or would you want #[non_exhaustive] landed first?
Target crate for feature
kube-client