Skip to content

Reload in-cluster CA bundle on rotation (rustls-tls) #1953

@chrnorm

Description

@chrnorm

Would you like to work on this feature?

yes

What problem are you trying to solve?

When a cluster rotates its root CA, the kubelet updates /var/run/secrets/kubernetes.io/serviceaccount/ca.crt atomically in every pod's projected service-account volume. But Config::incluster() reads that
file exactly once, into Config.root_cert: Vec<Vec<u8>>, and bakes the bytes into a rustls::RootCertStore when the client is built. After rotation, every new TLS handshake to the apiserver fails verification. The process has to restart.

Existing HTTP/2 connections keep working (they already handshook), so a running watch stream doesn't notice anything is wrong. Then it reconnects, and can't.

On clusters with short-lived CA roots, this effectively caps pod lifetime at CA lifetime: any pod that outlives the CA it started under will fail its next reconnect.

Describe the solution you'd like

Handle root CA rotation using the same mechanism that #768 introduced to handle token rotation: poll the CA file for changes and add a rustls::ServerCertVerifier that supports this.

I have a working patch here that I'd be happy to contribute if you'd accept a contribution for this: main...chrnorm:kube:incluster-ca-reload

Describe alternatives you've considered

As a workaround for this, I've implemented a wrapper library with my own ConfigExt::rustls_https_connector. I'd love to upstream an improvement for this though so that other folks can benefit.

Documentation, Adoption, Migration Strategy

No user action needed. Client::try_default() in a pod starts picking up CA rotation on its own. The field doc comment covers the behaviour; a sentence in the in-cluster config docs wouldn't hurt though.

The one wrinkle: Config isn't #[non_exhaustive], so adding a field breaks anyone who constructs it with a struct literal and no ..Default::default(). It looks like fields have been added in minor bumps before though. Is that fine here, or would you want #[non_exhaustive] landed first?

Target crate for feature

kube-client

Metadata

Metadata

Assignees

Labels

clientkube Client related

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions