Clarification on rotation of custom Intermidary certs

[davehouser1]

Hello, I have posted about using custom intermediary certs for K3s in the past. I already understand its best to use the custom certs at deployment time on your main K3s server node. I have already tested this and it works, certs are signed properly, added etcd nodes and agent nodes use the certs as well, all good.

I am looking for clarification on what happens when these certs expire, as in the certs that I generated to be used at deployment. From this page I am referring to this section

If you want to use existing root and intermediate CA certificates, provide the following files:

root-ca.pem
intermediate-ca.pem
intermediate-ca.key

Lets say the expiration date is set to 365 days for these. Does the expiration date matter? Are these certs used only once to generate K3s certs then never again? Or does the expiration date matter and these certs need to be rotated as well?

I read through this section It seems that the expiration time does matter, and I need to generate new ones, stage them in a specific directory, then run the k3s certificate rotate-ca --force to rotate them. It also seems that I can only use generate intermediary certs if they are generated form the same root CA the original ones were generated form. Is that true? I would plan to generate a new root-ca.pem, intermediate-ca.pem, intermediate-ca.key before the expiration time, load them into a K3s main etcd server node, and rotate. Also, does it matter which node I use?

Can someone confirm this?

[brandond]

If you provide your own root cert, that is the root of trust for your cluster. If it expires, all trust for TLS certificates within your cluster will break. All certificates in the chain (root, intermediates, and leaf) need to be valid in order for certificates to work.

Intermediate and leaf certificates can be rotated fairly effortlessly, but because the hash of the root CA is encoded in the secure join token, any changes to that cert (including extending its validity) will require updating tokens on nodes.

See:

k3s/pkg/clientaccess/token.go

Lines 160 to 165 in 696dd9d

	// hashCA returns the hex-encoded SHA256 digest of a CA bundle.
	// If the certificate bundle contains only a single certificate, a legacy hash is generated from
	// the literal bytes of the file; usually a PEM-encoded self-signed cluster CA certificate.
	// If the certificate bundle contains more than one certificate, the hash is instead generated
	// from the DER-encoded root certificate in the bundle. This allows for rotating or renewing the
	// cluster CA, as long as the root CA remains the same.

[davehouser1]

Thanks Brandon for the details.
I assume this is the process to rotate root-ca.pem
intermediate-ca.pem, intermediate-ca.key certs?
We plan to use the same root CA each time to generate new certs, so I assume we can rotate non-distributively? Referencing below from the same documentation:

A cluster that has been started with custom CA certificates can renew or rotate the CA certificates and keys non-disruptively, as long as the same root CA is used.

[brandond]

yep