Add letsencrypt clusterissuers + hetzner webhook + reflector + auto wildcard cert generation

[jaumebarber]

Add LetsEncrypt ClusterIssuers + Hetzner webhook + Reflector + auto wildcard cert generation

This PR adds LetsEncrypt ClusterIssuers Helm Chart installation so that the cluster is set up and ready to generate TLS certs by passing both HTTP-01 and DNS-01 ACME challenges. While the former requires no additional dependencies apart from the ClusterIssuer itself, the latter requires the Hetzner API Webhook for the Cert Manager, which is also installed and set up by another Helm Chart included in cluster_issuers.yaml.tpl.

I've created new var.hetzner_dns_api_token to set up Hetzner DNS API Token in there (Hetzner delegated DNS Zone is a must for this to work), and that variable is then encoded and passed on to cluster_issuers.yaml.tpl to generate hetzner-dns-secret, which the webhook uses to pass the challenge.

Finally, I've included a wildcard_cert.yaml.tpl which triggers auto wildcard cert secret generation using common_name, a template variable obtained from regexing the var.base_domain FQDN to obtain the root domain and request *.example.com, and flagged Reflector annotations for the secret. Reflector is a little utility that replicates annotated secrets in all or specifically selected namespaces. This is especially useful for TLS certs since they are often needed in more than one namespace. Both the ClusterIssuers and the Reflector installation are obviously flagged through var.enable_cluster_issuers and var.enable_reflector and disabled by default.

How to test

Assuming your DNS Zone is delegated to Hetzner:

1. Set var.enable_cluster_issuers and var.enable_reflector to true
2. Make sure your var.base_domain is duly populated within a valid FQDN and point its A record to your server (for single-node clusters) or cloud load balancer if you are using one
3. Make sure you've generated a Hetzner DNS Zone API Token and set it to var.hetzner_dns_api_token
4. Launch tofu apply or terraform apply to create your cluster

If everything is correct, whenever your cluster is ready you should see both ClusterIssuers (prod and staging) up and running.

• Run kubectl get clusterissuers -n cert-manager --kubeconfig k3s_kubeconfig.yaml

• Run kubectl get certs -n cert-manager --kubeconfig k3s_kubeconfig.yaml

• Inspect the cert by kubectl describe certs/wildcard-domain -n cert-manager --kubeconfig k3s_kubeconfig.yaml

• Inspect other namespaces to check the correct secret replication by Reflector: kubectl get secrets --all-namespaces --kubeconfig k3s_kubeconfig.yaml

Caveats

• If no Hetzner DNS Zone API Token is set or it is wrong, LetsEncrypt ClusterIssuers will still be properly installed and you will be able to pass http-01 challenges to generate FQDN domains but dns-01 challenge will fail and no wildcard cert will be generated. Same applies if your DNS Zone is not delegated to Hetzner.

• You can disable Reflector if you don't need secret replication and it will only affect that, ClusterIssuers will still work as intended.

• Bear in mind that dns-01 challenge needs to insert and propagate a record in the DNS Zone, so it can take up to 5 minutes to properly validate the wildcard cert. Everything is going as intended as long as the challenge logs do not return any errors.

Att: Repo maintainers: Please feel free to modify, comment, suggest, whatever you wish to do with this PR. Everything will be most welcome ;)

[mysticaltech]

@jaumebarber This is awesome seriously. But it adds specific sugar for people using Hetzner DNS and maintenance overhead for us. If you could turn this into an example in the examples folder, using extra_manifests, it would be wonderful and would merge ASAP.

[jaumebarber]

Will do, that was actually my second choice, only I thought flagging it and disabling by default would be enough, but it makes more sense to make it a user_kustomization since it is a bit too specific. Actually it is only the Hetzner webhook part that is specific, leaving that apart I do think it'd be nice for everyone to have at least the LetsEncrypt http-01 challenge ClusterIssuers, maybe leaving the wildcard webhook apart as a user_kustomization? @mysticaltech what are your thoughts on that?

[mysticaltech]

@jaumebarber You are right. But it still adds functionality that is not the core purpose of this project, so if this can be all in the examples folder, and refered to in the examples section in the readme, it would be great. The reason for this is just maintenance overhead. Thank you for your understanding 🙏