support assumeRoleWithWebIdentity for Route53 issuer

[pwhitehead-splunk]

Pull Request Motivation

Update the Route53 provider to support fetching credentials using AssumeRoleWithWebIdentity.

Kind

/kind feature

Release Note

Update the Route53 provider to support fetching credentials using AssumeRoleWithWebIdentity

[jetstack-bot]

Hi @pwhitehead-splunk. Thanks for your PR.

I'm waiting for a cert-manager member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jetstack-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign munnerz for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[erikgb]

/ok-to-test

[ThatsMrTalbot]

Looking at this implementation, it relies on a token mounted within the cert-manager Pod, meaning cert-manager needs a token with broad permissions mounted.

Would it be better to implement it in a similar way to Vault? Allowing for the token to either be loaded from a secret or to be issued from a service account.

Loading from a secret covers the use case of a non IRSA token, issuing from a service account covers the IRSA case.

Could look like something like this in the issuer config:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    ...
    solvers:
      - selector:
          dnsZones:
          - "example.com"
        dns01:
          route53:
            auth:
              kubernetes:
                serviceAccountRef:
                  name: my-irsa-service-account
                  audiences: ["sts.amazonaws.com"] # This should be the default value, as thats what IRSA tokens use

or

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    ...
    solvers:
      - selector:
          dnsZones:
          - "example.com"
        dns01:
          route53:
            auth:
              kubernetes:
                secretRef:
                  name: my-irsa-secret
                  key: token

[pwhitehead-splunk]

@ThatsMrTalbot Thank you for the feedback! The implementation was written with the expectation that the end user would use the token from the projected volume that's provided by IRSA or Workload Identity. The motivation for this PR was to issue certs in AKS when using Route53 solvers without requiring static credentials. I'm not familiar with GCP but I'd expect there's a similar implementation.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    ...
    solvers:
    - dns01:
        route53:
          region: us-west-2
          role: arn:aws:iam::YYYYYYYYYYYY:role/dns-manager # must trust the OIDC provider, should verify aud and sub
          webIdentityToken: /var/run/secrets/azure/tokens/azure-identity-token # provided by Workload Identity

Assuming the IAM Identity Provider and IAM role's trust is set up properly then the token's use is in line with its expected use case that IRSA/Workload Identity solves; we're exchanging a token for temporary credentials.

The secret use case you mentioned is out of scope for what I hope to achieve, but if I follow your question properly, you're suggesting that we issue a new token with a configurable audience when reconciling a cert? What would the payload look like? IAM trust policies typically have a conditional to trust the subject, as an end user I'd expect that to match the namespace and service account used in the Cert Manager deployment. Issuing a token with a custom audience feels like a reimplementation of IRSA/Workload Identity. That being said, if that's the path you prefer then I can make changes.

[ThatsMrTalbot]

Hiya @pwhitehead-splunk

Solutions like IRSA simply implement a mutating webhook to project a Kubernetes service account token with a different audience inside your pods. In the case of IRSA the audience is set to sts.amazonaws.com (see https://github.com/aws/amazon-eks-pod-identity-webhook).

This implementation is true of both IRSA in AWS (https://github.com/aws/amazon-eks-pod-identity-webhook/blob/master/pkg/handler/handler.go#L332-L347) and Azure Workload Identity in AKS (https://github.com/Azure/azure-workload-identity/blob/9b60b5240fcbee6c2e4660b435a20dc2683ef945/pkg/webhook/webhook.go#L419-L438).

What I was proposing is that instead of using the projected token in the cert-manager pod, we can issue a token ourselves using the TokenRequest API, this is something we already do in other parts the cert-manager codebase -

cert-manager/pkg/controller/certificatesigningrequests/vault/vault.go

Line 95 in 410b7a6

createTokenFn := func(ns string) internalvault.CreateToken { return v.kclient.CoreV1().ServiceAccounts(ns).CreateToken }

The TokenRequest API is the same API that is used by Kubelet for the projected service account tokens, which in turn are used by IRSA and Azure Workload Identity. So by requesting a token directly its achieving the same goal, without the need for cert-manager service account itself to have IAM permissions. The user can just reference a service account that has the needed IAM permissions and cert-manager can request a token itself. The token itself would look the same as one provided by IRSA or Azure Workload Identity.

So my suggestion is basically instead of doing:

• User pre-configures cert-manager with IRSA or Azure Workload Identity
• User specifies in the Issuer config the path that IRSA/Identity token lives at
• When cert-manager wants to talk to AWS it loads the token from disk, which we then use with the AssumeRoleWithWebIdentity call.

We do:

• User specifies in the Issuer config that a ServiceAccount should be used when talking to AWS
• When cert-manager wants to talk to AWS we use the token request API to issue a service account token, which we then use with the AssumeRoleWithWebIdentity call.

This has the added benefit of allowing different issuers to reference different service accounts with different IAM permissions in AWS.

[pwhitehead-splunk]

That makes sense, thanks for the clarification! I'll update the PR.

[pwhitehead-splunk]

@ThatsMrTalbot let me know what you think about the changes

[ThatsMrTalbot]

@ThatsMrTalbot let me know what you think about the changes

I really like this implementation, I think this will be a powerful secretless solution for AWS auth. I've had a quick skim and left a comment. Will give it a proper look tomorrow.

Thanks for implementing this, its really cool 😄

[ThatsMrTalbot]

/test pull-cert-manager-master-make-test

[ThatsMrTalbot]

/test pull-cert-manager-master-make-test

[ThatsMrTalbot]

Thanks very much for this!
/lgtm
/approve

[cert-manager-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ThatsMrTalbot

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ThatsMrTalbot]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment