IAM auth to ECR support

[jessesuen]

Proposed Feature

Kargo needs to support IAM auth to ECR

Motivation

ECR is a common container registry and we should have native support for it.

Suggested Implementation

Follow guidelines here:
https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html

[jessesuen]

I think it may be possible to workaround this limitation until Kargo natively supports ECR, with a CronJob that periodically updates an image repository Secret that Kargo understands.

https://linuxhandbook.com/auto-update-aws-ecr-token-kubernetes/
https://skryvets.com/blog/2021/03/15/kubernetes-pull-image-from-private-ecr-registry/

Will try and formulate an official workaround.

[joebowbeer]

Can only a service account be used for ECR authentication? This is needed for IRSA or the newer Pod Identity method, and avoids the need for secrets.

For IRSA, an eks.amazonaws.com/role-arn annotation would be added to Kargo's service account.
For Pod Identity, the association would be pre-configured in AWS.

Keel implements a cache:

https://github.com/keel-hq/keel/blob/master/extension/credentialshelper/aws/aws.go

[krancour]

@joebowbeer I don't think this is something that can realistically be handled at the pod level.

Kargo's credentials are managed at the Project level. This means that two Projects can use two different sets of credentials (with entirely different permissions) for talking to the same registry. It also means Projects that have no business talking to that registry cannot, for lack of credentials. This is essential to Kargo's tenancy model.

If credentials were handled at the pod level, all Projects, even those that do not need them, would have identical permissions on the registry and this would seem to undermine the tenancy model.

[joebowbeer]

@krancour I interpret what you're saying to mean that, in Kargo, some inherent container identity can't be used to obtain the AWS creds.

I'm saying that there should be some method other than the less secure secretRef to obtain the creds.

ESO, for example, can obtain AWS credsFromSecretRef or credsFromServiceAccount (arn annotation)

https://github.com/external-secrets/external-secrets/blob/main/pkg/provider/aws/auth/auth.go

There's an open issue to support pod identity directly, which has collected comments similar to yours:

external-secrets/external-secrets#2951

A workaround involving two associated roles is described here:

https://containscloud.com/2024/03/24/integrating-aws-secrets-manager-to-eks-using-external-secrets/

Paraphrasing in the context of Kargo:

We will create two IAM roles and a policy and associate them with the EKS cluster using EKS Pod Identity Agent. The first role will be associated with the [Kargo] service account and has no policy associated with it while the second role is configured in the [Kargo Project] resource and will be assumed exclusively in this context, thus following the concept of least privilegies.

Snippet from Kargo Project:

  provider:
    aws:
      service: ECR
      region: us-east-1
      # Role that will be assumed by Kargo to access the secret
      role: arn:aws:iam::<AWS_ACCOUNT_ID>:role/workload01-kargo-role

FWIW, I also found this useful:

https://securitylabs.datadoghq.com/articles/eks-pod-identity-deep-dive/

[krancour]

I interpret what you're saying to mean that, in Kargo, some inherent container identity can't be used to obtain the AWS creds.

Essentially, yes.

To re-iterate the problem I see, perhaps more eloquently than my middle-of-the-night response: Kargo, being intrinsically multi-tenant, should not be using one common set of credentials for accessing any given registry (ECR or otherwise). It should be using Project-specific credentials. (With the current exception to this being when the admin installing/managing Kargo has opted to designate a namespace as a home for shared credentials, usable by all Projects.)

ESO, for example, can obtain AWS credsFromSecretRef or credsFromServiceAccount (arn annotation)

https://github.com/external-secrets/external-secrets/blob/main/pkg/provider/aws/auth/auth.go

There's an open issue to support pod identity directly, which has collected comments similar to yours:

external-secrets/external-secrets#2951

A workaround involving two associated roles is described here:

https://containscloud.com/2024/03/24/integrating-aws-secrets-manager-to-eks-using-external-secrets/

Thank you for all of this. I found this very insightful. I can see how the issue is quite similar, indeed, to the one I have cited -- there is a need to use different credentials for different SecretStores/ClusterSecretStores.

FWIW, I also found this useful:

https://securitylabs.datadoghq.com/articles/eks-pod-identity-deep-dive/

I found it useful as well, so once again, thank you. You've given me quite a bit to think about here.

For sure, Kargo's use of specially formatted Secrets as credentials isn't going to change, primarily because it's a lowest common denominator approach that should work in all clusters, regardless of your cloud provider (or even if on-prem), and (once more authn methods are added) should work for all registries, regardless of where they're hosted -- Docker Hub, Quay, GCHR, ECR, ACR, etc. Indeed, consider the case of needing to access ECR from a non-EKS cluster. That is still going to need to work, but without the EKS pod identity agent in play, pod identity isn't going to be a viable way to approach that. Another reason is that Project-level credentials stored as specially formatted Secrets are more easily managed via UI/CLI for those who are inclined toward "clickops."

Getting this lowest common denominator to work with ECR (and SSH, and GitHub Applications, etc.) is my first priority, but I don't think this even remotely rules out what you've suggested. I see the use of pod identity, be it in EKS, GKE, AKS, etc. as being a possible (additional) alternative to the above that is more likely to be addressed in a follow-up.

[joebowbeer]

Even as a highest common factor (starting point), I hope it is possible to obtain access without needing to store credentials in Kubernetes secrets.