✨ feat: Add IRSA support for self-managed clusters (rebase)

[sl1pm4t]

What type of PR is this?

/kind feature

What this PR does / why we need it:

Adds IRSA functionality to self-managed clusters. This will bring the self-managed clusters inline in functionality with Managed clusters that provide most of this functionality out of the box.

With this PR, the following new resources are created:

• S3 bucket ( if not already created for the Ignition feature )
• Two public world readable objects in the S3 bucket:
  • <cluster_name>/.well-known/openid-configuration - OpenID Connect discovery document
  • <cluster_name>/openid/v1/jwks - Service Account signing public key
• AWS IAM Identity Provider, configured to trust the Issuer found at the S3 URL where the OIDC discovery doc and keys are published.

This is a continuation of an old unmerged PR #4094 - with some fixes and some functionality removed to reduce the scope of the PR.

The functionality removed includes:

• This PR no longer deploys the amazon-pod-identity-webhook addon to the workload cluster. I felt there are already many ways to manage cluster addons, including ClusterResourceSets or CAAPH, and that it was unnecessary to install the addon via the controller which then becomes an ongoing maintenance burden. Instead, the requirement for the addon has been added to the documentation.
• This PR no longer modifies the API Server service-account-issuer argument through kubeadm patches. This is easily covered in the documentation and only requires a single line of config to be added to the AWSCluster resource, but also during testing I experienced issues with this being applied inconsistently, resulting in different values across the control plane nodes.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):

Fixes #3560
Supersedes #4094

Special notes for your reviewer:

This PR adds a new ReconcileOIDCProvider to the AWSCluster reconciliation loop.

• Created a new IAM service, in the future the same logic for EKS could be combined as previously the OIDC code was buried in the EKS service. Details on the reconciler can be found in comments.
• Extend the S3 service to allow uploading data to arbitrary keys, that can also be marked as public (OIDC discovery docs need to be public for 3rd party systems to retrieve public keys etc.)
• Exposed a ManagementClient and RemoteClient for both cluster types and exported Client.
• Moved OIDCProvider status type to v1beta2 and migrated out of the EKS API to make one type both clusters can reference a single type.
• This PR adds a new Experimental feature flag to enable this functionality. This feature is dependent on the S3 bucket associated with Ignition node configuration, but it felt unintuitive to need to enable the Ignition feature flag to get OIDC support.

Checklist:

• squashed commits
• includes documentation
• includes emojis
• adds unit tests
• adds or updates e2e tests

Release note:

Add IRSA support for self-hosted clusters

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign ankitasw for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @sl1pm4t. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[luthermonson]

/ok-to-test

[sl1pm4t]

/test pull-cluster-api-provider-aws-e2e

[sl1pm4t]

There is some investigation of the failing e2e test happening over here.
It would appear the failure is unrelated to this PR.

[sl1pm4t]

/retest-required

[richardcase]

/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-eks

[sl1pm4t]

/retest

[richardcase]

/test pull-cluster-api-provider-aws-e2e-eks

[k8s-ci-robot]

@sl1pm4t: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cluster-api-provider-aws-e2e-eks	540420c	link	false	/test pull-cluster-api-provider-aws-e2e-eks

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.