kubetest2-kops: add support for boskos-resource-type flag

[justinsb]

This will cause AWS tests to acquire an aws-account from boskos (when
set to aws-account), and run the tests with those credentials.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from justinsb. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[justinsb]

I can verify this once we have kubernetes/test-infra#29907 merged

[hakman]

/test presubmit-kops-aws-boskos-kubetest2

[hakman]

/cc @ameukam

[justinsb]

Adding in dynamic bucket creation, so we don't need to give permissions to a shared bucket to all the boskos accounts

/test presubmit-kops-aws-boskos-kubetest2

[justinsb]

/test presubmit-kops-aws-boskos-kubetest2

[justinsb]

/test presubmit-kops-aws-boskos-kubetest2

[justinsb]

/test presubmit-kops-aws-boskos-kubetest2

[justinsb]

/test presubmit-kops-aws-boskos-kubetest2

[justinsb]

/test presubmit-kops-aws-boskos-kubetest2

[justinsb]

@ameukam so I hit some problems with the existing aws-account boskos pool. I got accounts like capa-user-XX (e.g. capa-user-02). I don't think they have access to the shared S3 bucket we were using previously. So I tried having kubetest2-kops create a bucket in each AWS account (GCE does something similar already). However, I think maybe the capa-user-XX accounts don't have S3 access (or I've messed up credentials somewhere, which is definitely plausible :-) )

Maybe we try creating some new accounts in a new boskos pool (e.g. resource-type == kops-aws-account ?), and make sure they have permissions to do S3? Doing separate pools will also help us with tracking back spend (e.g. kops vs capa).

[ameukam]

@ameukam so I hit some problems with the existing aws-account boskos pool. I got accounts like capa-user-XX (e.g. capa-user-02). I don't think they have access to the shared S3 bucket we were using previously. So I tried having kubetest2-kops create a bucket in each AWS account (GCE does something similar already). However, I think maybe the capa-user-XX accounts don't have S3 access (or I've messed up credentials somewhere, which is definitely plausible :-) )

Maybe we try creating some new accounts in a new boskos pool (e.g. resource-type == kops-aws-account ?), and make sure they have permissions to do S3? Doing separate pools will also help us with tracking back spend (e.g. kops vs capa).

@justinsb we can use those accounts https://github.com/kubernetes/k8s.io/blob/main/infra/aws/terraform/management-account/organization-accounts-boskos.tf#L17-L42 for kOps. They are currently unused. WDYT ?

[justinsb]

@ameukam that sounds good to me - it's great that they under terraform control because if we do need any other permissions we have a good path forwards. Are they loaded into a boskos pool? Any idea how to select them if so - I think I can select a different resource-type, not sure if there's any other way to select them...

[ameukam]

@justinsb They are currently not onboarded. I don't have access to the default cluster. Would be interesting to have a dedicated EKS cluster with a dedicated boskos where can test those changes. That means also move some (or all the) presubmits to this new cluster.

[k8s-ci-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[hakman]

@justinsb @ameukam Anything new here?

[ameukam]

@justinsb @ameukam Anything new here?

Nothing new for the moment. A few things needs to be done on my side.

[k8s-ci-robot]

@justinsb: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
presubmit-kops-aws-boskos-kubetest2	f9afe5f	link	true	/test presubmit-kops-aws-boskos-kubetest2
pull-kops-kubernetes-e2e-ubuntu-gce-build	f9afe5f	link	true	/test pull-kops-kubernetes-e2e-ubuntu-gce-build
pull-kops-verify-hashes	f9afe5f	link	true	/test pull-kops-verify-hashes

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle rotten
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten