Enable e2e tests using emulators for cloud providers [ aws, gcp, azure ] #743

anveshreddy18 · 2024-05-22T10:27:00Z

What this PR does / why we need it:

This PR enables e2e tests ( cluster based tests ) to be run with the emulators of 3 major cloud providers i.e AWS, GCP, AZURE. And of course can be run with the real infra as well. The reason for using emulators is to reduce the cost of using real infra every time a PR is raised because with the enablement of e2e tests with this PR, these tests will be run as prow job for every PR.

This PR also partially updates the outdated charts to reflect the changes made to the backup-restore over the years. This will help the users with setting up backup-restore effortlessly and also helps the developers with testing etcdbr standalone

Which issue(s) this PR fixes:
Fixes #722

Special notes for your reviewer:

Release note:

Enable e2e tests with cloud provider emulators and update helm charts for latest version of `etcdbr` to be used with etcd-wrapper

gardener-robot · 2024-05-22T10:27:07Z

@anveshreddy18 You need rebase this pull request with latest master branch. Please check.

gitguardian · 2024-05-22T10:27:37Z

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
-		RSA Private Key	`5454c29`	hack/e2e-test/infrastructure/kind/kubeconfig	View secret
-		RSA Private Key	`dd4a097`	hack/e2e-test/infrastructure/kind/kubeconfig	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secrets safely. Learn here the best practices.
Revoke and rotate these secrets.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

renormalize

Thanks for the much needed PR @anveshreddy18!

I haven't taken a look at the changes in test/integrationcluster just yet. Will take a look in a following review.

renormalize · 2024-06-24T13:52:10Z

.ci/integration_test

@@ -155,6 +158,9 @@ EOF
 }
 EOF
  export AWS_APPLICATION_CREDENTIALS_JSON="${credentials_file}"
+  export AWS_ACCESS_KEY_ID=$1


Why are these variables being exported as env variables when they weren't needed to be exposed before?
The aws cli picks up credentials from .aws that is created in this function anyway, right?

renormalize · 2024-06-24T14:56:05Z

chart/etcd-backup-restore/values.yaml

    pullPolicy: IfNotPresent
  # etcd-backup-restore image to use
  etcdBackupRestore:
    repository: europe-docker.pkg.dev/gardener-project/public/gardener/etcdbrctl
-    tag: v0.12.1
+    tag: v0.28.0


Is there a latest tag available for etcd-backup-restore the way there is for etcd-druid?
It is done that way in etcd-druid's values.yaml

renormalize · 2024-06-24T15:09:10Z

docs/development/testing_and_dependencies.md

- ETCD_VERSION: optional, defaults to `v3.4.13-bootstrap-1`
- ETCDBR_VERSION: optional, defaults to `v0.12.1`
+- ETCD_VERSION: optional, defaults to etcd-wrapper `v0.1.1`
+- ETCDBR_VERSION: optional, defaults to `v0.28.0`


If the latest tag is available as mentioned in this comment, this would have to be updated as well.

renormalize · 2024-06-28T08:46:52Z

docs/development/tests.md

+
+The e2e tests for etcd-backup-restore are the integrationcluster tests in the `test/integrationcluster` package. These tests are run on a Kubernetes cluster and test the full functionality of etcd-backup-restore. The tests create a provider namespace on the cluster and deploy the [etcd-backup-restore helm chart](../../chart/etcd-backup-restore) which in turn deploys the required secrets, configmap, services and finally the statefulset which deploys the pod that runs etcd and backup-restore as a sidecar.
+
+These tests are setup to be run with both emulators and real cloud providers. The emulators can be used for local development and testing as well as prow job to test code changes when a PR is raised. The real cloud providers can be used for testing in a real cloud environment to ensure that the changes work as expected in a real environment. 


Don't necessarily think mentioning testing through prow jobs here is relevant. But don't have a strong opinion.

renormalize · 2024-06-28T08:47:29Z

docs/development/tests.md

+make ci-e2e-kind PROVIDERS="aws,gcp"
+```
+
+


Suggested change

renormalize · 2024-06-28T08:55:47Z

docs/development/tests.md

+#### AWS
+
+For AWS, first get the AWS credentials and update the `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_DEFAULT_REGION` variables of the `hack/config/aws_config.sh` file with the correct values and also update the creation of `/tmp/aws.json` file with the correct values. It should look like below snippet after removing the `endpoint` and `s3ForcePathStyle` fields: 
+
+  ```sh
+  export AWS_APPLICATION_CREDENTIALS_JSON="/tmp/aws.json"
+  echo "{ \"accessKeyID\": \"${AWS_ACCESS_KEY_ID}\", \"secretAccessKey\": \"${AWS_SECRET_ACCESS_KEY}\", \"region\": \"${AWS_DEFAULT_REGION}\" }" > "${AWS_APPLICATION_CREDENTIALS_JSON}"
+  ``` 
+
+Apart from providing the required credentials for real provider, one should also remove the variables that signify the usage of an emulator, which are `AWS_ENDPOINT_URL_S3` and `LOCALSTACK_HOST` for Localstack. Remove them from the make command of the `hack/ci-e2e-kind.sh` script before running the tests.
+
+With these changes made, the tests can be run in the same way as with the emulators.
+
+#### GCP
+
+For GCP, first get the GCP credential service account json file and your project ID and replace the variables `GOOGLE_APPLICATION_CREDENTIALS` and `GCP_PROJECT_ID` in configuration file located at `hack/config/gcp_config.sh` with the path to this service account file and your project ID respectively. We also need to remove the environment variables for the fakegcs provider from the make command of the `hack/ci-e2e-kind.sh` file, for that one should remove the variables `GOOGLE_EMULATOR_ENABLED`, `GCS_EMULATOR_HOST` and `GOOGLE_STORAGE_API_ENDPOINT` and run the tests in the same way as with the emulators.
+
+#### Azure
+
+For Azure, first get the Azure credentials and update the `STORAGE_ACCOUNT` and `STORAGE_KEY` env variables in the configuration file located at `hack/config/azure_config.sh` with the correct values. Also, remove the variables for the azurite, which are `AZURE_STORAGE_API_ENDPOINT`, `AZURE_EMULATOR_ENABLED`, `AZURITE_HOST` and `AZURE_STORAGE_CONNECTION_STRING` from the make command of the `hack/ci-e2e-kind.sh` script. With these changes made, the tests can be run in the same way as with the emulators.


Currently, there is no heading signifying that these paragraphs are describing running end to end tests with real infrastructure.
A sub-heading could be added, and these three sections could be nested in that.

Also, I do not agree with the recommendation of changing the hardcoded credentials for the emulators with the credentials for real infrastructure in the scripts.
It is very likely that a user would forget that they updated the emulator credentials with real credentials, and they check those credentials into their code, and push it to their remotes.

If someone wants to do it at their own risk they could, but we should not specify in the documentation that this is the recommended way of testing with real infrastructure.

renormalize · 2024-06-28T09:06:10Z

.gitignore

+/bin
+/hack/tools/bin


Any reason to add the / in front of bin and hack?
It would be cleaner if these lines are consistent with the rest of the file where / is not added to the beginning.

renormalize · 2024-07-10T13:55:02Z

hack/e2e-test/run-e2e-test.sh

+function cleanup() {
+  if containsElement $STEPS "cleanup" || [ $cleanup_required = true ]; then
+    kind delete cluster --name etcdbr-e2e
+    echo "Cleaning up completed."


Could the kubeconfig file that is auto generated through kind, also be removed in the cleanup?
If you think that's unnecessary then we don't need to do it.

gardener-robot added the needs/rebase Needs git rebase label May 22, 2024

gardener-robot added needs/review Needs review size/xl Size of pull request is huge (see gardener-robot robot/bots/size.py) needs/second-opinion Needs second review by someone else labels May 22, 2024

gardener-robot-ci-2 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label May 22, 2024

gardener-robot-ci-2 added needs/ok-to-test Needs approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels May 22, 2024

gardener-robot-ci-3 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels May 22, 2024

anveshreddy18 self-assigned this May 23, 2024

gardener-robot-ci-1 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels May 29, 2024

anveshreddy18 added 5 commits May 29, 2024 20:23

Enable e2e tests with cloud provider emulators for aws,gcp,azure

dd4a097

Minor cleanup

8b7bc47

Remove kubeconfig and add it to .gitignore

5454c29

Fix make check error

3f37a55

add docs, cleanup

8d1c313

anveshreddy18 force-pushed the enable-e2e branch from 25224da to 8d1c313 Compare May 29, 2024 14:57

gardener-robot-ci-1 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels May 29, 2024

shreyas-s-rao added this to the v0.30.0 milestone Jun 5, 2024

Add steps for setup,test,clean & fixed a failing test

b7580db

gardener-robot-ci-3 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Jun 6, 2024

anveshreddy18 marked this pull request as ready for review June 6, 2024 09:07

anveshreddy18 requested a review from a team as a code owner June 6, 2024 09:07

renormalize self-assigned this Jun 6, 2024

Setup makefile for installing required tools

c40c8b5

gardener-robot-ci-2 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Jun 10, 2024

Make git track hack/tools/bin dir

a711dc5

gardener-robot-ci-3 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jun 10, 2024

gardener-robot-ci-2 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jun 10, 2024

Update docs & delete the objects of gcp bucket in parallel

ec1496b

gardener-robot-ci-1 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Jun 18, 2024

shreyas-s-rao self-assigned this Jun 24, 2024

renormalize reviewed Jun 28, 2024

View reviewed changes

renormalize reviewed Jul 10, 2024

View reviewed changes

anveshreddy18 modified the milestones: v0.30.0, v0.31.0 Aug 21, 2024

ishan16696 modified the milestones: v0.31.0, v0.33.0 Nov 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable e2e tests using emulators for cloud providers [ aws, gcp, azure ] #743

Enable e2e tests using emulators for cloud providers [ aws, gcp, azure ] #743

anveshreddy18 commented May 22, 2024 •

edited

Loading

gardener-robot commented May 22, 2024

gitguardian bot commented May 22, 2024 •

edited

Loading

renormalize left a comment

renormalize Jun 24, 2024

renormalize Jun 24, 2024

renormalize Jun 24, 2024

renormalize Jun 28, 2024

renormalize Jun 28, 2024

renormalize Jun 28, 2024

renormalize Jun 28, 2024

renormalize Jul 10, 2024


		The e2e tests for etcd-backup-restore are the integrationcluster tests in the `test/integrationcluster` package. These tests are run on a Kubernetes cluster and test the full functionality of etcd-backup-restore. The tests create a provider namespace on the cluster and deploy the [etcd-backup-restore helm chart](../../chart/etcd-backup-restore) which in turn deploys the required secrets, configmap, services and finally the statefulset which deploys the pod that runs etcd and backup-restore as a sidecar.

		These tests are setup to be run with both emulators and real cloud providers. The emulators can be used for local development and testing as well as prow job to test code changes when a PR is raised. The real cloud providers can be used for testing in a real cloud environment to ensure that the changes work as expected in a real environment.

		/bin
		/hack/tools/bin

Enable e2e tests using emulators for cloud providers [ aws, gcp, azure ] #743

Are you sure you want to change the base?

Enable e2e tests using emulators for cloud providers [ aws, gcp, azure ] #743

Conversation

anveshreddy18 commented May 22, 2024 • edited Loading

gardener-robot commented May 22, 2024

gitguardian bot commented May 22, 2024 • edited Loading

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

renormalize left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anveshreddy18 commented May 22, 2024 •

edited

Loading

gitguardian bot commented May 22, 2024 •

edited

Loading