Fix local provider setup for running etcd clusters & e2e tests

[anveshreddy18]

How to categorize this issue?

/area testing
/area usability
/kind bug

What happened:

When running etcd-druid locally with local provider, I've seen the functionality is broken with a volume mount error for the pods as shown below:

MountVolume.SetUp failed for volume "local-backup" : hostPath type check failed: /etc/gardener/local-backupbuckets/etcd-bucket is not a directory

This is due to the HostPath type being set to Directory instead of DirectoryOrCreate here.

Also all the e2e tests for the local provider fail with the following error:

[FAILED] Unexpected error:
      <*errors.StatusError | 0x140005ff5e0>: 
      jobs.batch "purge-local-snapstore" already exists
      {
          ErrStatus: {
              TypeMeta: {Kind: "", APIVersion: ""},
              ListMeta: {
                  SelfLink: "",
                  ResourceVersion: "",
                  Continue: "",
                  RemainingItemCount: nil,
              },
              Status: "Failure",
              Message: "jobs.batch \"purge-local-snapstore\" already exists",
              Reason: "AlreadyExists",
              Details: {
                  Name: "purge-local-snapstore",
                  Group: "batch",
                  Kind: "jobs",
                  UID: "",
                  Causes: nil,
                  RetryAfterSeconds: 0,
              },
              Code: 409,
          },
      }
  occurred
  In [It] at: /Users/i586337/work/etcd-druid/test/e2e/etcd_multi_node_test.go:264

This needs to be looked into and fix it.

This issue acts as a tracker to fix the problems with local provider.

What you expected to happen:

The etcd-druid setup with local provider should be as robust as it is with other providers/emulators.

How to reproduce it (as minimally and precisely as possible):

In the examples/etcd/druid_v1alpha1_etcd.yaml file, change the spec.backup.store to the following and try deploying the Etcd CR and observe the pod events.

store:
  provider: local # options: aws,azure,gcp,openstack,alicloud,dell,openshift,local
  prefix: etcd-prefix
  container: etcd-bucket

For e2e tests, just run them locally by doing below and observe the logs:

make kind-up
make test-e2e PROVIDERS="local" TEST_ID="anvesh" STEPS="setup,deploy,test"

[anveshreddy18]

/assign