Introduce Azure VMSS orchestration mode VM (VMO) based Shoot clusters #217

dkistner · 2020-12-07T14:10:00Z

How to categorize this PR?

/kind enhancement
/priority normal
/platform azure

What this PR does / why we need it:
Add support for Azure virtual machine scaleset orchestration mode vm (VMO) based Shoot clusters.
Each worker pool will get its own VMO instead its of a shared resource like for AvailabilitySets based clusters.

The Shoot cluster need to be non-zoned and have the annotation alpha.azure.provider.extensions.gardener.cloud/vmo=true to enable VMO which we currently treat as a preview feature.

Details:

In case the corresponding fault domain count in the CloudProfile changes then the VMO will be replaced. This will lead to a rolling update of the machines in each worker pool.
Orphan Gardener managed VMOs will be removed during the Worker reconciliation. So it is safe to be deployed into existing resource group without leaking resources.
VMO based clusters will make use of the Standard SKU Loadbalancer.
The NatGateway will be available for VMO clusters. This is a difference to AvailabilitySet based clusters which does not support NatGateway.
Accelerated network will work with VMO clusters. We can't enable it for AvailabilitySet based clusters as all machines assigned to it need to be compatible/running with accelerated network support which is we can't archive for existing clusters with a central AvailabilitySet, see Properly Enable Accelerated Networking for AvSet Clusters #101

More details:

The Azure SDK is updated to support compute client in version 2020-06-30 (min 2019-12-01 is required to use VMO)
VMO clusters will use the vmss version of the vm-controller in the Azure cloud-controller-manager (activated in the cloud-provider config via vmType=vmss).
A vmss client has been added to the client factory.
Added mocks for the client factory and the vmss client.
The Worker controller tests have been restructured partially to allow more reuse for the machine dependency tests.
The machineclass chart structure to configure AvailabilitySets has been aligned to the way to configure VMOs.

Which issue(s) this PR fixes:
Intend to fix #61

Release note:

Gardener is now supporting Azure clusters based on Virtual MachineScale Orchestration Mode Vm (VMO) as a preview feature. 
VMO should be perspectively a replacement for AvailabilitySet based clusters. You need to make sure that your subscription is eligible for VMO usage. You can find more details here: https://github.com/gardener/gardener-extension-provider-azure/blob/master/docs/usage-as-end-user.md#preview-shoot-clusters-with-vmss-orchestration-mode-vm-vmo

/invite @kon-angelo

ialidzhikov

Minor nits from my initial "go through".

pkg/apis/azure/helper/helper.go

pkg/apis/azure/types_worker.go

pkg/controller/worker/machines.go

pkg/controller/worker/utils_test.go

pkg/azure/types.go

pkg/controller/controlplane/valuesprovider.go

pkg/controller/worker/machine_dependencies_test.go

pkg/controller/worker/machines.go

ialidzhikov · 2020-12-23T14:00:59Z

/test

testmachinery · 2020-12-23T14:01:03Z

Testrun: e2e-sdh4d
Workflow: e2e-sdh4d-wf
Phase: Failed

+---------------------+---------------------+--------+----------+
|        NAME         |        STEP         | PHASE  | DURATION |
+---------------------+---------------------+--------+----------+
| infrastructure-test | infrastructure-test | Failed | 12m26s   |
+---------------------+---------------------+--------+----------+

ialidzhikov

@dkistner , the infrastructure test needs also to be adapted. Ref https://tm.gardener.cloud/testrun/default/e2e-sdh4d

Other than that I played locally with the change and it worked well for lifecycle operations for Shoot with enabled VMO.

dkistner · 2020-12-23T14:45:17Z

Awesome. Thanks @ialidzhikov for the review.
I will address the changes in the new year :)

ialidzhikov

Thank you, @dkistner !
Adding one last comment that I missed to add in the previous review.

pkg/azure/client/factory.go

pkg/controller/controlplane/valuesprovider.go

pkg/internal/infrastructure/terraform.go

Add support for Azure virtual machine scaleset orchestration mode vm (VMO) based Shoot clusters. Each worker pool will get its own VMO instead its of a shared resource like for AvailabilitySets based clusters. The Shoot cluster need to be non-zoned and have the annotation "alpha.azure.provider.extensions.gardener.cloud/vmo=true" to enable VMO which we currently treat as a preview feature. - In case the corresponding fault domain count in the CloudProfile changes then the VMO will be replaced. This will lead to a rolling update of the machines in each worker pool. - Orphan Gardener managed VMOs will be removed during the Worker reconcilation. So it is safe to be deployed into existing resource group without leaking resources. - VMO based clusters will make use of the Standard SKU Loadbalancer. - The NatGateway will be available for VMO clusters. This is a difference to AvailabilitySet based clusters which does not support NatGateway. - Accelerated network will work with VMO clusters. Other changes: - The Azure SDK is updated to support compute client in version 2020-06-30 (required to support VMO is min 2019-12-01) - VMO clusters will use the "vmss" version of the vm-controller in the Azure cloud-controller-manager (activated in the cloud-provider config via "vmType=vmss"). - A vmss client has been added to the client factory. - Added mocks for the client factory and the vmss client. - The Worker controller tests have been restructured partially to allow more reuse for the machine dependency tests. - The machineclass chart structure to configure AvailabilitySets has been alligned to the way to configure VMOs.

dkistner · 2021-01-12T16:51:30Z

Thanks for the feedback. I have addressed it.
Also I added infrastructure integration tests for VMO as they are missing before.

/test

testmachinery · 2021-01-12T16:51:34Z

Testrun: e2e-mw78k
Workflow: e2e-mw78k-wf
Phase: Succeeded

+---------------------+---------------------+-----------+----------+
|        NAME         |        STEP         |   PHASE   | DURATION |
+---------------------+---------------------+-----------+----------+
| infrastructure-test | infrastructure-test | Succeeded | 36m6s    |
+---------------------+---------------------+-----------+----------+

ialidzhikov

/lgtm

rfranzke · 2021-01-21T06:06:37Z

@dkistner @kon-angelo Can this PR be merged?

kon-angelo

/lgtm too

Let's wait a final confirmation from @dkistner.

dkistner · 2021-01-21T11:21:27Z

Thanks for confirming. Yes lets go on and merge :)

danielfoehrKn · 2021-01-21T12:50:44Z

Very nice PR!

I could not find the place in the MCM azure machine driver where the VM is created with a reference to the ScaleSet. Can you by any chance point me to it? Thanks!

dkistner · 2021-01-21T13:08:52Z

Hi Daniel, see here: https://github.com/gardener/machine-controller-manager/blob/master/pkg/driver/driver_azure.go#L191-L194

Chokiis82

``

dkistner requested review from a team as code owners December 7, 2020 14:10

gardener-robot added kind/api-change API change with impact on API users needs/second-opinion Needs second review by someone else labels Dec 7, 2020

gardener-robot requested a review from kon-angelo December 7, 2020 14:10

gardener-robot added kind/enhancement Enhancement, improvement, extension needs/review Needs review platform/azure Microsoft Azure platform/infrastructure priority/normal size/xl Size of pull request is huge (see gardener-robot robot/bots/size.py) labels Dec 7, 2020

dkistner force-pushed the feature-vmo-per-workerpool branch from 19e9136 to d7bb323 Compare December 7, 2020 14:43

gardener-robot-ci-1 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Dec 7, 2020

ialidzhikov reviewed Dec 22, 2020

View reviewed changes

gardener-robot added the kind/test Test label Dec 23, 2020

ialidzhikov requested changes Dec 23, 2020

View reviewed changes

gardener-robot added the needs/changes Needs (more) changes label Dec 23, 2020

ialidzhikov reviewed Dec 23, 2020

View reviewed changes

pkg/azure/client/factory.go Outdated Show resolved Hide resolved

kon-angelo reviewed Dec 28, 2020

View reviewed changes

pkg/controller/controlplane/valuesprovider.go Outdated Show resolved Hide resolved

kon-angelo reviewed Dec 28, 2020

View reviewed changes

pkg/internal/infrastructure/terraform.go Outdated Show resolved Hide resolved

dkistner force-pushed the feature-vmo-per-workerpool branch from d7bb323 to cf0beff Compare January 12, 2021 16:47

gardener-robot-ci-2 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jan 12, 2021

gardener-robot-ci-1 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jan 12, 2021

ialidzhikov approved these changes Jan 13, 2021

View reviewed changes

gardener-robot added reviewed/lgtm Has approval for merging needs/changes Needs (more) changes and removed needs/changes Needs (more) changes needs/review Needs review needs/second-opinion Needs second review by someone else reviewed/lgtm Has approval for merging labels Jan 13, 2021

kon-angelo approved these changes Jan 21, 2021

View reviewed changes

gardener-robot added reviewed/lgtm Has approval for merging and removed needs/changes Needs (more) changes labels Jan 21, 2021

dkistner merged commit 9363d41 into gardener:master Jan 21, 2021

dkistner deleted the feature-vmo-per-workerpool branch January 21, 2021 11:21

gardener-robot added priority/3 Priority (lower number equals higher priority) and removed priority/3 Priority (lower number equals higher priority) labels Mar 8, 2021

Chokiis82 reviewed Jun 7, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce Azure VMSS orchestration mode VM (VMO) based Shoot clusters #217

Introduce Azure VMSS orchestration mode VM (VMO) based Shoot clusters #217

dkistner commented Dec 7, 2020 •

edited

Loading

ialidzhikov left a comment

ialidzhikov commented Dec 23, 2020

testmachinery bot commented Dec 23, 2020 •

edited

Loading

ialidzhikov left a comment

dkistner commented Dec 23, 2020

ialidzhikov left a comment

dkistner commented Jan 12, 2021

testmachinery bot commented Jan 12, 2021 •

edited

Loading

ialidzhikov left a comment

rfranzke commented Jan 21, 2021

kon-angelo left a comment

dkistner commented Jan 21, 2021

danielfoehrKn commented Jan 21, 2021

dkistner commented Jan 21, 2021

Chokiis82 left a comment

Introduce Azure VMSS orchestration mode VM (VMO) based Shoot clusters #217

Introduce Azure VMSS orchestration mode VM (VMO) based Shoot clusters #217

Conversation

dkistner commented Dec 7, 2020 • edited Loading

ialidzhikov left a comment

Choose a reason for hiding this comment

ialidzhikov commented Dec 23, 2020

testmachinery bot commented Dec 23, 2020 • edited Loading

ialidzhikov left a comment

Choose a reason for hiding this comment

dkistner commented Dec 23, 2020

ialidzhikov left a comment

Choose a reason for hiding this comment

dkistner commented Jan 12, 2021

testmachinery bot commented Jan 12, 2021 • edited Loading

ialidzhikov left a comment

Choose a reason for hiding this comment

rfranzke commented Jan 21, 2021

kon-angelo left a comment

Choose a reason for hiding this comment

dkistner commented Jan 21, 2021

danielfoehrKn commented Jan 21, 2021

dkistner commented Jan 21, 2021

Chokiis82 left a comment

Choose a reason for hiding this comment

dkistner commented Dec 7, 2020 •

edited

Loading

testmachinery bot commented Dec 23, 2020 •

edited

Loading

testmachinery bot commented Jan 12, 2021 •

edited

Loading