terraform, azure and utoronto: an upgrade, misc to support it, and misc opportunistic details

[consideRatio]

The terraform changes are applied already. I've done some things besides just upgrading the k8s cluster version that I believed were resonable to go for opportunistically.

Changes overviewed

• a user node pool switch from Standard_E8s_v3 to _v5, which is like r3.2xlarge to r5.2xlarge in AWS. This price of these are the same.
• a user node pool switch and a core node pool a prepared transition from use of persistent machine disks on user nodes to ephemeral machine disks (like in GKE also for example)
• (reverted, autoscaling and 3 replicas remains) a pinning of core node pool count to 2 in order to avoid paying for an empty core node pool due to AKS / UToronto: calico-typha scaled to three replicas, forcing three core nodes #3592
• added resolution of configuration in order to
  • enable upgrade of control plan and node pools separately
  • enable a non-disruptive transitions from misc machine types
• misc fixme notes and comments with notable research behind them
• removal of no longer relevant label (hub.jupyter.org/node-size)
• an AKS transition towards referencing "notebook nodes" as "user nodes"
• (added post-review) refactored use of multiple node pool variables to a single variable to reduce duplication of node pool variable definitions

[consideRatio]

@yuvipanda thank you soo much for the quick an thorugh review!

• (requested) I went back to autoscaling the core node pool and applied the change
• (requested) I made name required for node pools
• (not requested) I combined multiple node pool variables into a single variable to reduce the three ~identical variable definitions to one, and made a variable map into a list as the keys were entirely ignored anyhow
• (not requested) I added and refined some comments

[yuvipanda]

Thanks a lot @consideRatio! This mostly looks good to me - however, I'd like us to do one of two things:

1. Have separate core_node_pool, notebook_nodepools and dask_nodepool, to match what we have on GCP
2. Encode via type definition somehow that the only supported keys for the node_pools map are "core", "user" and "dask"

Currently, (2) is encoded in the code - if we use any other key, it will just silently fail.

I think (2) is preferable (here and in GCP too, although that's out of scope) but if that's not possible I'd like us to do (1) than leave it implicit.

I appreciate your cleanup efforts, @consideRatio!

[consideRatio]

Thank you for reviewing @yuvipanda!!

if we use any other key, it will just silently fail

I think I can come up with a way to make this fail loudly, but not via the type definition. I tried to do this myself but failed to find a way to do it without breaking the DRY principle and verbosly duplicating the type definition for any individual node pool. I don't want to break DRY in this case, I find it gets too messy and hard to maintain.

I'm aiming to meet your request for changes by failing loudly if:

• node_pools variable includes keys besides core, user, and dask
• node_pools value for core doesn't include exactly one node pool (since one and only one node pool maps to default_node_pool)

[consideRatio]

Ahh googling for "terraform assertions" I ended up finding that there can be a "custom validation" block for variables like documented in https://developer.hashicorp.com/terraform/language/values/variables#custom-validation-rules - I'll go for that.

[yuvipanda]

This is awesome work, and TIL about terraform validation too. Thank you for digging into this, @consideRatio! LGTM now.

[consideRatio]

Thank you for reviewing @yuvipanda !!

[yuvipanda]

@consideRatio thank you for patiently doing a couple rounds with me async on this!