Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add azure machine learning module #1372

Open
wants to merge 11 commits into
base: master
Choose a base branch
from
3 changes: 2 additions & 1 deletion examples/azure/terraform-azure-aks-example/output.tf
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
output "client_key" {
value = azurerm_kubernetes_cluster.k8s.kube_config.0.client_key
value = azurerm_kubernetes_cluster.k8s.kube_config.0.client_key
sensitive = true
}

output "client_certificate" {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Terraform Machine Learning Example

This folder contains a simple Terraform module that deploys resources in [Azure](https://azure.microsoft.com/) to demonstrate
how you can use Terratest to write automated tests for your Azure Terraform code. This module deploys an Azure Machine Learning Workspace and Azure Machine Learning Compute Cluster.

- [Azure Machine Learning](https://learn.microsoft.com/en-us/azure/machine-learning/?view=azureml-api-2e)

Check out [test/azure/terraform_azure_machine_learning_example_test.go](/test/azure/terraform_azure_machine_learning_example_test.go) to see how you can write
automated tests for this module.

Note that the resources deployed in this module don't actually do anything; it just runs the resources for
demonstration purposes.

**WARNING**: This module and the automated tests for it deploy real resources into your Azure account which can cost you
money. The resources are all part of the [Azure Free Account](https://azure.microsoft.com/free/), so if you haven't used that up,
it should be free, but you are completely responsible for all Azure charges.

## Running this module manually

1. Sign up for [Azure](https://azure.microsoft.com/)
1. Configure your Azure credentials using one of the [supported methods for Azure CLI
tools](https://docs.microsoft.com/cli/azure/azure-cli-configuration?view=azure-cli-latest)
1. Install [Terraform](https://www.terraform.io/) and make sure it's on your `PATH`
1. Ensure [environment variables](../README.md#review-environment-variables) are available
1. Run `terraform init`
1. Run `terraform apply`
1. When you're done, run `terraform destroy`

## Running automated tests against this module

1. Sign up for [Azure](https://azure.microsoft.com/).
1. Configure your Azure credentials using one of the [supported methods for Azure CLI
tools](https://docs.microsoft.com/cli/azure/azure-cli-configuration?view=azure-cli-latest).
1. Install [Terraform](https://www.terraform.io/) and make sure it's on your `PATH`.
1. [Review environment variables](#review-environment-variables).
1. Install [Golang](https://golang.org/) and make sure this code is checked out into your `GOPATH`.
1. `cd test`
1. Make sure [the azure-sdk-for-go versions match](#check-go-dependencies) in [/go.mod](/go.mod) and in [test/azure/terraform_azure_machine_learning_example_test.go](/test/azure/terraform_azure_machine_learning_example_test.go).
1. `go build test/azure/terraform_azure_machine_learning_example_test.go`
1. `go test -v -run TestTerraformAzureMachineLearningWorkspaceExample`
83 changes: 83 additions & 0 deletions examples/azure/terraform-azure-machinelearning-example/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# ---------------------------------------------------------------------------------------------------------------------
# DEPLOY AN AZURE MACHINE LEARNING WORSPACE
# This is an example of how to deploy an Azure machine learning workspace with compute
# ---------------------------------------------------------------------------------------------------------------------
# See test/azure/terraform_azure_machine_learning_example_test.go for how to write automated tests for this code.
# ---------------------------------------------------------------------------------------------------------------------

provider "azurerm" {
features {
key_vault {
purge_soft_delete_on_destroy = true
}
}
}

terraform {
# This module is now only being tested with Terraform 0.13.x. However, to make upgrading easier, we are setting
# 0.12.26 as the minimum version, as that version added support for required_providers with source URLs, making it
# forwards compatible with 0.13.x code.
required_version = ">= 0.12.26"
required_providers {
azurerm = {
version = "~> 2.29"
source = "hashicorp/azurerm"
}
}
}

data "azurerm_client_config" "current" {}

resource "azurerm_resource_group" "ml_rg" {
name = "rg-ml-${var.postfix}"
location = var.location
}

resource "azurerm_application_insights" "ml_appinsights" {
name = "ai-mlworspace-${var.postfix}"
location = azurerm_resource_group.ml_rg.location
resource_group_name = azurerm_resource_group.ml_rg.name
application_type = "web"
}

resource "azurerm_key_vault" "ml_keyvault" {
name = "kv-mlworkspace-${var.postfix}"
location = azurerm_resource_group.ml_rg.location
resource_group_name = azurerm_resource_group.ml_rg.name
tenant_id = data.azurerm_client_config.current.tenant_id
sku_name = "premium"
}

resource "azurerm_storage_account" "ml_storageaccount" {
name = "samlwkspce${var.postfix}"
location = azurerm_resource_group.ml_rg.location
resource_group_name = azurerm_resource_group.ml_rg.name
account_tier = "Standard"
account_replication_type = "GRS"
}

resource "azurerm_machine_learning_workspace" "ml_workspace" {
name = "mlworkspace-${var.postfix}"
location = azurerm_resource_group.ml_rg.location
resource_group_name = azurerm_resource_group.ml_rg.name
application_insights_id = azurerm_application_insights.ml_appinsights.id
key_vault_id = azurerm_key_vault.ml_keyvault.id
storage_account_id = azurerm_storage_account.ml_storageaccount.id

identity {
type = "SystemAssigned"
}
}

resource "azurerm_machine_learning_compute_cluster" "ml_compute_cluster" {
name = "mlcomputecluster-${var.postfix}"
location = azurerm_resource_group.ml_rg.location
vm_priority = "LowPriority"
vm_size = "STANDARD_NC4AS_T4_V3"
machine_learning_workspace_id = azurerm_machine_learning_workspace.ml_workspace.id
scale_settings {
min_node_count = 0
max_node_count = 4
scale_down_nodes_after_idle_duration = "PT2M"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
output "resource_group_name" {
value = azurerm_resource_group.ml_rg.name
}

output "workspace_name" {
value = azurerm_machine_learning_workspace.ml_workspace.name
}

output "compute_name" {
value = azurerm_machine_learning_compute_cluster.ml_compute_cluster.name
}




Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# ---------------------------------------------------------------------------------------------------------------------
# ENVIRONMENT VARIABLES
# Define these secrets as environment variables
# ---------------------------------------------------------------------------------------------------------------------

# ARM_CLIENT_ID
# ARM_CLIENT_SECRET
# ARM_SUBSCRIPTION_ID
# ARM_TENANT_ID

# ---------------------------------------------------------------------------------------------------------------------
# REQUIRED PARAMETERS
# You must provide a value for each of these parameters.
# ---------------------------------------------------------------------------------------------------------------------

# ---------------------------------------------------------------------------------------------------------------------
# OPTIONAL PARAMETERS
# These parameters have reasonable defaults.
# ---------------------------------------------------------------------------------------------------------------------

variable "postfix" {
description = "A postfix string to centrally mitigate resource name collisions"
type = string
default = "resource"
}

variable "location" {
description = "The Azure region in which to deploy your resources to"
type = string
default = "East US"
}
11 changes: 11 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -48,15 +48,26 @@ require (

require (
cloud.google.com/go/cloudbuild v1.9.0
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.4.0
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.2
github.com/slack-go/slack v0.10.3
gotest.tools/v3 v3.0.3
)

require (
github.com/Azure/azure-sdk-for-go/sdk/internal v1.2.0 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v0.9.0 // indirect
github.com/golang-jwt/jwt/v4 v4.5.0 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 // indirect
)

require (
cloud.google.com/go/compute v1.19.1 // indirect
cloud.google.com/go/compute/metadata v0.2.3 // indirect
cloud.google.com/go/iam v0.13.0 // indirect
cloud.google.com/go/longrunning v0.4.1 // indirect
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/machinelearning/armmachinelearning/v3 v3.1.1
github.com/Azure/go-autorest v14.2.0+incompatible // indirect
github.com/Azure/go-autorest/autorest/adal v0.9.13 // indirect
github.com/Azure/go-autorest/autorest/azure/cli v0.4.2 // indirect
Expand Down
17 changes: 17 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,14 @@ dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7
github.com/Azure/azure-sdk-for-go v16.2.1+incompatible/go.mod h1:9XXNKU+eRnpl9moKnB4QOLf1HestfXbmab5FXxiDBjc=
github.com/Azure/azure-sdk-for-go v51.0.0+incompatible h1:p7blnyJSjJqf5jflHbSGhIhEpXIgIFmYZNg5uwqweso=
github.com/Azure/azure-sdk-for-go v51.0.0+incompatible/go.mod h1:9XXNKU+eRnpl9moKnB4QOLf1HestfXbmab5FXxiDBjc=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.4.0 h1:rTnT/Jrcm+figWlYz4Ixzt0SJVR2cMC8lvZcimipiEY=
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.4.0/go.mod h1:ON4tFdPTwRcgWEaVDrN3584Ef+b7GgSJaXxe5fW9t4M=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.2 h1:uqM+VoHjVH6zdlkLF2b6O0ZANcHoj3rO0PoQ3jglUJA=
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.2.2/go.mod h1:twTKAa1E6hLmSDjLhaCkbTMQKc7p/rNLU40rLxGEOCI=
github.com/Azure/azure-sdk-for-go/sdk/internal v1.2.0 h1:leh5DwKv6Ihwi+h60uHtn6UWAxBbZ0q8DwQVMzf61zw=
github.com/Azure/azure-sdk-for-go/sdk/internal v1.2.0/go.mod h1:eWRD7oawr1Mu1sLCawqVc0CUiF43ia3qQMxLscsKQ9w=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/machinelearning/armmachinelearning/v3 v3.1.1 h1:yedPrI/9Zg64W8ErU5WrEj6CKSanTYEAUwjj1iToh0Q=
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/machinelearning/armmachinelearning/v3 v3.1.1/go.mod h1:t/CNx0HBjwue7108IQKbHLiS2kFZ/5prJ9sjXEzb6Cc=
github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78/go.mod h1:LmzpDX56iTiv29bbRTIsUNlaFfuhWRQBWjQdVyAevI8=
github.com/Azure/go-autorest v10.8.1+incompatible/go.mod h1:r+4oMnoxhatjLLJ6zxSWATqVooLgysK6ZNox3g/xq24=
github.com/Azure/go-autorest v14.2.0+incompatible h1:V5VMDjClD3GiElqLWO7mz2MxNAK/vTfRHdAubSIPRgs=
Expand Down Expand Up @@ -225,6 +233,8 @@ github.com/Azure/go-autorest/logger v0.2.1 h1:IG7i4p/mDa2Ce4TRyAO8IHnVhAVF3RFU+Z
github.com/Azure/go-autorest/logger v0.2.1/go.mod h1:T9E3cAhj2VqvPOtCYAvby9aBXkZmbF5NWuPV8+WeEW8=
github.com/Azure/go-autorest/tracing v0.6.0 h1:TYi4+3m5t6K48TGI9AUdb+IzbnSxvnvUMfuitfgcfuo=
github.com/Azure/go-autorest/tracing v0.6.0/go.mod h1:+vhtPC754Xsa23ID7GlGsrdKBpUA79WCAKPPZVC2DeU=
github.com/AzureAD/microsoft-authentication-library-for-go v0.9.0 h1:UE9n9rkJF62ArLb1F3DEjRt8O3jLwMWdSoypKV4f3MU=
github.com/AzureAD/microsoft-authentication-library-for-go v0.9.0/go.mod h1:kgDmCTgBzIEPFElEF+FK0SdjAor06dRq2Go927dnQ6o=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/Microsoft/go-winio v0.4.11/go.mod h1:VhR8bwka0BXejwEJY73c50VrPtXAaKcyvVC4A4RozmA=
Expand Down Expand Up @@ -436,6 +446,8 @@ github.com/dimchansky/utfbom v1.1.0/go.mod h1:rO41eb7gLfo8SF1jd9F8HplJm1Fewwi4mQ
github.com/dimchansky/utfbom v1.1.1 h1:vV6w1AhK4VMnhBno/TPVCoK9U/LP0PkLCS9tbxHdi/U=
github.com/dimchansky/utfbom v1.1.1/go.mod h1:SxdoEBH5qIqFocHMyGOXVAybYJdr71b1Q/j0mACtrfE=
github.com/dnaeon/go-vcr v1.0.1/go.mod h1:aBB1+wY4s93YsC3HHjMBMrwTj2R9FHDzUr9KyGc8n1E=
github.com/dnaeon/go-vcr v1.1.0 h1:ReYa/UBrRyQdant9B4fNHGoCNKw6qh6P0fsdGmZpR7c=
github.com/dnaeon/go-vcr v1.1.0/go.mod h1:M7tiix8f0r6mKKJ3Yq/kqU1OYf3MnfmBWVbPx/yU9ko=
github.com/docker/cli v20.10.7+incompatible h1:pv/3NqibQKphWZiAskMzdz8w0PRbtTaEB+f6NwdU7Is=
github.com/docker/cli v20.10.7+incompatible/go.mod h1:JLrzqnKDaYBop7H2jaqPtU4hHvMKP+vjCwu2uszcLI8=
github.com/docker/distribution v0.0.0-20190905152932-14b96e55d84c/go.mod h1:0+TTO4EOBfRPhZXAeF1Vu+W3hHZ8eLp8PgKVZlcvtFY=
Expand Down Expand Up @@ -539,6 +551,8 @@ github.com/gogo/protobuf v1.3.0/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXP
github.com/gogo/protobuf v1.3.1/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o=
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/golang-jwt/jwt/v4 v4.5.0 h1:7cYmW1XlMY7h7ii7UhUyChSgS5wUJEnm9uZVTGqOWzg=
github.com/golang-jwt/jwt/v4 v4.5.0/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/groupcache v0.0.0-20160516000752-02826c3e7903/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
Expand Down Expand Up @@ -883,6 +897,8 @@ github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/9
github.com/pelletier/go-toml v1.8.1/go.mod h1:T2/BmBdy8dvIRq1a/8aqjN41wvWlN4lrapLU/GW4pbc=
github.com/pelletier/go-toml v1.9.3/go.mod h1:u1nR/EPcESfeI/szUZKdtJ0xRNbUoANCkoOuaOx1Y+c=
github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU=
github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 h1:KoWmjvw+nsYOo29YJK9vDA65RGE3NrOnUtO7a+RF9HU=
github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8/go.mod h1:HKlIX3XHQyzLZPlr7++PzdhaXEj94dEiJgZDTsxEqUI=
github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.8.1-0.20171018195549-f15c970de5b7/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
Expand Down Expand Up @@ -1319,6 +1335,7 @@ golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20210514084401-e8d321eab015/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210603125802-9665404d3644/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210616045830-e2b7044e8c71/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210806184541-e5e7981a1069/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
Expand Down
47 changes: 47 additions & 0 deletions modules/azure/client_factory.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@ import (
"github.com/Azure/azure-sdk-for-go/profiles/latest/sql/mgmt/sql"
"github.com/Azure/azure-sdk-for-go/profiles/preview/cosmos-db/mgmt/documentdb"
"github.com/Azure/azure-sdk-for-go/profiles/preview/preview/monitor/mgmt/insights"
"github.com/Azure/azure-sdk-for-go/sdk/azcore/arm/policy"
"github.com/Azure/azure-sdk-for-go/sdk/azidentity"
machinelearning "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/machinelearning/armmachinelearning/v3"
"github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2019-07-01/compute"
"github.com/Azure/azure-sdk-for-go/services/containerinstance/mgmt/2018-10-01/containerinstance"
"github.com/Azure/azure-sdk-for-go/services/containerregistry/mgmt/2019-05-01/containerregistry"
Expand Down Expand Up @@ -919,6 +922,50 @@ func CreateDataFactoriesClientE(subscriptionID string) (*datafactory.FactoriesCl
return &dataFactoryClient, nil
}

// CreateMachinelearningWorkspaceClientE is a helper function that will setup a machine learning workspace client.
func CreateMachinelearningWorkspaceClientE(subscriptionID string, clientOptions *policy.ClientOptions) (*machinelearning.WorkspacesClient, error) {
// Validate Azure subscription ID
subscriptionID, err := getTargetAzureSubscription(subscriptionID)
if err != nil {
return nil, err
}

cred, err := azidentity.NewDefaultAzureCredential(nil)
if err != nil {
return nil, err
}

// Create a machinelearning client
mlWorkspaceClient, err := machinelearning.NewWorkspacesClient(subscriptionID, cred, clientOptions)
if err != nil {
return nil, err
}

return mlWorkspaceClient, nil
}

// CreateMachinelearningComputeClientE is a helper function that will setup a machine learning compute client.
func CreateMachinelearningComputeClientE(subscriptionID string) (*machinelearning.ComputeClient, error) {
// Validate Azure subscription ID
subscriptionID, err := getTargetAzureSubscription(subscriptionID)
if err != nil {
return nil, err
}

cred, err := azidentity.NewDefaultAzureCredential(nil)
if err != nil {
return nil, err
}

// Create a machinelearning client
mlComputeeClient, err := machinelearning.NewComputeClient(subscriptionID, cred, nil)
if err != nil {
return nil, err
}

return mlComputeeClient, nil
}

// GetKeyVaultURISuffixE returns the proper KeyVault URI suffix for the configured Azure environment.
// This function would fail the test if there is an error.
func GetKeyVaultURISuffixE() (string, error) {
Expand Down
Loading