Avi ALB Controller Deployment on GCP Terraform module

This Terraform module creates and configures an AVI (NSX ALB) Controller on GCP

Module Functions

The module is meant to be modular and can create all or none of the prerequiste resources needed for the Avi GCP Deployment including:

• VPC and Subnet for the Controller (optional with create_networking variable)
• IAM Roles and Role Bindings for supplied Service Account (optional with create_iam variable)
• GCP Compute Image from the provided bucket controller file
• Firewall Rules for Avi Controller and SE communication
• GCP Compute Instance using the Controller Compute Image

During the creation of the Controller instance the following initialization steps are performed:

• Change default password to user specified password
• Copy Ansible playbook to controller using the assigned public IP
• Run Ansible playbook to configure initial settings and GCP Full Access Cloud

Optionally the following Avi configurations can be created:

• Avi IPAM Profile (configure_ipam_profile variable)
• Avi DNS Profile (configure_dns_profile variable)
• DNS Virtual Service (configure_dns_vs variable)

Environment Requirements

Google Cloud Platform

The following are GCP prerequisites for running this module:

• Service Account created for the Avi Controller
• Projects identified for the Controller, Network, Service Engines, Storage, and Backend Servers. By default this be the a single project as set by the "project" variable. Optionally the "network_project", "service_engine_project", "storage_project", and "server_project" variables can be set to use a different project than the project the Controller will be deployed to.
• If more than 1 project will be used "Disable Cross-Project Service Account Usage" organizational policy must be set to "Not enforced" and the the Service Account must be added to those additional projects.

Google Provider

For authenticating to GCP you must leverage either the "GOOGLE_APPLICATION_CREDENTIALS={{path_to_service_account_key}}" environment variable or use "gcloud auth application-default login"

Controller Image

The AVI Controller image for GCP should be uploaded to a GCP Cloud Storage bucket before running this module with the path specified in the controller-image-gs-path variable. This can be done with the following gsutil commands:

gsutil mb <bucket>
gsutil -m cp ./gcp_controller-<avi-version>.tar.gz  gs://<bucket>/

Host OS

The following packages must be installed on the host operating system:

• curl

Usage

terraform {
  backend "local" {
  }
}
provider "google" {
  project = "PROJECT"
  region  = "REGION"
}
module "avi_controller_gcp" {
  source  = "vmware/avi-alb-deployment-gcp/google"
  version = "1.0.x"

  region = "us-west1"
  create_networking = "true"
  create_iam = "false"
  controller_default_password = "Value Redacted and available within the VMware Customer Portal"
  avi_version = "21.1.1"
  service_account_email = "<sa-account>@<project>.iam.gserviceaccount.com"
  controller_image_gs_path = "<bucket>/gcp_controller-21.1.1.tar.gz"
  controller_password = "password"
  name_prefix = "avi"
  project = "gcp-project"
}
output "controller_address" { 
  value = module.avi_controller_gcp.controllers
} 

GSLB Deployment

For GSLB to be configured successfully the configure_gslb and configure_dns_vs variables must be configured. By default a new Service Engine Group (g-dns) and user (gslb-admin) will be created for the configuration.

The following is a description of the configure_gslb variable parameters and their usage:

Parameter	Description	Type
enabled	Must be set to "true" for Active GSLB sites	bool
leader	Must be set to "true" for only one GSLB site that will be the leader	bool
site_name	Name of the GSLB site	string
domains	List of GSLB domains that will be configured	list(string)
create_se_group	Determines whether a g-dns SE group will be created	bool
se_size	The CPU, Memory, Disk Size of the Service Engines. The default is 2 vCPU, 8 GB RAM, and a 30 GB Disk per Service Engine	string
additional_sites	Additional sites that will be configured. This parameter should only be set for the primary GSLB site	string

The example below shows a GSLB deployment with 2 regions utilized.

terraform {
  backend "local" {
  }
}
provider "google" {
  project = "PROJECT"
  region  = "us-east1"
  alias   = "east"
}
provider "google" {
  project = "PROJECT"
  region  = "us-west1"
  alias   = "west"
}
module "avi_controller_east" {
  source  = "vmware/avi-alb-deployment-gcp/google"
  version = "1.0.x"
  providers = {
    google = google.east
  }

  region                      = "us-east1"
  create_networking           = "false"
  custom_vpc_name             = "vpc"
  custom_subnetwork_name      = "subnet-east-1"
  create_iam                  = "false"
  avi_version                 = "22.1.2"
  controller_public_address   = "true"
  service_account_email       = "<email>@<account>.iam.gserviceaccount.com"
  controller_ha               = "true"
  controller_default_password = "<default-password>"
  controller_image_gs_path    = "<bucket>/gcp_controller-21.1.1.tar.gz"
  controller_password         = "<new-password>"
  name_prefix                 = "east1"
  project                     = "<project>"
  configure_ipam_profile      = { enabled = "true", networks = [{ network = "192.168.252.0/24" , static_pool = ["192.168.252.1", "192.168.252.254"]}] }
  configure_dns_profile       = { "enabled" = "true", usable_domains = ["east.avidemo.net"] }
  configure_dns_vs            = { "enabled" = "true", allocate_public_ip = "false", network = "192.168.252.0/24" }
  configure_gslb              = { enabled = "true", site_name = "East1" }
}
module "avi_controller_west" {
  source  = "vmware/avi-alb-deployment-gcp/google"
  version = "1.0.x"
  providers = {
    google = google.west
  }

  region                          = "us-west1"
  create_networking               = "false"
  custom_vpc_name                 = "vpc"
  custom_subnetwork_name          = "subnet-west-1"
  create_iam                      = "false"
  avi_version                     = "22.1.2"
  controller_public_address       = "true"
  service_account_email           = "<email>@<project>.iam.gserviceaccount.com"
  controller_ha                   = "true"
  controller_default_password     = "<default-password>"
  controller_image_gs_path        = "<bucket>/gcp_controller-21.1.1.tar.gz"
  controller_password             = "<new-password>"
  name_prefix                     = "west1"
  project                         = "<project>"
  configure_ipam_profile          = { enabled = "true", networks = { [{ network = "192.168.251.0/24" , static_pool = ["192.168.251.1", "192.168.251.254"]}] }
  configure_dns_profile           = { "enabled" = "true", allocate_public_ip = "false", network = "192.168.251.0/24" }
  configure_dns_vs            = { "enabled" = "true", usable_domains = ["west.avidemo.net"] }
  configure_gslb              = { enabled = "true", site_name = "West1", domains = ["gslb.avidemo.net"], [{name = "East1", ip_address_list = module.avi_controller_east.controllers[*].private_ip_address }] }
}
output "west_controller_ip" { 
  value = module.avi_controller_west.controllers
}
output "east_controller_ip" { 
  value = module.avi_controller_east.controllers
}

Controller Sizing

The controller_size variable can be used to determine the vCPU and Memory resources allocated to the Avi Controller. There are 3 available sizes for the Controller as documented below:

Size	vCPU Cores	Memory (GB)
small	8	24
medium	16	32
large	24	48

Additional resources on sizing the Avi Controller:

https://avinetworks.com/docs/latest/avi-controller-sizing/ https://avinetworks.com/docs/latest/system-limits/

Day 1 Ansible Configuration and Avi Resource Cleanup

The module copies and runs an Ansible play for configuring the initial day 1 Avi config. The plays listed below can be reviewed by connecting to the Avi Controller by SSH. In an HA setup the first controller will have these files.

avi-controller-gcp-all-in-one-play.yml

This play will configure the Avi Cloud, Network, IPAM/DNS profiles, DNS Virtual Service, GSLB depending on the variables used. The initial run of this play will output into the ansible-playbook.log file which can be reviewed to determine what tasks were ran.

Example run (appropriate variable values should be used):

~$ ansible-playbook avi-controller-gcp-all-in-one-play.yml -e password=${var.controller_password} > ansible-playbook-run.log

avi-upgrade.yml

This play will upgrade or patch the Avi Controller and SEs depending on the variables used. When ran this play will output into the ansible-playbook.log file which can be reviewed to determine what tasks were ran. This play can be ran during the initial Terraform deployment with the avi_upgrade variable as shown in the example below:

avi_upgrade = { enabled = "true", upgrade_type = "patch", upgrade_file_uri = "URL Copied From portal.avipulse.vmware.com"}

An full version upgrade can be done by changing changing the upgrade_type to "system". It is recommended to run this play in a lower environment before running in a production environment and is not recommended for a GSLB setup at this time.

Example run (appropriate variable values should be used):

~$ ansible-playbook avi-upgrade.yml -e password=${var.controller_password} -e upgrade_type=${var.avi_upgrade.upgrade_type} -e upgrade_file_uri=${var.avi_upgrade.upgrade_file_uri} > ansible-playbook-run.log

avi-cloud-services-registration.yml

This play will register the Controller with Avi Cloud Services. This can be done to enable centralized licensing, live security threat updates, and proactive support. When ran this play will output into the ansible-playbook.log file which can be reviewed to determine what tasks were ran. This play can be ran during the initial Terraform deployment with the register_controller variable as shown in the example below:

register_controller = { enabled = "true", jwt_token = "TOKEN", email = "EMAIL", organization_id = "LONG_ORG_ID" }

The organization_id can be found as the Long Organization ID field from https://console.cloud.vmware.com/csp/gateway/portal/#/organization/info.

The jwt_token can be retrieved at https://portal.avipulse.vmware.com/portal/controller/auth/cspctrllogin.

Example run (appropriate variable values should be used):

~$ ansible-playbook avi-cloud-services-registration.yml -e password=${var.controller_password} -e register_controller.jwt_token=${var.register_controller.jwt_token} > ansible-playbook-run.log

avi-cleanup.yml

This play will disable all Virtual Services, delete all existing Avi service engines, and de-register the controller from Cloud Services. This playbook should be ran before deleting the controller with terraform destroy to clean up the resources created by the Avi Controller. Note that additional items created by the controller may be created and need to be manually removed.

Example run (appropriate variable values should be used and -e register_controller.jwt_token is only needed when register_controller.enabled is set to true):

~$ ansible-playbook avi-cleanup.yml -e password=${var.controller_password} -e register_controller.jwt_token=${var.register_controller.jwt_token}

Contributing

The terraform-google-avi-alb-deployment-gcp project team welcomes contributions from the community. Before you start working with this project please read and sign our Contributor License Agreement (https://cla.vmware.com/cla/1/preview). If you wish to contribute code and you have not signed our Contributor Licence Agreement (CLA), our bot will prompt you to do so when you open a Pull Request. For any questions about the CLA process, please refer to our FAQ. For more detailed information, refer to CONTRIBUTING.md.

Requirements

Name	Version
terraform	>= 1.3.0
google	~> 4.41.0
null	3.2.0

Providers

Name	Version
google	4.41.0
null	3.2.0

Modules

No modules.

Resources

Name	Type
google_compute_firewall.avi_controller_mgmt	resource
google_compute_firewall.avi_controller_to_controller	resource
google_compute_firewall.avi_se_data	resource
google_compute_firewall.avi_se_mgmt	resource
google_compute_firewall.avi_se_to_se	resource
google_compute_image.controller	resource
google_compute_instance.avi_controller	resource
google_compute_network.vpc_network	resource
google_compute_router.avi	resource
google_compute_subnetwork.avi	resource
google_project_iam_custom_role.autoscaling_se	resource
google_project_iam_custom_role.cluster_vip	resource
google_project_iam_custom_role.ilb_byoip	resource
google_project_iam_custom_role.network	resource
google_project_iam_custom_role.server	resource
google_project_iam_custom_role.serviceengine	resource
google_project_iam_custom_role.storage	resource
google_project_iam_member.avi_autoscaling_role	resource
google_project_iam_member.avi_cluster_vip_role	resource
google_project_iam_member.avi_ilb_byoip_role	resource
google_project_iam_member.avi_network_role	resource
google_project_iam_member.avi_se_role	resource
google_project_iam_member.avi_se_service_account_role	resource
google_project_iam_member.avi_server_role	resource
google_project_iam_member.avi_storage_role	resource
null_resource.ansible_provisioner	resource
google_compute_zones.available	data source
google_service_account.avi	data source

Inputs

Name	Description	Type	Default	Required
avi_subnet	The CIDR that will be used for creating a subnet in the Avi VPC	string	"10.255.1.0/24"	no
avi_upgrade	This variable determines if a patch upgrade is performed after install. The enabled key should be set to true and the url from the Avi Cloud Services portal for the should be set for the upgrade_file_uri key. Valid upgrade_type values are patch or system	object({ enabled = bool, upgrade_type = string, upgrade_file_uri = string })	{ "enabled": "false", "upgrade_file_uri": "", "upgrade_type": "patch" }	no
avi_version	The version of Avi that will be deployed	string	n/a	yes
boot_disk_size	The boot disk size for the Avi controller	number	128	no
ca_certificates	Import one or more Root or Intermediate Certificate Authority SSL certificates for the controller. The certificate must be in the PEM format and base64 encoded without line breaks. An example command for generating the proper format is 'base64 -w 0 ca.pem > ca.base64'	list(object({ name = string, certificate = string }))	[ { "certificate": "", "name": "" } ]	no
cluster_ip	Sets the IP address of the Avi Controller cluster. This address must be in the same subnet as the Avi Controller VMs.	string	null	no
configure_controller	Configure the Avi Controller via Ansible after controller deployment. If set to false all configuration must be done manually with the desired config. The avi-controller-gcp-all-in-one-play.yml Ansible play will still be generated and copied to the first controller in the cluster	bool	"true"	no
configure_dns_profile	Configure a DNS Profile for DNS Record Creation for Virtual Services. The usable_domains is a list of domains that Avi will be the Authoritative Nameserver for and NS records may need to be created pointing to the Avi Service Engine addresses. Supported profiles for the type parameter are AWS or AVI	object({ enabled = bool, type = optional(string, "AVI"), usable_domains = list(string), ttl = optional(string, "30"), aws_profile = optional(object({ iam_assume_role = string, region = string, vpc_id = string, access_key_id = string, secret_access_key = string })) })	{ "enabled": false, "type": "AVI", "usable_domains": [] }	no
configure_dns_vs	Create Avi DNS Virtual Service. The subnet_name parameter must be an existing AWS Subnet. If the allocate_public_ip parameter is set to true a EIP will be allocated for the VS. The VS IP address will automatically be allocated via the AWS IPAM	object({ enabled = bool, allocate_public_ip = optional(bool, false), network = string, auto_allocate_ip = optional(bool, true), vs_ip = optional(string) })	{ "enabled": "false", "network": "" }	no
configure_firewall_se_data	Configure Firewall rules for SE dataplane traffic. If set the firewall_se_data_rules and firewall_se_data_source_range must be set	bool	"false"	no
configure_gslb	Configures GSLB. In addition the configure_dns_vs variable must also be set for GSLB to be configured. See the GSLB Deployment README section for more information.	object({ enabled = bool, leader = optional(bool, false), site_name = string, domains = optional(list(string)), create_se_group = optional(bool, true), se_size = optional(list(string), ["2", "8", "30"]), additional_sites = optional(list(object({ name = string, ip_address_list = list(string) }))) })	{ "domains": [ "" ], "enabled": "false", "site_name": "" }	no
configure_ipam_profile	Configure Avi IPAM Profile for Virtual Service Address Allocation. Example: { enabled = "true", networks = [{ network = "192.168.1.0/24" , static_pool = ["192.168.1.10","192.168.1.30"]}] }	object({ enabled = bool, networks = list(object({ network = string, static_pool = list(string) })) })	{ "enabled": "false", "networks": [ { "network": "", "static_pool": [ "" ] } ] }	no
controller_default_password	This is the default password for the Avi controller image and can be found in the image download page.	string	n/a	yes
controller_ha	If true a HA controller cluster is deployed and configured	bool	"false"	no
controller_image_gs_path	The Google Storage path to the GCP Avi Controller tar.gz image file using the bucket/filename syntax	string	n/a	yes
controller_password	The password that will be used authenticating with the Avi Controller. This password be a minimum of 8 characters and contain at least one each of uppercase, lowercase, numbers, and special characters	string	n/a	yes
controller_public_address	This variable controls if the Controller has a Public IP Address. When set to false the Ansible provisioner will connect to the private IP of the Controller.	bool	"false"	no
controller_size	This value determines the number of vCPUs and memory allocated for the Avi Controller. Possible values are small, medium, or large.	string	"small"	no
create_cloud_router	This variable is used to create a GCP Cloud Router when both the create_networking variable = true and the vip_allocation_strategy = ILB	bool	"false"	no
create_firewall_rules	This variable controls the VPC firewall rule creation for the Avi deployment. When set to false the necessary firewall rules must be in place before the deployment	bool	"true"	no
create_iam	Create IAM Roles and Role Bindings necessary for the Avi GCP Full Access Cloud. If not set the Roles and permissions in this document must be associated with the controller service account - https://Avinetworks.com/docs/latest/gcp-full-access-roles-and-permissions/	bool	"false"	no
create_networking	This variable controls the VPC and subnet creation for the Avi Controller. When set to false the custom_vpc_name and custom_subnetwork_name must be set.	bool	"true"	no
custom_machine_type	This value overides the machine type used for the Avi Controller	string	""	no
custom_subnetwork_name	This field can be used to specify an existing VPC subnetwork for the controller and SEs. The create_networking variable must also be set to false for this network to be used.	string	null	no
custom_vpc_name	This field can be used to specify an existing VPC for the controller and SEs. The create_networking variable must also be set to false for this network to be used.	string	null	no
dns_search_domain	The optional DNS search domain that will be used by the controller	string	""	no
dns_servers	The optional DNS servers that will be used for local DNS resolution by the controller. Example ["8.8.4.4", "8.8.8.8"]	list(string)	null	no
email_config	The Email settings that will be used for sending password reset information or for trigged alerts. The default setting will send emails directly from the Avi Controller	object({ smtp_type = string, from_email = string, mail_server_name = string, mail_server_port = string, auth_username = string, auth_password = string })	{ "auth_password": "", "auth_username": "", "from_email": "[email protected]", "mail_server_name": "localhost", "mail_server_port": "25", "smtp_type": "SMTP_LOCAL_HOST" }	no
firewall_controller_allow_source_range	The IP range allowed to connect to the Avi Controller. Access from all IP ranges will be allowed by default	string	"0.0.0.0/0"	no
firewall_se_data_rules	The ports allowed for Virtual Services hosted on Services Engines. The configure_firewall_se_data variable must be set to true for this rule to be created	list(object({ protocol = string, port = list(string) }))	[ { "port": [ "443", "53" ], "protocol": "tcp" }, { "port": [ "53" ], "protocol": "udp" } ]	no
firewall_se_data_source_range	The IP range allowed to access Virtual Services hosted on Service Engines. The configure_firewall_se_data and firewall_se_data_rules variables must also be set	string	"0.0.0.0/0"	no
license_key	The license key that will be applied when the tier is set to ENTERPRISE with the license_tier variable	string	""	no
license_tier	The license tier to use for Avi. Possible values are ENTERPRISE_WITH_CLOUD_SERVICES or ENTERPRISE	string	"ENTERPRISE_WITH_CLOUD_SERVICES"	no
name_prefix	This prefix is appended to the names of the Controller and SEs	string	n/a	yes
network_project	The GCP Network project that the Controller and SEs will use. If not set the project variable will be used	string	""	no
ntp_servers	The NTP Servers that the Avi Controllers will use. The server should be a valid IP address (v4 or v6) or a DNS name. Valid options for type are V4, DNS, or V6	list(object({ addr = string, type = string }))	[ { "addr": "0.us.pool.ntp.org", "type": "DNS" }, { "addr": "1.us.pool.ntp.org", "type": "DNS" }, { "addr": "2.us.pool.ntp.org", "type": "DNS" }, { "addr": "3.us.pool.ntp.org", "type": "DNS" } ]	no
portal_certificate	Import a SSL certificate for the controller's web portal. The key and certificate must be in the PEM format and base64 encoded without line breaks. An example command for generating the proper format is 'base64 -w 0 certificate.pem > cert.base64'	object({ key = string, certificate = string, key_passphrase = optional(string) })	{ "certificate": "", "key": "" }	no
project	The project used for the Avi Controller	string	n/a	yes
region	The Region that the Avi controller and SEs will be deployed to	string	n/a	yes
register_controller	If enabled is set to true the controller will be registered and licensed with Avi Cloud Services. The Long Organization ID (organization_id) can be found from https://console.cloud.vmware.com/csp/gateway/portal/#/organization/info. The jwt_token can be retrieved at https://portal.avipulse.vmware.com/portal/controller/auth/cspctrllogin. Optionally the controller name and description used during the registration can be set; otherwise, the name_prefix and configure_gslb.site_name variables will be used.	object({ enabled = bool, jwt_token = string, email = string, organization_id = string, name = optional(string), description = optional(string) })	{ "email": "", "enabled": "false", "jwt_token": "", "organization_id": "" }	no
se_ha_mode	The HA mode of the Service Engine Group. Possible values active/active, n+m, or active/standby	string	"active/active"	no
se_service_account	This is the service account that will be leveraged by the Avi Service Engines. This is optional and only needed if using service accounts are used for GCP firewall rules in 20.1.7 - https://avinetworks.com/docs/20.1/gcp-firewall-rules/#firewall-rule-filtering-with-service-accounts	string	null	no
se_size	The CPU, Memory, Disk Size of the Service Engines. The default is 2 vCPU, 2 GB RAM, and a 15 GB Disk per Service Engine. Syntax ["cpu_cores", "memory_in_GB", "disk_size_in_GB"]	list(string)	[ "2", "2", "15" ]	no
securechannel_certificate	Import a SSL certificate for the controller's secure channel communication. Only if there is strict policy that requires all SSL certificates to be signed a specific CA should this variable be used otherwise the default generated certificate is recommended. The full cert chain is necessary and can be provided within the certificate PEM file or separately with the ca_certificates variable. The key and certificate must be in the PEM format and base64 encoded without line breaks. An example command for generating the proper format is 'base64 -w 0 certificate.pem > cert.base64'	object({ key = string, certificate = string, key_passphrase = optional(string) })	{ "certificate": "", "key": "" }	no
server_project	The backend server GCP Project. If not set the project variable will be used	string	""	no
service_account_email	This is the service account that will be leveraged by the Avi Controller. If the create-iam variable is true then this module will create the necessary custom roles and bindings for the SA	string	n/a	yes
service_engine_project	The project used for Avi Service Engines. If not set the project variable will be used	string	""	no
storage_project	The storage project used for the Avi Controller and SE Image. If not set the project variable will be used	string	""	no
vip_allocation_strategy	The VIP allocation strategy for the GCP Cloud - ROUTES or ILB	string	"ROUTES"	no

Outputs

Name	Description
controllers	The AVI Controller(s) Information