[Bug] Imported and re-deployed Machine Catalog deletion results in error

[anderswagner]

Describe the bug

Machine catalog started as a catalog spun up by calling the rest API, it was then imported into terraform state and re-deployed (along with the delivery group), and trying to remove it now results in an error.
The delivery group (which was also re-deployed after the import) was successfully deleted.

Terraform command (import, apply, etc): terraform apply
Resource impacted: citrix_machine_catalog

Versions

Use the terraform -v command to find the Terraform and Citrix Provider versions.
Terraform: 1.10.4
citrix/citrix provider: 1.0.13
Operation system: Linux

Terraform configuration files

resource "citrix_machine_catalog" "default" {
  name                        = local.catalog_name
  allocation_type             = var.catalog.allocationType
  zone                        = data.citrix_zone.azure_location_zone.id
  machine_catalog_folder_path = "${upper(var.environment_type)}\\${upper(var.tenant_short_name)}"

  provisioning_type = "MCS"
  session_support   = "SingleSession"
  scopes            = ["00000000-0000-0000-0000-000000000000"]

  persist_user_changes     = var.catalog.persistUserChanges != null ? var.catalog.persistUserChanges : "Discard"
  minimum_functional_level = var.catalog.functionalLevel

  delete_virtual_machines = true
  delete_machine_accounts = "Delete"

  provisioning_scheme = {
    hypervisor               = var.hypervisor_id
    hypervisor_resource_pool = var.hypervisor_resource_pool_id
    identity_type            = "ActiveDirectory"
    number_of_total_machines = var.catalog.numberOfMachines

    machine_domain_identity = {
      domain                   = var.domain_info.domain
      service_account          = var.domain_info.domain_admin_username
      service_account_password = var.domain_admin_password

      domain_ou = var.catalog.ou != null ? var.catalog.ou : local.default_ou
    }

    network_mapping = [{
      network        = var.catalog.subnetName
      network_device = "0"
    }]

    azure_machine_config = {
      storage_type = "Premium_LRS"
      license_type = "Windows_Client"

      service_offering   = var.catalog.vmSize
      vda_resource_group = var.vda_resource_group
      master_image_note  = var.image_version
      use_azure_compute_gallery = {
        replica_maximum = 40
        replica_ratio   = 100
      }

      azure_master_image = {
        resource_group      = var.sharedServices.imageGallery.resourceGroup
        shared_subscription = var.sharedServices.subscriptionId

        gallery_image = {
          gallery    = var.sharedServices.imageGallery.name
          definition = var.image_name
          version    = var.image_version
        }
      }
    }

    machine_account_creation_rules = {
      naming_scheme      = local.naming_scheme
      naming_scheme_type = "Numeric"
    }
  }
}

resource "citrix_delivery_group" "default" {
  name                       = local.catalog_name
  minimum_functional_level   = var.catalog.functionalLevel
  delivery_group_folder_path = "${upper(var.environment_type)}\\${upper(var.tenant_short_name)}"
  delivery_type              = var.catalog.deliveryType

  associated_machine_catalogs = [{
    machine_catalog = citrix_machine_catalog.default.id
    machine_count   = citrix_machine_catalog.default.provisioning_scheme.number_of_total_machines
  }]

  desktops = [{
    enabled                = true
    published_name         = var.catalog.publishedName
    enable_session_roaming = true
  }]

  autoscale_settings = {
    autoscale_enabled = true
  }

  restricted_access_users = {
    allow_list = [for group in var.catalog.deliveryCatalogGroups : lower("${var.tenant_short_name}\\${group}")]
  }

  lifecycle {
    ignore_changes = [
      autoscale_settings
    ]
    replace_triggered_by = [
      citrix_machine_catalog.default.id
    ]
  }
}

Terraform console output

╷
│ Error: Error deleting Machine Catalog DevOps - net99 - Admin - West Europe
│ 
│ TransactionId: 31d9caa9-83d1-4c1f-93fc-1ecb1058617b
│ JobId: 9ace27be-371d-4399-b540-551cc8675462
│ Error message: Failed to remove all machines and catalogs.
╵

[anderswagner]

A workaround we are using is using the powershell SDK to run the command Get-ProvScheme -ProvisioningScheme '*' | Remove-ProvScheme -PurgeDB and then deleting the machine catalogs manually in the citrix cloud UI. Unsure if this is an issue with the provider or the API in general. We would love for this to work by just making terraform do the deletion for us.

[aneeshk-citrix]

@anderswagner While we are investigating this, can you tell me what the behavior is when deleting this redeployed catalog using Studio UI alone?

[anderswagner]

Yes, I will try and get this scenario set up today so I can test it and tell you @aneeshk-citrix

[aneeshk-citrix]

@anderswagner Did you get the chance to replicate this behavior using Studio UI?

[anderswagner]

Sorry for the late response. I have not had the time to test it in the studio and will not be able to do so at the moment due to other priorities from our side sadly

[aneeshk-citrix]

@anderswagner I tried this in our test environments and was not able to repro.
Let us know if you're still facing issues. Please provide the new transaction ID so I can take a look at the latest logs.
Also, if you can try reproducing it using Studio UI that would also help.

[anderswagner]

This has been parked for us for now. Maybe to not polute your issue list this could be closed as stale.

[nicoBruhn]

@aneeshk-citrix Experienced this issue during terraform destroy on a terraform provisioned catalog. Looking in Citrix Daas no machines are left in the catalog and in Azure there are no machines left. It seems like the terraform provider is not cleaning up ProvVM's correctly in all cases.

│ Error: Error deleting Machine Catalog XXXXXXX
│ 
│ TransactionId: e0fd5cff-8a0c-4e5f-84ef-7e0ac1ad9b26
│ JobId: c17076b4-58eb-471b-a902-452a90bd4488
│ 
│ Error Message : Failed to remove all machines and catalogs.