Errors when deploying a firewall rule with multiple IP groups

[leelax22]

Bicep version
Bicep CLI version 0.14.85 (f4a4d48)

Describe the bug
When deploying firewallpolicy, it seems to cause an error when two ipgroups are used.

To Reproduce
This is code I used.

param fwpObject object

resource fwpolicy 'Microsoft.Network/firewallPolicies@2022-07-01' = {
  name: fwpObject.name
  location: location
  properties: {
    sku: {      tier: 'Standard'    }
  }
}

resource ipgroup_nat 'Microsoft.Network/ipGroups@2022-07-01' = {
  name: 'ipgroup_nat'
  location: location
  properties: {
    ipAddresses: fwpObject.ipgroup_nat
  }
}

resource ipgroup_aks 'Microsoft.Network/ipGroups@2022-07-01' = {
  dependsOn: [ipgroup_nat]
  name: 'ipgroup_aks'
  location: location
  properties: {
    ipAddresses: fwpObject.ipgroup_aks
  }
}

resource rulecollection1 'Microsoft.Network/firewallPolicies/ruleCollectionGroups@2022-07-01' = {
  name: 'from_Internet_to_Jumpbox'
  parent: fwpolicy
  properties: {
    priority: 1000
    ruleCollections: [
      {
        name: 'from_Internet_to_Jumpbox'
        priority: 1010
        ruleCollectionType: 'FirewallPolicyNatRuleCollection'
        action: {
          type: 'DNAT'
        }
        rules: [
          {
            description: 'NAT TO Jumpbox'
            name: 'NAT_to_Jumpbox_linux'
            ruleType: 'NatRule'
            destinationAddresses: fwpObject.fw_addr
            destinationPorts: ['22']
            ipProtocols: ['TCP']
            // sourceAddresses: fwpObject.src_addr
            sourceIpGroups: [ipgroup_aks.id]
            translatedAddress: fwpObject.jumpbox_dst_addr
            // translatedFqdn: 'string'
            translatedPort: '22'
          }
        ]
      }
    ]
  }
}

resource rulecollection2 'Microsoft.Network/firewallPolicies/ruleCollectionGroups@2022-07-01' = {
  dependsOn: [rulecollection1]
  name: 'from_Jumpbox_to_Internet'
  parent: fwpolicy
  properties: {
    priority: 2000
    ruleCollections: [
      {
        name: 'from_DEV-VNet_to_Internet_AzCli_FQDN'
        priority: 2010
        ruleCollectionType: 'FirewallPolicyFilterRuleCollection'
        action: {
          type: 'allow'
        }
        rules: [
          {
            name: 'objects.githubusercontent.com'
            ruleType: 'ApplicationRule'
            // destinationAddresses: ['string']
            // fqdnTags: ['string']
            protocols: [{port: 443, protocolType: 'https'}]
            // sourceAddresses: ['string']
            sourceIpGroups: [ipgroup_aks.id]
            targetFqdns: ['objects.githubusercontent.com']
            // targetUrls: ['string']
            // terminateTLS: bool
            // webCategories: ['string']
          }
       ]
     }
   ]
  }
}

just deploy NAT rule collection using ipgroup_nat and APP rule collection using ipgroup_aks.

error message was:

New-AzDeployment: 오후 3:16:24 - The deployment 'aks-outbound-firewall' failed with error(s). Showing 1 out of 1 error(s).
Status Message: At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. (Code: DeploymentFailed)
 - {
  "status": "Failed",
  "error": {
    "code": "ResourceDeploymentFailure",
    "message": "The 'AzureAsyncOperationWaiting' resource operation completed with terminal provisioning state 'Failed'.",
    "details": [
      {
        "code": "IpGroupsUpdateFailed",
        "message": "Put on IP Groups ipgroup_nat Failed with 1 faulted referenced firewalls"
      }
    ]
  }
} (Code:Conflict)
CorrelationId: 8b410d6d-3199-424f-9e4f-34d69791ba0e

after using 1 ip groups (I unified the use of one IP group), it deployed succesfully.

It appears that other users have encountered similar issues with Terraform.
hashicorp/terraform-provider-azurerm#19843

[majastrz]

This looks like a question about a specific resource provider and not a Bicep bug. We will convert this to a discussion.

[kevball2]

Have you tried adding a dependency from the first IP Group to the second (dependsOn: [ipgroup_nat])? Forcing a dependcy on the completion of the first IPGroup may give it time to complete successfully. You could also put your IPGroups into an array and loop over them using the @batchsize(1) decorator to run the deployments serially.

I am seeing the same issue with deploying more than one firewall rule collection group. Azure support has recommended adding some kind of delay to allow for each to complete successfully, The @batchsize decorator should create this delay by adding a dependency for the previous iteration of the loop.

@batchSize(1)
module firewallPolicy_ruleCollectionGroups '../../CRML/ruleCollectionGroups/deploy.bicep' = [for (ruleCollectionGroup, index) in varApplicationFirewallRules: {
  name: '${uniqueString(deployment().name, parLocation)}-firewallPolicy_ruleCollectionGroups-${index}'
  params: {
    firewallPolicyName: resFirewallPolicies.name
    name: ruleCollectionGroup.name
    priority: ruleCollectionGroup.priority
    ruleCollections: ruleCollectionGroup.ruleCollections
  }
  dependsOn:[
    resHubVnet
  ]
}]

[dima-b]

I've tried to add dependencies, but it doesn't help.
It seems like a Bicep bug. If you run for the 1st time the template deploys without issues, but if you run the same code even without a change it fails. If you remove IP group, you can deploy the bicep again.

[bendiksygnestveit]

This is what we had to do, for each IPGroup we add we add a dependency on the previous IPGroup. It works fine.

What error are you getting @dima-b?

[leelax22]

Thank you for your response. I think the error is caused by trying to update the rule collection group of the firewall at the same time. I need to write the dependency and batchsize carefully to prevent the error.

[dima-b]

This is what we had to do, for each IPGroup we add we add a dependency on the previous IPGroup. It works fine.

What error are you getting @dima-b?

I keep getting "Put on IP Groups ipg-ipgname Failed with 1 faulted referenced firewalls", but it seems I discovered the root cause. It happens in case I use the same IP Group in 2 FirewallPolicyFilterRuleCollection. If to use one sourceAddresses instead of sourceIpGroups in any of them the deploy work with no issues.

resource workloadIpGroup 'Microsoft.Network/ipGroups@2022-09-01' = {
  name: workloadIpGroupName
  location: location
  properties: {
    ipAddresses: [
      '10.0.128.0/20'
    ]
  }
}

resource applicationRuleCollectionGroup 'Microsoft.Network/firewallPolicies/ruleCollectionGroups@2022-09-01' = {
  parent: firewallPolicy
  name: 'DefaultApplicationRuleCollectionGroup'
  dependsOn: [
    networkRuleCollectionGroup
  ]
  properties: {
    priority: 300
    ruleCollections: [      
      {
        ruleCollectionType: 'FirewallPolicyFilterRuleCollection'
        name: 'global-rule-url-arc'
        priority: 1000
        action: {
          type: 'Allow'
        }
        rules: [
          {
            ruleType: 'ApplicationRule'
            name: 'winupdate-rule-01'
            protocols: [
              {
                protocolType: 'Https'
                port: 443
              }
              {
                protocolType: 'Http'
                port: 80
              }
            ]
            fqdnTags: [
              'WindowsUpdate'
            ]
            terminateTLS: false            
            sourceIpGroups: [ //if to replace with sourceAddresses  deploy works.
              workloadIpGroup.id            
            ]
           }
        ]
      }
      {
        ruleCollectionType: 'FirewallPolicyFilterRuleCollection'
        action: {
          type: 'Allow'
        }
        name: 'Global-rules-arc'
        priority: 1202
        rules: [
          {
            ruleType: 'ApplicationRule'
            name: 'global-rule-01'
            protocols: [
              {
                protocolType: 'Https'
                port: 443
              }
            ]
            targetFqdns: [
              'www.microsoft.com'
            ]
            terminateTLS: false
            sourceIpGroups: [
              workloadIpGroup.id
            ]
          }
        ]
      }
    ]
  }
}

[kro-cat]

Just ran into this.
Updating an IP group will also (non-intuitively) update the firewall policy because the policy has sub resources which depend on the IP group.
Adding the IP group to the firewall policy's dependsOn parameter will avoid racing:

//...
resource workloadIpGroup 'Microsoft.Network/ipGroups@2022-09-01' = {
  name: workloadIpGroupName
  location: location
  properties: {
    ipAddresses: [
      '10.0.128.0/20'
    ]
  }
}

resource applicationRuleCollectionGroup 'Microsoft.Network/firewallPolicies/ruleCollectionGroups@2022-09-01' = {
  parent: firewallPolicy
  name: 'DefaultApplicationRuleCollectionGroup'
  dependsOn: [
    networkRuleCollectionGroup
    /*
     * wait for updates from workloadIpGroup to finish before attempting to
     * update this firewall policy, since it has sub resources which depend on the IP group.
     */
    workloadIpGroup
  ]
  //...
}