Deploy without Azure Firewall and Private DNS Zones?

[baatch]

In the guidance docs for cost estimates (https://github.com/Azure/data-management-zone/blob/main/docs/guidance/EnterpriseScaleAnalytics-CostEstimates.md) there are references to:

Pricing Calculator - Data Management Zone w/o Azure Firewall and Private DNS Zones
Pricing Calculator - Data Management Zone w/ Azure Firewall and Private DNS Zones

How do you deploy the solution without Azure Firewall and Private DNS Zones?

In the params.dev.json file I could find the parameters "enableDnsAndFirewallDeployment" but even when setting that to false I still get deployment errors.

"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"BadRequest\",\r\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"LinkedInvalidPropertyId\\\",\\r\\n \\\"message\\\": \\\"Property id '' at path 'properties.privateDnsZoneConfigs[0].properties.privateDnsZoneId' is invalid. Expect fully qualified resource Id that start with '/subscriptions/{subscriptionId}' or '/providers/{resourceProviderNamespace}/'.\\\"\\r\\n }\\r\\n}\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}"

Some documentation around how to setup this would be helpful as deploying a POC/MVP that cost 1400$ per month is a no go for many companies.

[marvinbuss]

Hi @baatch,
Can you please click on containerResources and then on containerRegistry001 and provide me the error message, so that I can fix the issue?
Feel free to also ping me on Teams, if you are a Microsoft employee, to troubleshoot this quickly.
Thanks for your help!

[marvinbuss]

Thanks for confirming! 👍
I am glad that it all works now.

The Azure Firewall is required for any outbound traffic from the respective services (Data Factory, Azure Databricks, Azure Machine Learning, etc.). You can either rely on an existing Firewall in the tenant (most customers have a central NVA already deployed) or you deploy it as part of ESA. If the central Firewall is not available, outbound traffic may not work as expected.

Private DNS Zones are required for resolving the FQDNs to the private IPs of the private endpoints. Again, if customer already have an existing DNS infrastructure, ESA allows to make use of it. If none of this is available the DNS resolution to the private IPs of the private endpoints will not work and hence connectivity to services like storage or Synapse or other services will fail as they are all blocking public network access.

In our case, we are using the Azure Firewall also as DNS forwarder. If there is no DNS forwarder available, there will be no option to centrally resolve all DNS requests from the Data Landing Zones. Again, ESA allows to make use of existing infrastructure. If nothing of this is available, DNS resolution will again not work.

Some Firewall, DNS infrastructure and DNS forwarder is required. Eitehr it gets deployed through ESA or exusting infrastructure is used within the tenant. I hope that is clear?

[baatch]

@marvinbuss thanks, I think it is a little more clear.

But still my understanding is that there needs to deploy/configure something in place of the Azure Firewall and Global DNS if they are not included in the deployment to be able to fully evaluate the solution correct?

Can you build or recommend alternative solutions (linux firewalls) that will fit into the ESA and can be used for this scenario (small POC/MVP) that will be cheaper than 1400$ /month?

Otherwise I would think this can be a huge barrier for many to both adopt and learn if you need to get a budget of 1400$ / month just for the data management zone.

[marvinbuss]

Yes, that is correct. If you are not usng our Firewall, then you need to use an alternative.
In most environments and tenants, there is already an existing shared network infrastructure that can be used (Firewall, DNS, etc.).

If we move to a different solution, user will have to start manage the solution. Also, there is most likely no native ARM integration that we will be able to use to provide the same one-click experience. If you want to save some money, you can deploy ESA via CI/CD and delete the Firewall at the end of the day and then redeploy, when you start working in the morning through a pipeline.
Would that work?

I do understand that for a small use case team that just looks at integrating a dataset and developing some workflows, this may look expensive. However, we are targeting the data platform layer, which may span hundreds or thousands of use cases over time. For those scenarios, this should not be the biggest blocker.

Also, I believe that we should not trade less cost for much higher security risks.. We should be more concerned about cost and proper security measures and less about cost in these scenarios.

[baatch]

If you want to save some money, you can deploy ESA via CI/CD and delete the Firewall at the end of the day and then redeploy, when you start working in the morning through a pipeline.
Would that work?

By doing this, you mentioned that outbound traffic may not work? Can the solution still be evaluated fully and properly?

I agree that you should not trade cost for security and I fully understand that this is not targeting small teams/companies.

The larger enterprise companies and customers I'm working with are fine with the cost of the Azure firewall in production, but before getting that far they usually want try/evaluate the solution/architecture first to prove it's value. Can you provide guidance, tips or develop something for these scenarios?

[marvinbuss]

That is correct. Running workloads while the Azure Firewall is deleted will also fail, as the DNS resolution will not work. Therefore, the suggestion would only be an option while developing.
Thank you for acknowledging the trade-off between cost and security. 👍

What would you like to see from a value standpoint?
I believe that the current cost is justifiable if you are developing a new data platform for your organization or business.