Recently, one of my customers asked for help in setting up their Azure Kubernetes Cluster specifically with associating an Azure Fileshare. In this post, I describe the requirements, the challenges I encountered, and the completed ARM template to automate the deployment of the Storage Account Private Endpoint.

Azure Kubernetes can be deployed in two styles: one being as a traditional cluster that is publicly available, and the other as a completely private cluster, where access is only available either through a bastion living on the same Virtual Network, or a tied network using ExpressRoute or VPN. For a sandbox environment, setting up a cluster using the former option is the easiest, but for organizations that answer to a compliance team, the private cluster option is the only choice they have.

With the private cluster deployed, the next requirement was to have shared storage available to the nodes in the cluster via SMB. I read through the documentation for AKS storage options¹ and confirmed that the Azure Files integration with AKS would satisfy the requirement. At each step of the way, I had to ensure that I was addressing the functional and compliance requirements to that one would not conflict with the other. Once it was decided to use Azure Files I verified that the compliance requirements to enforce transit over TLS and to deny public access to the storage account could be met. To confirm the functionality, I consulted the ARM Template Storage Account reference documentation². The requirements could be met with the following properties:

supportsHttpsTrafficOnly
allowBlobPublicAccess
minimumTlsVersion
largeFileSharesState

With the requirements in hand, I figured I would try building the storage account through the portal first. I followed the Storage Account wizard making sure that I toggled each of checkboxes in accordance to the above requirements. After successful verification, I set the deployment off to be processed. As expected, the storage account and associated private endpoint were created. One of the nicest features in the Azure portal that I regularly forget about is the fact that every single deployment through the portal can be visible as an ARM template. In this case, I took a look at the template that was created to see how well it mapped to the one that I was building.

For reference, I have a set of Virtual Networks (vnet) and associated subnets deployed in my sandbox account. I use them to test with on a regular basis but I never kept track of which I deployed via the portal and through ARM. In this case, I deployed the storage account and associated private endpoint to one of those subnets. After comparing my template to the one generated by the portal, I realized that the one from the portal was significantly more robust. There were several options set that I assumed were extra and I didn’t add them to mine.

I decided to run my template against an alternate pair of vnet and subnet…

{
"Deployment failed. Correlation ID": "redacted" {
    "error": {
          "code":
          "PrivateEndpointCannotBeCreatedInSubnetThatHasNetworkPoliciesEnabled",
              "message": "Private endpoint
              /subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Network/privateEndpoints/redacted
              cannot be created in a subnet
              /subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Network/virtualNetworks/redacted/subnets/redacted
              since it has private endpoint network policies enabled.",
                  "details": []
                    }
}

… and it failed!

I was baffled by the lack of consistency between deploying the solution in the portal, and using an ARM template. I was clearly missing something in my template that was being applied through the portal. I searched for the error:code in the Azure documentation which returned one reference: Disable network policies for private endpoints³

In the second paragraph in that piece of documentation is the following:

When using the portal to create a private endpoint, this setting is automatically disabled as part of the create process. Deployment using other clients requires an additional step to change this setting. You can disable the setting using cloud shell from the Azure portal, or local installations of Azure PowerShell, Azure CLI, or use Azure Resource Manager templates.

The field that I added to the template is on line 5 in the excerpt below:


1
2
3
4
5
6

{
  "name": "[parameters('subnetName')]",
  "properties": {
    "addressPrefix": "[parameters('subnetAddressSpace')]",
    "privateEndpointNetworkPolicies": "Disabled"
  }

As expected, as soon as I changed my template to include the field for the subnet, the deployment succeeded. The final version of the ARM template that I used to create the resources successfully is located in GitHub here.

This was a great exercise for me to learn more about building private endpoints in Azure, as well as practicing a bit more with ARM templates and resource dependencies within them.

Related Posts