A look at the Azure DNS Private Resolver

Posted on May 18, 2022 by mattfeltonma

10/12/22 Update – Private Resolver is now Generally Available!

Hello again!

Today I’m going to cover the new Azure DNS Private Resolver feature that recently went into public preview. I’ve written extensively about Azure DNS in the past and I recommend reading through that series if you’re new to the platform. It has grown to be significantly important in Azure architectures due to its role in name resolution for Private Endpoints. A common pain point for customers using Private Endpoints from on-premises is the requirement to have a VM in Azure capable of acting as a DNS proxy. This is explained in detail in this post. The Azure DNS Private Resolver seeks to ease that pain by providing a managed DNS solution capable of acting as a DNS proxy and conditional forwarder facilitating hybrid DNS resolution (for those of you coming from AWS, this is Azure’s Route 53 Resolver). Alexis Plantin beat me to the punch and put together a great write-up on the basics of the feature so my focus instead be on some additional scenarios and a pattern that I tested and validated.

I’m a big fan of keeping infrastructure services such as DNS centralized and under the management of central IT. This is one reason I’m partial to a landing zone with a dedicated shared services virtual network attached to the transit virtual network as illustrated in the image below. In this shared services virtual network you put your DNS, patching/update infrastructure, and potentially identity services such as Windows Active Directory. The virtual network and its resources can then be dropped into a dedicated subscription and locked down to central IT. Additionally, as an added bonus, keeping the transit virtual network dedicated to firewalls and virtual network gateways makes the eventual migration to Azure Virtual WAN that must easier.

The design I had in mind would place the Private Resolver in the shared services virtual network and would funnel all traffic to and from the resolver and on-premises or another spoke through the firewall in the transit virtual network. This way I could control the conversation, inspect the traffic if needed, and centrally log it. The lab environment I built to test the design is pictured below.

The first question I had was whether or not the inbound endpoint would obey the user defined routes in the custom route table I associated with the inbound endpoint subnet. To test this theory I made a DNS query from the VM running in spoke 2 to resolve an A record in a Private DNS Zone. This Private DNS Zone was only linked to the virtual network where the Private Resolvers were. If the inbound endpoint wasn’t capable of obeying the custom routes, then the return traffic would be dropped and my query would fail.

**Result of query from VM in another spoke**

Success! The inbound endpoint is returning traffic back through the firewall. Logs on the firewall confirm the traffic flowing through.

**Firewall logs showing DNS traffic from spoke**

Next I wanted to see if traffic from the outbound endpoint would obey the custom routes. To test this, I configured a DNS forwarding rule (conditional forwarding component of the service) to send all DNS queries for jogcloud.com back to the domain controller running in my lab. I then performed a DNS query from the VM running in spoke 2.

**Firewall logs showing DNS traffic to on-premises**

Success again as the query was answered! The traffic from the outbound endpoint is seen traversing the firewall on its way to my domain controller on-premises. This confirmed that both the inbound and outbound endpoints obey custom routing making the design I presented above viable.

Beyond the above, I also confirmed the Private Resolver is capable of resolving reverse lookup zones (for PTR records). I was happy to see reverse zones weren’t forgotten.

One noticeable gap today is the Private Resolver does not yet offer DNS query logging. If that is important to you, you may want to retain your existing DNS Proxy. If you happen to be using Azure Firewall, you could make use of the DNS Proxy feature which allows for logging of DNS queries. Azure Firewall could then be configured to use the Private Resolver as its resolver providing that conditional forward capability Azure Firewall’s DNS Proxy feature lacks.

That wraps up this post.

Thanks!

A Pattern for App Services with Private Endpoints

Posted on July 8, 2021 by mattfeltonma

Hello again fellow geeks.

Last year Microsoft announced the general availability of Private Endpoint support for App Services. This was a feature I was particularly excited about because when combined with regional Vnet integration, it offered an alternative to an ASE (App Service Environment) for customers who wanted to run internally-facing applications or who didn’t feel comfortable exposing their publicly facing application directly via a public IP.

Over the past year I created a few different labs that demonstrated these features including a web app and function app. Each lab was built around a hub and spoke architecture where the application was serving as an internally-facing application. Given my recent post on Azure Firewall Premium, and the fact I still had the lab environment up and running, I thought it would be interesting to switch the virtual machine out and switch App Services in, which resulted in the lab environment pictured below.

I established the following goals for the lab environment:

Perform IDPS on traffic to the web app
Mediate and inspect traffic initiated from the web app to third-party APIs
Expose a web application to the Internet without giving it a direct public IP address

To accomplish these goals I was able to re-use much of the pattern I described in my last post.

The first step in the process was to create a very simple hub and spoke architecture as pictured above. The environment would consist of a transit Vnet (virtual network), shared services Vnet, and spoke Vnet. The transit Vnet would contain the guts of the solution with an Azure Firewall Premium SKU instance and Azure Application Gateway v2 instance. In the shared services Vnet I used a Windows Server VM (virtual machine) running the DNS Server service and providing DNS resolution for the environment. I could have taken a shortcut here and used Azure Firewall’s DNS proxy capability, but I like to leave the options open for conditional forwarding which Azure Firewall and Azure Private DNS do not support at this time. Finally, in the spoke Vnet I created two subnets. One subnet hosted the private endpoint for App Services and the other subnet was delegated to App Services to support regional Vnet integration.

Once all components were in place, I deployed a simple Python web application I wrote. All it does it queries two public APIs, one for the current time and the other for a random Breaking Bad quote (greatest show of all time!). I took the cheating way out and deployed the app directly via Visual Studio using the Azure App Service add-in prior to deploying the private endpoint and regional Vnet integration capabilities. This allowed me to test the app to validate it was still working as intended while also allowing me to deploy the app from my home machine. Deploying a private endpoint for app services locks down not only access to the running application but also to the SCM (source control management) endpoint that is used for deploying code to the app.

The next step was to create the Azure Private DNS Zone for app services which is named privatelink.azurewebsites.net. I then linked this zone to my shared services Vnet and setup the DNS server running in that Vnet with a standard forward to 168.63.129.16. This setup allows DNS queries made to the Private DNS zone to be resolved by the DNS server. I’ve written extensively about how DNS works with Azure Private Link so I won’t go into any more detail on that flow. Last step in the DNS process is to configure each Vnet to use the DNS server IP in its DNS Server settings. This also needs to configured directly on the Azure Firewall.

Now that the infrastructure would be able to resolve queries to the Private DNS Zone used by App Services, it was time to create the private endpoint for the web app. This registered the appropriate DNS records in the Private DNS Zone for my web app. With the private endpoint created, the last step was to enable Vnet integration.

Private DNS Zone with records for App Service instance

At this point I had all the guts of my solution and it was time to configure traffic to flow the way I wanted it to flow. Before I could begin work in Azure I needed to do the work outside of Azure. This involved creating an A record in my DNS hosting provider to point the public DNS name of my web app to the AGW’s (Application Gateway) public IP. This name can be whatever you want as long as you provide a certificate to the AGW such that it will identify itself as that name to the user. The AGW configuration is very straightforward and this tutorial will get you most of the way there. One thing to note is you’ll need to set the backend to point to the name given to the app service. This allows you to take advantage of the certificate Microsoft automatically provisions to the given app service.

Since I wanted to route traffic destined for the web app through the Azure Firewall Premium instance, I needed to ensure the AGW trusted the certificate served up by Azure Firewall. This is done by modifying the HTTP setting used in the AGW rule for the app. Here you can upload the root certificate that issued the Azure Firewall Premium intermediate certificate.

Now that the AGW is configured, I needed to create a route table for the subnet the AGW has its private IP address in. In this route table I disabled BGP (border gateway protocol) propagation to ensure the default route since this AGW v2 requires a default route pointing directly to the Internet. Now this is where it gets interesting from a routing perspective. Whenever you create a private endpoint for a service, a system route is added to the route table of all the subnets within the private endpoint’s vnet (virtual network) as well as any vnet the private endpoint’s vnet is peered with. If I want traffic from the AGW to route through the Azure Firewall, I need to override that system route with a /32 UDR. As you can imagine, this can become extremely tedious at scale and even risks hitting the max routes per route table depending on the scale we’re talking about. On the positive side, the issues around this are something Microsoft is aware of so hopefully that means this will be addressed at some point. In the meantime you’ll need to use the /32 route in a pattern such as this.

Azure Firewall Premium needs to be configured to allow traffic from the AGW to the web app and inspect this traffic. You can check out my last post for instructions on configured Azure Firewall to perform these activities.

Excellent, work is done right? Nope! If we influence routing on one side, we have to ensure the other side routes the same way. Toss a route table on the private endpoint subnet you say? Sorry, that isn’t supported my friend. Instead I needed to enable the Azure Firewall instance to SNAT (source NAT) traffic to the spoke Vnet. This ensures routing is symmetric and will eliminate any potential connection issues by creating only the UDR on the AGW subnet.

Lastly, I needed to create and apply a route table to the subnet I delegated to App Services for regional Vnet integration. This route table can be very simple and be configured with a single UDR for the default route pointing to the Azure Firewall private IP. This results in the traffic flow below:

This one was another fun pattern to solve. I got a chance to mess around more with AGW and also get a pattern I had theorized would work actually working. I find implementing the pretty pictures I draw helps drive home the benefits and considerations of such a pattern. With this pattern specifically, it suffers from similar considerations as the pattern in my last post. These include:

Challenges with observability
Operational overhead of certificate management
Possible latency issues depending on latency requirements and traffic patterns

This pattern tacks on more operational overhead by requiring that /32 route be added to the AGW route table each time a new app service is provisioned with a private endpoint. Observability further suffers when tracing a packet flow end-to-end due to the additional layer of SNAT required via Azure Firewall. One thing to note about this is you may be able to get around the SNAT requirement for web-specific traffic because of the transparent proxy functionality behind Azure Firewall application rules. I want to highlight I have not tested this myself and I typically tend to SNAT for this use case in my lab because many of my customers may use similar patterns but with a 3rd party firewall.

So there you go folks, another pattern to add to your inventory. Hopefully we’ll see the /32 issue issue private endpoints resolved sometime in the near future. Have a great weekend!

Azure Private Link and DNS – Part 2

Posted on March 6, 2020 by mattfeltonma

Hello again!

In this post I’ll be continuing my series on Azure Private Link and DNS. In my first post I gave some background into Private Link, how it came to be, and what it offers. For this post I’ll be diving into some DNS patterns you can use to support name resolution with Private Link Endpoints for Azure services. I’ll be covering the six scenarios below:

Default DNS pattern without Private Link Endpoint
Azure Private DNS pattern with a single virtual network
BYODNS (Bring your own DNS) in a hub and spoke architecture
BYODNS with a custom DNS forwarder in a hub and spoke architecture
BYODNS with the use of root hints in a hub and spoke architecture
BYODNS with the use of a custom DNS zone hosted in the BYODNS in a hub and spoke architecture

Before I jump into the scenarios, I want to cover some basic (and not so basic) DNS concepts. If you know nothing about DNS, I’d highly suggest you stop reading here and take a quick few minutes to read through this DNS 101 by RedHat. If you’ve operated a DNS service in a large enterprise, you can skip this section and jump into the scenarios. If you only know the basics, read through the below or else you may not get much out of this post.

A record – Translates a hostname to an IP address such as http://www.journeyofthegeek.com to 5.5.5.5
CNAME record – Alias record where you can point on (FQDNs) fully qualified domain name to another to make it a domain a human can remember and for high availability configurations
Recursive Name Resolution – A DNS query where the client asks for a definitive answer to a query and relies on the upstream DNS server to resolve it to completion. Forwarders such as Google DNS function as recursive resolvers.
Iterative Name Resolution – A DNS query where a client receives back a referral to another server in place of resolving the query to completion. Querying root hints often involves iterate name resolution.
DNS Forwarder – Forward all queries the DNS service can’t resolve to an upstream DNS service. These upstream servers are typically configure to perform recursive name resolution, but depending on your DNS service (such as Infoblox), you can configure it to request iterative name resolution.
Conditional Forwarder – Forward queries for a specific DNS namespace to an upstream DNS service for resolution.
Split-brain / Split Horizon DNS – A DNS configuration where a DNS namespace exists authoritatively across one or more DNS implementations. A common use case is to have a single DNS namespace defined on Internet-resolvable public facing DNS servers and also on Intranet private facing DNS servers. This allows trusted clients to reach the service via a private IP address and untrusted clients to reach the service via a public IP address.

If you can grasp the topics above, you’ll be in good shape for the rest of this post.

Scenario 1 – Default DNS Pattern Without Private Link Endpoint

Scenario 1

Before we jump into how DNS for Azure services works when Private Link Endpoint is introduced, let’s first look at how it works without it. For this example, let’s look at a scenario where I’m using an VM (virtual machine) running in an VNet (virtual network) and am attempting to connect to an Azure SQL instance named db1.database.windows.net. No Private Link Endpoint has been configured for the Azure SQL instance and the VNet is configured to use Azure-provided DNS and thus sends its DNS queries out the 168.63.129.16 virtual IP. I explain how Azure-provided DNS works with the virtual IP in a prior blog post. When I open SQL Server Management Studio and try to connect to d1.database.windows.net, my VM first needs to determine the IP address of the resource it needs to establish a TCP connection with. For this it issues a DNS query to the Azure DNS service.

The FQDN (fully-qualified domain name) for your specific instance of an Azure service will more than likely have two or more CNAME records associated with it. I don’t have any super secret information as to the official reasons behind these CNAMEs and can only theorize that they are used to orchestrate high availability of the service. By using the CNAMEs Microsoft is able to to provide you with DNS record you can customize to your requirements and place in code. Any failures in the backend require a simple modification of the alias the CNAME is pointing to without requiring changes to your code such as modifications to the connection string.

Since Azure DNS is a recursive DNS resolver, it handles resolving each of these records for you and returns the public IP address of your Azure SQL instance. Your VM will then use this public IP address to setup a TCP connection and establish a connection to your database.

Scenario 2 – Azure Private DNS pattern with a single virtual network

Scenario 2

Now let’s cover how things change when we add a Private Link Endpoint and configure it to integrate with Azure Private DNS. If you’re unfamiliar with how Azure Private DNS works take a read from my prior post on the topic.

In this scenario I’ve added a Private Link Endpoint for my Azure SQL instance. I’ve configured the Endpoint to integrate with an Azure Private DNS zone named privatelink.database.windows.net and have linked the VNet to the Azure Private DNS zone.

Notice the changes to the records in Azure Public DNS. The hostname for my Azure SQL instance now has a CNAME record with an alias defined for db1.privatelink.database.windows.net. There is also a new CNAME record for db1.privatelink.database.windows.net which points to the same dataslice4.eastus2.database.windows.net record as we saw in the last scenario. This is done for two reasons. The first reason is it allows clients accessing to instance through a public IP to continue to do so because Microsoft has established a split-brain DNS configuration for the privatelink.database.windows.net zone. The second reason is it allows Microsoft to work some magic in the backend (I have no idea how they’re doing it) that redirects queries originating from an Azure VNet that is linked to the Azure Private DNS zone to be resolved against the record in the Azure Private DNS zone.

This means that clients outside the linked Azure VNet will receive back the public IP address of the Azure SQL instance and clients within the Azure VNet linked to the Azure Private DNS zone will receive back the private IP address of Private Link Endpoint.

Scenario 3 – BYODNS in a Hub and Spoke Architecture

Scenario 3

Scenarios 1 and 2 are important to understand, but the reality is very few organization have such a simple DNS pattern for their Azure footprints. Most enterprises using Azure will be using a hub and spoke architecture. Shared services such as a DNS service (Windows DNS, InfoBlox, BIND, whatever) are placed in the hub VNet and are shared among spoke VNets containing various workloads. This DNS service will typically provide advanced features not provided by Azure Private DNS (at this time) such as conditional forwarders and DNS query logging. You can check out my prior post on this pattern if you want to understand the details.

In the scenario below I’ve provisioned a DNS service in the hub VNet and configured it to forward all queries it can’t resolve to the 168.63.129.16 virtual IP. Notice that I’ve now linked the Azure Private DNS zone to the hub VNet instead of the spoke VNet. This is to ensure the DNS service can resolve the queries to this Azure Private DNS zone. It also lets me take advantage of the advanced features of the DNS service such as those I discussed above.

The resolution with Azure-provided DNS occurs in the same manner as scenario 2 with the exception being that the DNS service performs the query and returns the results to the VM running in the spoke.

Scenario 4 – BYODNS With a Custom DNS Forwarder in a Hub and Spoke Architecture

Scenario 4

Next up we have a scenario similar to the above where we have a hub and spoke architecture and have the DNS service in the hub configured to forward all queries it can’t resolve to an upstream forwarder. Maybe it’s to some on-premises DNS server, a 3rd party threat service, or simply Google’s DNS service. Whatever the case, this scenario means we now have to care about recursive resolution and conditional forwarders.

If the upstream DNS service you’re using supports recursive name resolution and the DNS service you’re using in your hub is configured to send recursive queries to it, then any queries for db1.database.windows.net will resolve to the public IP address of the service. The reason for this is with recursion you’re asking the upstream DNS service to chase down the answer for you and that upstream DNS service only knows about the public privatelink.database.windows.net DNS zone and does not have access to the Azure Private DNS zone.

To handle this scenario want to create a conditional forwarder for database.windows.net (or the recommended zone for the service you’re using) and point it to Azure-provided DNS via the 168.63.129.16 virtual IP. This enables you to let the Azure platform handle the split-brain DNS challenge as it has been engineered to do.

Scenario 5 – BYODNS With The Use of Root Hints in a Hub and Spoke Architecture

Scenario 5

In scenario 5 we again have the same architecture as the prior scenarios with a few differences. First off we are now sending iterative queries to the DNS Root Hints instead of an upstream forwarder. This means our DNS service will chase the entirety of the resolution requesting referrals back from each DNS server in the path to resolve the FQDN. The usage of iterative queries gives us the option of creating a conditional forwarder (our second difference) to the 168.63.129.16 for the privatelink.database.windows.net or optionally sending that query to some other DNS service we’re running in an on-premises data center or another cloud.

The key takeaway of this configuration is that using root hints puts a bigger burden on your DNS service because you are resolving a whole bunch more queries vs using an upstream DNS service like Azure DNS. Additionally, if you opt to maintain your own DNS zone, it’s on you to figure out how to manage the whole lifecycle of the DNS records for the Private Link Endpoints.

Scenario 6 – BYODNS With The Use of a Custom DNS zone Hosted in The BYODNS In a Hub and Spoke Architecture

Scenario 6

The last scenario I’ll cover is the use of a custom DNS zone named something outside of the Microsoft recommended zones (more required than recommended) that is hosted in your BYODNS service. Let me save you any pain and suffering by telling you this will not work. You’re probably asking why it won’t work. The answer to that question requires understanding how data is secured in transit to Azure services.

Since you surely don’t want your data flowing through a network in clear text, most Azure services will either require or support encryption of data in transit using TLS (Transport Layer Security). If you’re not familiar with TLS flow, you get a reasonably good overview here. The key thing you want to understand is that TLS session is often established by using the certificate being served up by the Azure service. In addition to confidentiality, it also authenticates the service to your client.

The authentication piece is what we care about here. Without going too deep into the weeds, the certificate contains a property called the SAN (subject alternative name) which lists the identities of the services the certificate should be used for. These identities are typically DNS names such as db1.database.windows.net. If you try to go ahead and create a custom DNS zone and attempt to access the Azure service through that name, you’ll run into a certificate mismatch error which is due to DNS name of the service you typed into your browser or that was called by your library not matching the identities listed in the certificate.

Yes I know there are ways to get around this by ignoring certificate mismatches (terrible security decision) or doing something funky like overriding database.windows.net (this is against Microsoft recommendations) with your own zone. Don’t do this. If you want the service to support this type of functionality, submit a feedback request.

Now if anyone is aware of a way to get around this limitation that is supported and not insane, I’d definitely be interested in hearing about it.

Before I conclude this series I want to provide one more gotcha. Take note that while Private Link Endpoints can be integrated Azure Private DNS and the records can be automatically created, they do not share the full lifecycle. This means that if you delete a private link endpoint and create a new one for the same resource, the NIC (network interface) associated with the endpoint may get a new IP. This will cause your queries to fail to resolve because they will resolve to the prior IP. You will need to manually clean up the A record hosted in the Azure Private DNS zone before creating the new endpoint.

Well folks that wraps it up. Hopefully you found this information helpful and it cleared up some of the mystery of DNS patterns with Private Link Endpoints. I want to plug a stellar write-up by Dan Mauser, who is one of the networking all stars over at Microsoft. He wrote up an incredibly detailed post on this topic which covers the topic more exhaustively than I did above.

Thanks!

Azure Private Link and DNS – Part 1

Posted on March 5, 2020 by mattfeltonma

Hi there fellow geeks!

Azure Private Link is becoming a frequent topic of discussion among peers and my customers. One of the often discussed topics is how to handle DNS with Private Link Endpoints. I spent the past few days deep diving into the documentation and doing some labbing to better understand what the patterns and gotchas were. There seemed to be enough value to the findings to share it with you all.

Before I dive into the guts of Private Link Endpoints, I want to spend a post walking through how Private Link came to be.

Last September Microsoft released the Azure Private Link service. One of the primary drivers behind the introduction of the service was to address the customer demand for secure and private connectivity to Azure services such as Azure SQL and Azure Storage as well as third-party services. Azure PaaS services used to be accessible only via public IP addresses which required a path out to the Internet. From a network security perspective, your only option to use the firewall feature built into many of the services to filter the IPs allowed to communicate with the service. While technically feasible, there had to be something better.

The first attempt at something better was Service Endpoints, which started to be introduced into general availability in February 2018. For you AWS folk, the Service Endpoints are probably closest to VPC Gateway Endpoints. Service Endpoints attempted to improve the experience of accessing the services from a VNet (virtual network) by providing a direct route for resources in a VNet (virtual network) to Azure services in order to optimize routing. To mitigate the risk of the service being accessible over an public IP, Service Endpoints also added an identity to the VNet. This allowed customers to expand context of the filtering being done by the service firewall beyond IP to the identity of the VNet containing resources that need to access the relevant service.

Service Endpoints

While Service Endpoints made some great improvements there was more work to be done. Service Endpoints did nothing to mitigate the risk of data exfiltration. If an attacker was able to compromise a VM (virtual machine) in your VNet, that attacker could use that optimized route to their advantage piping whatever data they were able to get access to out to an attacker controlled instance of the resource such as an Azure Storage Account. Service Endpoint policies were then introduced to help address this risk.

Well that’s great an all, but Service Endpoints did nothing to address accessing Azure services from outside the VNet such as from an on-premises data center or another public cloud. Customers were still stuck accessing the services over the Internet or using an ExpressRoute using Microsoft Peering. Wouldn’t it be great there was a service with all of those features?

In comes Azure Private Link to the rescue. Azure Private Link includes the concept of an Azure Private Link Service and Private Link Endpoint. Those of you coming from AWS, yeah, I’ll let you guess which AWS service this is like :-). I won’t be covering Private Link Services in this series beyond saying it’s way to build your own third party services and make them directly accessible from a customer VNet. Instead we’ll keep our focus on Private Link Endpoints, specifically in the context of Microsoft-provided services.

The Private Link services introduces two new features that seek to address the gaps Service Endpoints did not and to include features from Service Endpoints that were beneficial. These features are:

Private access to services running on the Azure platform through the provisioning of a virtual network interface within the customer VNet that is assigned one of the VNet IP addresses from the RFC1918 address space.
Makes the services accessible over private IP space to resources running outside of Azure such as machines running in an on-premises data center or virtual machines running in other clouds.
Protects against data exfiltration by the endpoint providing access to only a specific instance of a PaaS service.

Azure Private Link

As you can see from the above, the service solves a lot of problems and is going to be a necessary component of any Azure footprint. Now when it comes to design and implementation, there are some options as to how you use DNS to resolve the name of the service resource being exposed by the endpoint to the private IP address of the Private Link Endpoint. This is what I’ll be focusing on for this series.

In the next post I’ll walk you through what happens within Azure DNS when you create a Private Link endpoint, some patterns you can use for DNS resolution, and some of the gotchas.

The series is continued in my second post.

Journey Of The Geek

The chronicles of a Bostonian tech geek navigating through life, technology, and general geekiness.

Tag Archives: dns