Hello again!

In this post I’ll be continuing my series on Azure Private Link and DNS.  In my first post I gave some background into Private Link, how it came to be, and what it offers.  For this post I’ll be diving into some DNS patterns you can use to support name resolution with Private Link Endpoints for Azure services.  I’ll be covering the six scenarios below:

1. Default DNS pattern without Private Link Endpoint
2. Azure Private DNS pattern with a single virtual network
3. BYODNS (Bring your own DNS) in a hub and spoke architecture
4. BYODNS with a custom DNS forwarder in a hub and spoke architecture
5. BYODNS with the use of root hints in a hub and spoke architecture
6. BYODNS with the use of a custom DNS zone hosted in the BYODNS in a hub and spoke architecture

Before I jump into the scenarios, I want to cover some basic (and not so basic) DNS concepts.  If you know nothing about DNS, I’d highly suggest you stop reading here and take a quick few minutes to read through this DNS 101 by RedHat.  If you’ve operated a DNS service in a large enterprise, you can skip this section and jump into the scenarios.  If you only know the basics, read through the below or else you may not get much out of this post.

• A record – Translates a hostname to an IP address such as http://www.journeyofthegeek.com to 5.5.5.5
• CNAME record – Alias record where you can point on (FQDNs) fully qualified domain name to another to make it a domain a human can remember and for high availability configurations
•  Recursive Name Resolution – A DNS query where the client asks for a definitive answer to a query and relies on the upstream DNS server to resolve it to completion.  Forwarders such as Google DNS function as recursive resolvers.
• Iterative Name Resolution – A DNS query where a client receives back a referral to another server in place of resolving the query to completion.  Querying root hints often involves iterate name resolution.
• DNS Forwarder – Forward all queries the DNS service can’t resolve to an upstream DNS service.  These upstream servers are typically configure to perform recursive name resolution, but depending on your DNS service (such as Infoblox), you can configure it to request iterative name resolution.
• Conditional Forwarder – Forward queries for a specific DNS namespace to an upstream DNS service for resolution.
• Split-brain / Split Horizon DNS – A DNS configuration where a DNS namespace exists authoritatively across one or more DNS implementations.  A common use case is to have a single DNS namespace defined on Internet-resolvable public facing DNS servers and also on Intranet private facing DNS servers.  This allows trusted clients to reach the service via a private IP address and untrusted clients to reach the service via a public IP address.

If you can grasp the topics above, you’ll be in good shape for the rest of this post.

Scenario 1 – Default DNS Pattern Without Private Link Endpoint

Scenario 1

Before we jump into how DNS for Azure services works when Private Link Endpoint is introduced, let’s first look at how it works without it.  For this example, let’s look at a scenario where I’m using an VM (virtual machine) running in an VNet (virtual network) and am attempting to connect to an Azure SQL instance named db1.database.windows.net.  No Private Link Endpoint has been configured for the Azure SQL instance and the VNet is configured to use Azure-provided DNS and thus sends its DNS queries out the 168.63.129.16 virtual IP.  I explain how Azure-provided DNS works with the virtual IP in a prior blog post.  When I open SQL Server Management Studio and try to connect to d1.database.windows.net, my VM first needs to determine the IP address of the resource it needs to establish a TCP connection with.  For this it issues a DNS query to the Azure DNS service.

The FQDN (fully-qualified domain name) for your specific instance of an Azure service will more than likely have two or more CNAME records associated with it.  I don’t have any super secret information as to the official reasons behind these CNAMEs and can only theorize that they are used to orchestrate high availability of the service.  By using the CNAMEs Microsoft is able to to provide you with DNS record you can customize to your requirements and place in code.  Any failures in the backend require a simple modification of the alias the CNAME is pointing to without requiring changes to your code such as modifications to the connection string.

Since Azure DNS is a recursive DNS resolver, it handles resolving each of these records for you and returns the public IP address of your Azure SQL instance.  Your VM will then use this public IP address to setup a TCP connection and establish a connection to your database.

Scenario 2 – Azure Private DNS pattern with a single virtual network

Scenario 2

Now let’s cover how things change when we add a Private Link Endpoint and configure it to integrate with Azure Private DNS.  If you’re unfamiliar with how Azure Private DNS works take a read from my prior post on the topic.

In this scenario I’ve added a Private Link Endpoint for my Azure SQL instance.  I’ve configured the Endpoint to integrate with an Azure Private DNS zone named privatelink.database.windows.net and have linked the VNet to the Azure Private DNS zone.

Notice the changes to the records in Azure Public DNS.  The hostname for my Azure SQL instance now has a CNAME record with an alias defined for db1.privatelink.database.windows.net.  There is also a new CNAME record for db1.privatelink.database.windows.net which points to the same dataslice4.eastus2.database.windows.net record as we saw in the last scenario.  This is done for two reasons.  The first reason is it allows clients accessing to instance through a public IP to continue to do so because Microsoft has established a split-brain DNS configuration for the privatelink.database.windows.net zone.  The second reason is it allows Microsoft to work some magic in the backend (I have no idea how they’re doing it) that redirects queries originating from an Azure VNet that is linked to the Azure Private DNS zone to be resolved against the record in the Azure Private DNS zone.

This means that clients outside the linked Azure VNet will receive back the public IP address of the Azure SQL instance and clients within the Azure VNet linked to the Azure Private DNS zone will receive back the private IP address of Private Link Endpoint.

Scenario 3 – BYODNS in a Hub and Spoke Architecture

Scenario 3

Scenarios 1 and 2 are important to understand, but the reality is very few organization have such a simple DNS pattern for their Azure footprints.  Most enterprises using Azure will be using a hub and spoke architecture.  Shared services such as a DNS service (Windows DNS, InfoBlox, BIND, whatever) are placed in the hub VNet and are shared among spoke VNets containing various workloads.  This DNS service will typically provide advanced features not provided by Azure Private DNS (at this time) such as conditional forwarders and DNS query logging.  You can check out my prior post on this pattern if you want to understand the details.

In the scenario below I’ve provisioned a DNS service in the hub VNet and configured it to forward all queries it can’t resolve to the 168.63.129.16 virtual IP.  Notice that I’ve now linked the Azure Private DNS zone to the hub VNet instead of the spoke VNet.  This is to ensure the DNS service can resolve the queries to this Azure Private DNS zone.  It also lets me take advantage of the advanced features of the DNS service such as those I discussed above.

The resolution with Azure-provided DNS occurs in the same manner as scenario 2 with the exception being that the DNS service performs the query and returns the results to the VM running in the spoke.

Scenario 4 – BYODNS With a Custom DNS Forwarder in a Hub and Spoke Architecture

Scenario 4

Next up we have a scenario similar to the above where we have a hub and spoke architecture and have the DNS service in the hub configured to forward all queries it can’t resolve to an upstream forwarder.  Maybe it’s to some on-premises DNS server, a 3rd party threat service, or simply Google’s DNS service.   Whatever the case, this scenario means we now have to care about recursive resolution and conditional forwarders.

If the upstream DNS service you’re using supports recursive name resolution and the DNS service you’re using in your hub is configured to send recursive queries to it, then any queries for db1.database.windows.net will resolve to the public IP address of the service.  The reason for this is with recursion you’re asking the upstream DNS service to chase down the answer for you and that upstream DNS service only knows about the public privatelink.database.windows.net DNS zone and does not have access to the Azure Private DNS zone.

To handle this scenario want to create a conditional forwarder for database.windows.net (or the recommended zone for the service you’re using) and point it to Azure-provided DNS via the 168.63.129.16 virtual IP.  This enables you to let the Azure platform handle the split-brain DNS challenge as it has been engineered to do.

Scenario 5 – BYODNS With The Use of Root Hints in a Hub and Spoke Architecture

Scenario 5

In scenario 5 we again have the same architecture as the prior scenarios with a few differences.  First off we are now sending iterative queries to the DNS Root Hints instead of an upstream forwarder.  This means our DNS service will chase the entirety of the resolution requesting referrals back from each DNS server in the path to resolve the FQDN.  The usage of iterative queries gives us the option of creating a conditional forwarder (our second difference) to the 168.63.129.16 for the privatelink.database.windows.net or optionally sending that query to some other DNS service we’re running in an on-premises data center or another cloud.

The key takeaway of this configuration is that using root hints puts a bigger burden on your DNS service because you are resolving a whole bunch more queries vs using an upstream DNS service like Azure DNS.   Additionally, if you opt to maintain your own DNS zone, it’s on you to figure out how to manage the whole lifecycle of the DNS records for the Private Link Endpoints.

Scenario 6 – BYODNS With The Use of a Custom DNS zone Hosted in The BYODNS In a Hub and Spoke Architecture

Scenario 6

The last scenario I’ll cover is the use of a custom DNS zone named something outside of the Microsoft recommended zones (more required than recommended) that is hosted in your BYODNS service.  Let me save you any pain and suffering by telling you this will not work.  You’re probably asking why it won’t work.  The answer to that question requires understanding how data is secured in transit to Azure services.

Since you surely don’t want your data flowing through a network in clear text, most Azure services will either require or support encryption of data in transit using TLS (Transport Layer Security).  If you’re not familiar with TLS flow, you get a reasonably good overview here.  The key thing you want to understand is that TLS session is often established by using the certificate being served up by the Azure service.  In addition to confidentiality, it also authenticates the service to your client.

The authentication piece is what we care about here.  Without going too deep into the weeds, the certificate contains a property called the SAN (subject alternative name) which lists the identities of the services the certificate should be used for.  These identities are typically DNS names such as db1.database.windows.net.  If you try to go ahead and create a custom DNS zone and attempt to access the Azure service through that name, you’ll run into a certificate mismatch error which is due to DNS name of the service you typed into your browser or that was called by your library not matching the identities listed in the certificate.

Yes I know there are ways to get around this by ignoring certificate mismatches (terrible security decision) or doing something funky like overriding database.windows.net (this is against Microsoft recommendations) with your own zone.  Don’t do this.  If you want the service to support this type of functionality, submit a feedback request.

Now if anyone is aware of a way to get around this limitation that is supported and not insane, I’d definitely be interested in hearing about it.

Before I conclude this series I want to provide one more gotcha.  Take note that while Private Link Endpoints can be integrated Azure Private DNS and the records can be automatically created, they do not share the full lifecycle.  This means that if you delete a private link endpoint and create a new one for the same resource, the NIC (network interface) associated with the endpoint may get a new IP.  This will cause your queries to fail to resolve because they will resolve to the prior IP.  You will need to manually clean up the A record hosted in the Azure Private DNS zone before creating the new endpoint.

Well folks that wraps it up.  Hopefully you found this information helpful and it cleared up some of the mystery of DNS patterns with Private Link Endpoints.  I want to plug a stellar write-up by Dan Mauser, who is one of the networking all stars over at Microsoft.  He wrote up an incredibly detailed post on this topic which covers the topic more exhaustively than I did above.

Thanks!