Interesting behaviors with Private Endpoints

Interesting behaviors with Private Endpoints

Hi folks!

Working for and with organizations in highly regulated industries like federal and state governments and commercial banks often necessitates diving REALLY deep into products and technologies. This means peeling back the layers of the onion most people do not. The reason this pops up is because these organizations tend to have extremely complex environments due the length of time the organization has existed and the strict laws and regulations they must abide by. This is probably the reason why I’ve always gravitated towards these industries.

I recently ran into an interesting use case where that willingness to dive deep was needed.

A customer I was working with was wrapping up its Azure landing zone deployment and was beginning to deploy its initial workloads. A number of these workloads used Microsoft Azure PaaS (platform-as-a-service) services such as Azure Storage and Azure Key Vault. The customer had made the wise choice to consume the services through Azure Private Endpoints. I’m not going to go into detail on the basics of Azure Private Endpoints. There is plenty of official Microsoft documentation that can cover the basics and give you the marketing pitch. You can check out my pasts posts on the topic such as my series on Azure Private DNS and Azure Private Endpoints.

This particular customer chose to use them to consume the services over a private connection from both within Azure and on-premises as well as to mitigate the risk of data exfiltration that exists when egressing the traffic to Internet public endpoints or using Azure Service Endpoints. One of the additional requirements the customer had as to mediate the traffic to Azure Private Endpoints using a security appliance. The security appliance was acting as a firewall to control traffic to the Private Endpoints as well to perform deep packet inspection sometime in the future. This is the requirement that drove me down into the weeds of Private Endpoints and lead to a lot of interesting observations about the behaviors of network traffic flowing to and back from Private Endpoints. Those are the observations I’ll be sharing today.

For this lab, I’ll be using a slightly modified version of my simple hub and spoke lab. I’ve modified and added the following items:

  • Virtual machine in hub runs Microsoft Windows DNS and is configured to forward all DNS traffic to Azure DNS (168.63.129.16)
  • Virtual machine in spoke is configured to use virtual machine in hub as a DNS server
  • Removed the route table from the spoke data subnet
  • Azure Private DNS Zone hosting the privatelink.blob.core.windows.net namespace
  • Azure Storage Account named mftesting hosting some sample objects in blob storage
  • Private Endpoint for the mftesting storage account blob storage placed in the spoke data subnet
Lab environment

The first interesting observation I made was that there was a /32 route for the Private Endpoint. While this is documented, I had never noticed it. In fact most of my peers I ran this by were never aware of it either, largely because the only way you would see it is if you enumerated effective routes for a VM and looked closely for it. Below I’ve included a screenshot of the effective routes on the VM in the spoke Virtual Network where the Private Endpoint was provisioned.

Effective routes on spoke VM

Notice the next hop type of InterfaceEndpoint. I was unable to find the next hop type of InterfaceEndpoint documented in public documentation, but it is indeed related to Private Endpoints. The magic behind that next hop type isn’t something that Microsoft documents publicly.

Now this route is interesting for a few reasons. It doesn’t just propagate to all of the route tables of subnets within the Virtual Network, it also propagates to all of the route tables in directly peered Virtual Networks. In the hub and spoke architecture that is recommended for Microsoft Azure, this means that every Private Endpoint you create in a spoke Virtual Network is propagated to as a system route to route tables of each subnet in the hub Virtual Network. Below you can see a screen of the VM running in the hub Virtual Network.

Effective routes on hub VM

This can make things complicated if you have a requirement such as the customer I was working with where the customer wants to control network traffic to the Private Endpoint. The only way to do that completely is to create a /32 UDRs (user defined routes) in every route table in both the hub and spoke. With a limit of 400 UDRs per route table, you can quickly see how this may break down at scale.

There is another interesting thing about this route. Recall from effective routes for the spoke VM, that there is a /32 system route for the Private Endpoint. Since this is the most specific route, all traffic should be routed directly to the Private Endpoint right? Let’s check that out. Here I ran a port scan against the Private Endpoint using nmap using the ICMP, UDP, and TCP protocols. I then opened the Log Analytics Workspace and ran a query across the Azure Firewall logs for any traffic to the Private Endpoint from the VM and lo and behold, there is the ICMP and UDP traffic nmap generated.

Captured UDP and ICMP traffic

Yes folks that /32 route is protocol aware and will only apply to TCP traffic. UDP and ICMP traffic will not be affected. Software defined networking is grand isn’t it? 🙂

You may be asking why the hell I decided to test this particular piece. The reason I followed this breadcrumb was my customer had setup a UDR to route traffic from the VM to an NVA in the hub and attempted to send an ICMP Ping to the Private Endpoint. In reviewing their firewall logs they saw only the ICMP traffic. This finding was what drove me to test all three protocols and make the observation that the route only affects TCP traffic.

Microsoft’s public documentation mentions that Private Endpoints only support TCP at this time, but the documentation does not specify that this system route does not apply to UDP and ICMP traffic. This can result in confusion such as it did for this customer.

So how did we resolve this for my customer? Well in a very odd coincidence, a wonderful person over at Microsoft recently published some patterns on how to approach this problem. You can (and should) read the documentation for the full details, but I’ll cover some of the highlights.

There are four patterns that are offered up. Scenario 3 is not applicable for any enterprise customer given that those customers will be using a hub and spoke pattern. Scenario 1 may work but in my opinion is going to architect you into a corner over the long term so I would avoid it if it were me. That leaves us with Scenario 2 and Scenario 4.

Scenario 2 is one I want to touch on first. Now if you have a significant background in networking, this scenario will leave you scratching your head a bit.

Microsoft Documentation Scenario 2

Notice how a UDR is applied to the subnet with the VM which will route traffic to Azure Firewall however, there is no corresponding UDR applied to the Private Endpoint. Now this makes sense since the Private Endpoint would ignore the UDR anyway since they don’t support UDRs at this time. Now you old networking geeks probably see the problem here. If the packet from the VM has to travel from A (the VM) to B (stateful firewall) to C (the Private Endpoint) the stateful firewall will make a note of that connection in its cache and be expecting packets coming back from the Private Endpoint representing the return traffic. The problem here is the Private Endpoint doesn’t know that it needs to take the C (Private Endpoint) to B (stateful firewall) to A (VM) because it isn’t aware of that route and you’d have an asymmetric routing situation.

If you’re like me, you’d assume you’d need to SNAT in this scenario. Oddly enough, due the magic of software defined routing, you do not. This struck me as very odd because in scenario 3 where everything is in the same Virtual Network you do need to SNAT. I’m not sure why this is, but sometimes accepting magic is part of living in a software defined world.

Finally, we come to scenario 4. This is a common scenario for most customers because who doesn’t want to access Azure PaaS services over an ExpressRoute connection vs an Internet connection? For this scenario, you again need to SNAT. So honestly, I’d just SNAT for both scenario 2 and 4 to make maintain consistency. I have successfully tested scenario 2 with SNAT so it does indeed work as you expect it would.

Well folks I hope you found this information helpful. While much of it is mentioned in public documentation, it lacks the depth that those of us working in complex environments need and those of us who like to geek out a bit want.

See you next post!

Azure Files and AD DS – Part 2

Azure Files and AD DS – Part 2

Hi there and welcome to the second post in my series about Azure Files integration with AD DS. In the first post I gave an overview of the service, the value proposition, its current limitations, and described the lab I’ll be using for this post. For this post I’ll be walking through the setup, examining some packet captures and Fiddler captures, and touching on a few of the gotchas I ran into.

Before I jump into the technical gooey goodness, I’m going to cover some prerequisites.

Prerequisites

One obvious factoid is you’ll need a Windows AD domain up and running and the machine you connect to the share from will need to be joined to that domain. One disclaimer to keep in mind is you have a multiple Windows AD forest scenario, such as an account and resource forest, you’ll need to be aware of which domain you’re integrating the Azure File share with. If you integrate it with a resource forest but have your user accounts in the account forest, you’ll need to use name suffix routing. I’m not going to go into the details of name suffix routing, but if you’re curious you can read through this article. The short of it is the service principal name associated with the computer or service account used to represent the Azure Storage account the file share is created on uses the domain of files.core.windows.net. When performing the Kerberos authentication, the domain controller in the account forest wouldn’t know how to to direct the request to the resource forest because that domain will not be associated with your resource forest. For this lab I created a Windows AD domain with the namespace jogcloud.local

You will also need to ensure that you are synchronizing the users and groups from your Windows AD domain you want to be able to access the Azure File share to Azure AD. The tenant you synchronize to must be the same tenant the Azure subscription containing the Azure Storage account is associated with. Don’t worry about why right now, I’ll cover that later. For this lab I’ll be using my jogcloud.com tenant.

Logical Layout

To store those wonderful files you’ll need an Azure Storage account. The storage account should be created in the same region (or closest region if on-premises) to the clients that will access the file share. This will ensure optimal performance and avoid cross region costs if your clients are in Azure. You can use either a storage account with the standard GPv2 (General Purpose v2) SKU or Premium FileStorage SKU if you need better performance and scale. For this lab I’ll be using the GPv2 SKU.

Lastly, networking requirements. Like all Azure PaaS offerings, Azure Storage is by default available over the public Internet. Since no sane human being wants to send SMB traffic over the public Internet, you have the option of using a private endpoint. For this lab I’ll be going the private endpoint route.

So prerequisites are now set, let’s jump into the setup.

Integrating Azure Storage account with Windows AD

The first step in the process to get this integration working is to get the Azure Storage account you’ll be using setup with an identity in Windows AD. A kind human being over at Microsoft wrote a wonderful Azure PowerShell module that makes what I’m about to do a hundred times easier and is the recommended way to go about this. I’m not going to use it for this demonstration because I want to walk through each of the steps in the process to better your understanding of the magic within the module.

Before you run any commands you’ll need to ensure you have the PowerShell modules below installed. You can validate this by running Get-Module -ListAvailable to display the PowerShell modules installed on the machine.

Now we need to create the security principal that is going to represent the Azure Storage account in Windows AD. You have the option of either using a traditional service account (user account) or a computer account. As of August 28th 2020, there are some limitations you’ll run into if you use a service account over a computer account. My recommendation is use a computer account for now. I’ll cover the limitation later on in this entry. For the purposes of this blog post, I’ll be using a service account.

One important thing to note here is you need to treat this just like you would a traditional service account. By this I mean you will want to create the account with a non-expiring password and put in appropriate controls to perform a controlled rotation of the credential to avoid service disruption.

Here I’ve created a service account with the name azurestorage and have set it with a non-expiring password.

Service account for Azure Storage account

Next up I’m going create a SPN (service principal name) for the service account. The SPN is going to identify the Azure Storage account to Windows AD and instruct the user’s system which service it needs to obtain a Kerberos ticket for. The SPN is going to use the CIFS service class and include the FQDN of the Azure Files endpoint on your storage account. It will look like cifs/stjogfileshare.file.core.windows.net. You can register the SPN using the setspn -S <SPN> <ACCOUNT_NAME>. The -S switch will validate there the SPN is not already registered to another security principal in the domain.

Setting the SPN

So you have a service account and an SPN. Now you need to create a credential in Azure Storage and associate that credential with the service account. To create that credential you’ll need to hop over to PowerShell and connect to Azure. Once connected you’ll use the New-AzStorageAccountKey and Get-AzStorageAccountKey cmdlets to create and retrieve the storage account key used for the integration. It’s important to note that this key (named kerb1 or kerb2) is only used to setup this integration and can’t be used for any control or data plane operations against the storage account.

Configure the service account with this key as its password.

Creating the account key

The last piece in this step of the process is to enable the AD DS feature support for the storage account. To do this you’ll use the Set-AzStorageAccount cmdlet using the syntax below.

Set-AzStorageAccount syntax

All of the inputs between Name and ActiveDirectoryAzureStorageSid can be obtained by using the Get-ADDomain cmdlet as seen below.

Get-ADDomain Output

The ActiveDirectoryAzureStorageSid parameter can be obtain by using the Get-ADUser cmdlet as seen below.

Get-ADUser Output

Once you have the inputs, you’ll plug them in Set-AzStorageAccount cmdlet. If successful you’ll get a return similar to below.

Set-AzStorageAccount Output

If you’d like you can confirm the feature is enabled you can do that with the steps documented here. The AzFilesHybrid module I mentioned earlier also has some great debugging tools as outlined here.

Now that the integration is complete, I need to create a file share and configure authorization at the management plane. There are two separate layers of authorization occurring, one for access to the file share itself and the other for access to the files and folders within the file share. Access to the share itself is controlled by Azure RBAC and thus controlled by the Azure management plane. There are three roles built in roles provided that should service most use cases and these are:

  1. Storage File Data SMB Share Read which allows read access to the file share over SMB
  2. Storage File Data SMB Share Contributor which allows read, write, and delete access over the file share over SMB
  3. Storage File Data SMB Share Elevated Contributor which allows read, write, delete, and modify of Windows ACLs of the file share over SMB

You are free to design your own custom roles, but those three built in roles are pretty much spot on as to what you’d see in your typical Windows File share-level permissions.

The second layer of authorization is controlled by the Windows ACLs (access control lists) associated with the share, files, and folders. These are your classic Windows ACLs you know and love and will be enforced by the Windows OS.

Just like on a traditional Windows file share, the most restrictive of controls will apply. This means if you’ve only been granted the Storage File Data SMB Share Read role, it won’t matter if you have full permissions in the Windows ACLs, you will only be able to read and will not be able to write.

Let me demonstrate this.

Here I have assigned the Bob Gray user the Storage File Data SMB Share Reader role on the stjogfileshare storage account. Bob Gray is Domain administrator on the jogcloud.local Windows AD domain and is a local administrator on the member server.

Role Assignment on Storage Account

As seen below I’m able to successfully map the shared folder, but I’m unable to create folder on it because the management plane is restricting my access.

Unable to created a folder when holding Read RBAC role

Running the klist command on the machine shows I successfully obtained a Kerberos ticket for file share.

klist Output

The packet capture I ran when I mapped the share shows in the SMB conversation that the client and server are using the negotiate protocol (which includes the Kerberos protocol).

Session Negotiation

After the Kerberos ticket is obtained from the domain controller the client sets up the session with the storage account.

Session Setup

From this point forward, the encryption capabilities of SMB 3 are used to encrypt the session between the client and Azure storage account.

Remember the limitation around using a service account as the security principal representing Azure Files I mentioned earlier? Well that limitation is around the encryption algorithm used to encrypt the Kerberos tickets passed to Azure Files. Out of the gates, the Azure Files with AD DS feature only supported the RC4-HMAC encryption algorithm. This is a deprecated algorithm according to the IETF (Internet Engineering Task Force) and should not be used. Many organizations in regulated environments have disabled this encryption algorithm in Active Directory due to its deprecation.

As of August 28th, 2020, the integration now supports AES256 encryption for Kerberos. However, this is only supported if you’re using a Computer account. This means that if you are already using the service with a service account and you want to move to AES256, you’ll need to migrate to using a Computer account. I expect support for a service account will be added sometime in the future, so check the official documentation for updates.

If you try to use a service account and attempt to enforce the use of AES256, your connection will fail. If you do a packet capture you’ll see the Azure Storage account throw a KRB Error: KRB5RB_AP_ERR_MODIFIED indicating the Azure Storage account was unable to validate the ticket that was passed because it doesn’t support the encryption algorithm used to secure it.

I went through and created a group in Windows AD named engineering and added Bob Gray to it. I then removed the Storage File Data SMB Share Read role assignment for Bob Gray and created a role assignment for the engineering group for the Storage File Data SMB Share Contributor role. I’m now able to create files and folders on the share as seen below.

Creating folder on share

Bringing up the permissions on the folders you’ll observe that that are a few default permissions which come out of the box. You can modify these default permission if you’d like (for example by removing Authenticated Users Read/Modify which is overly permissive). You do this by mounting the share as a super user using the standard storage keys. The process is outlined here.

Default permissions

That is pretty much all there is to it to the technical configuration.

So when you use this service what are some of the best practices that I would recommend?

  1. Use a computer account if you require an encryption algorithm better than RC-HMAC. At this time, computer accounts are the only type of security principal which supports AES256 encryption.
  2. Ensure you rotate the password at whatever interval aligns with your organizational security policy and any laws and regulations you may be subject to.
  3. When you create your Azure RBAC role assignments, use synchronized groups vs synchronized users. You do this for the same reason you would on-premises, granting access per user is not scalable.
  4. Dedicate the storage account you use for the file share to only file shares. Storage accounts have fixed limits that are shared across blobs, queues, tables, and files. You don’t want to get into a situation where you have to share those limits.
  5. Do your research to determine if Premium FileStorage makes more sense than GPv2. It’s more costly but provides better performance and scale.
  6. Try to deploy one file share per storage account if possible to ensure you get the maximum IOPS available for that file share. You can certainly put multiple file shares in the same storage account, but they will share the total IOPS available for the storage account.
  7. Ensure you are replicating files to another storage account. Unlike blobs, you can’t read from the second region if you’re using a RA-GRS storage account. If you’re using the Premium Files SKU, the storage account will only support LRS and ZRS which makes this replication to a storage account in another region so important. You could use AzCopy, PowerShell, or Azure Data Factory.

That’s it folks! Hope this post helped you understand feature that much better.

Thanks and see you next post!

Azure Files and AD DS – Part 1

Azure Files and AD DS – Part 1

Welcome back folks.

I recently had a few customers reach out to me with questions around Azure Files integration with Windows Active Directory Domain Services (AD DS). Since I had never used it, I decided to build a small lab and test the functionality and better understand the service. In this series I’ll be walking through what the functionality provides, how I observed it working, and how to set it up.

In any enterprise you will have a number of Windows file shares hosting critical corporate data. If you’ve ever maintained or supported those file shares you’re quite familiar with the absolute sh*t show that occurs across an organization with the coveted H drive is no longer accessible. Maintaining a large cluster of Windows Servers backing corporate file shares can be significantly complex to support. You have your upgrades, patches, needs to scale, failures with DFS-R (distributed file system replication) or FRS-R (file replication service replication) for some of you more unfortunate souls. Wouldn’t it be wonderful if all that infrastructure could be abstracted and managed by someone else besides you? That is the major value proposition of Azure Files.

Azure Files is a PaaS (platform-as-a-service) offering provided by Microsoft Azure that is built on top of Azure Storage. It provides fully managed file shares over a protocol you know and love, SMB (Server Message Block). You simply create an Azure File share within an Azure Storage account and connect to the file share using SMB from a Windows, Linux, or MacOS machine.

Magic right? Well what about authentication and authorization? How does Microsoft validate that you are who you say you are and that you’re authorized to connect to the file share? That my friends will be what we cover from this point on.

Azure File shares supports methods of authentication:

  1. Storage account access keys
  2. Azure AD Domain Services
  3. AD DS (Active Directory Domain Services)

Of the three methods, I’m going cover authentication using AD DS (which I’ll refer to as Windows AD).

Support for Windows AD with Azure Files graduated to general availability last month. As much as we’d like it to not be true, Windows AD and traditional SMB file shares will be with us many years to come. This is especially true for enterprises taking the hybrid approach to cloud, which is a large majority of the customer base I work with. The Windows AD integration allows organizations to leverage their existing Windows AD identities for authentication and protect the files using Windows ACLs (access control lists) they’ve grown to love. This provides a number of benefits:

  • Single sign-on experience for users (via Kerberos)
  • Existing Windows ACLs can be preserved if moving files to a Azure File share integrated with Windows AD
  • Easier to supplement existing on-premises file servers without affecting the user experience
  • Better support for lift and shift workloads which may have dependencies on SMB
  • No infrastructure to manage
  • Support for Azure File Sync which allows you to store shares in Azure Files and create cache on Windows Servers

There are a few key dependencies and limitations I want to call out. Keep in mind you’ll want to visit the official documentation as these will change over time.

  • The Windows AD identities you want to use to access the file shares must be synchronized to Azure AD (we’ll cover the why later)
  • The storage account hosting the Azure File share must be in the same tenant you’re syncing the identities to
  • Linux VMs are not supported at this time
  • Services using the computer account will not be able to access an Azure File share so plan on using a traditional service account (aka User account) instead
  • Clients accessing the file share must be Window 7/Server 2008 R2 ore above
Lab Environment

So I’ve given you the marketing pitch, let’s take a look at the lab environment I’ll be using for this walkthrough.

For my lab I’ve provisioned three VMs (virtual machines) in a VNet (virtual network). I have a domain controller which provides the jogcloud.local Windows AD forest, an Azure AD Connect server which is synchronizing users to the jogcloud Azure AD tenant, and a member server which I’ll use to access the file share.

In addition to the virtual machines, I also have an Azure Storage account where I’ll create the shares. I’ve also configured a PrivateLink Endpoint which will allow me to access the file share without having to traverse the Internet. Lastly, I have an Azure Private DNS zone hosting the necessary DNS namespace needed to handle resolution to my Private Endpoint. I won’t be covering the inner workings of Azure Private DNS and Private Endpoints, but you can read my series on how those two features work together here.

In my next post I’ll dive in how to setup the integration, walk through some Wireshark and Fiddler captures, and walk through some of the challenges I ran into when running through this lab for this series.

See you next post!

Azure Key Vault, Certificates, and Python, oh my!

Azure Key Vault, Certificates, and Python, oh my!

Welcome back folks! I recently had a few customers ask me about using certificates with Azure Key Vault and switching from using a client secret to a client certificate for their Azure AD (Active Directory) service principals. The questions put me on a path of diving deeper around the topics which results in some great learning and opportunity to create some Python code samples.

Azure Key Vault is Microsoft’s solution for secure secret, key, and credential management. If you’re coming from the AWS (Amazon Web Services) realm, you can think of it as AWS KMS (Key Management Services) with a little bit of AWS Secrets Manager and AWS Certificate Manager thrown in there. The use cases for secrets and keys are fairly well known and straightforward, so I’m going instead focus time on the certificates use case.

In a world where passwordless is the newest buzzword, there is an increasing usage of secrets (or passwords) in the non-human world. These secrets are often used to programmatically interact with APIs. In the Microsoft world you have your service principals and client secrets, in the AWS world you have your IAM Users with secret access keys, and many more third-parties out there require similar patterns that require the use of an access key. Vendors like Microsoft and AWS have worked to mitigate this growing problem in the scope of their APIs by introducing features such as Azure Managed Identities and AWS IAM Roles which use short lived dynamic secrets. However, both of these solutions work only if your workload is running within the relevant public cloud and the service it’s running within supports the feature. What about third-party APIs, multi-cloud workloads, or on-premises workloads? In those instances you’re many of times forced to fall back to the secret keys.

There is a better option to secret keys, and that is client certificates. While a secret falls into the “something you know” category, client certificates fall into the “something you have” category. They provide an higher assurance of identity (assuming you exercise good key management practices) and can have more flexibility in their secure storage and usage. Azure Service Principals support certificate-based authentication in addition to client secrets and Azure Key Vault supports the secure storage certificates. Used in combination, it can yield some pretty cool patterns.

Before I get into those patterns, I want to cover some of the basics in how Azure Key Vault stores certificates. There are some nuances to how it’s designed that is incredibly useful to understand. I’m not going to provide a deep dive on the inner workings of Key Vault, the public documentation does a decent enough job of that, but I am going to cover some of the basics which will help get you up and running.

Certificates can be both imported into and generated within Azure Key Vault. These certificates generated can be self-signed, generated from a selection of public CAs (certificate authorities) it is integrated with, or can be used to generate a CSR (certificate signing request) you can full-fill with your own CA. These processes are well detailed in the documentation, so I won’t be touching further on them.

Once you’ve imported or generated a certificate and private key into Key Vault, we get into the interesting stuff. The components of the certificate and private key are exposed in different ways through different interfaces as seen below.

Key Vault and certificates

Metadata about the certificate and the certificate itself are accessible via the certificates interface. This information includes the certificate itself provided in DER (distinguished encoded rules) format, properties of the certificate such as the expiration date, and metadata about the private key. You’ll use this interface to get a copy of the certificate (minus private key) or pull specific properties of the certificate such as the thumbprint.

Operations using the private key such as sign, verify, encrypt, and decrypt, are made available through the key interface. Say you want to sign a JWT (JSON Web Token) to authenticate to an API, you would use this interface.

Lastly, the private key is available through the secret interface. This is where you could retrieve the private key in PEM (privacy enhanced mail) or PKCS#12 (public key cryptography standards) format if you’ve set the private key to be exportable. Maybe you’re using a library like MSAL (Microsoft Authentication Library) which requires the private key as an input when obtaining an OAuth access token using a confidential client.

Now that you understand those basics, let’s look at some patterns that you could leverage.

In the first pattern consider that you have a CI/CD (continuous integration / continuous delivery) running on-premises that you wish to use to provision resources in Azure. You have a strict requirement from your security team that the infrastructure remain on-premises. In this scenario you could provision a service principal that is configured for certificate authentication and use the MSAL libraries to authenticate to Azure AD to obtain the access tokens needed to access the ARM API (Azure Resource Manager). Here is Python sample code demonstrating this pattern.

On-premises certificate authentication with MSAL

In the next pattern let’s consider you have a workload running in the Azure AD tenant you dedicate to internal enterprise workloads. You have a separate Azure AD tenant used for customer workloads. Within an Azure subscription associated with the customer tenant, there is an instance of Azure Event Hub you need to access from a workload running in the enterprise tenant. For this scenario you could use a pattern where the workload running in the enterprise tenant uses an Azure Managed Identity to retrieve a client certificate and private key from Key Vault to use with the MSAL library to obtain an access token for a service principal in the customer tenant which it will use to access the Event Hub.

Here is some sample Python code that could be used to demonstrate this pattern.

For the last pattern, let’s consider you have the same use case as above, but you are using the Premium SKU of Azure Key Vault because you have a regulatory requirement that the private key never leaves the HSM (hardware security module) and all cryptographic operations are performed on the HSM. This takes MSAL out of the picture because MSAL requires the private key be provided as a variable when using a client certificate for authentication of the OAuth client. In this scenario you can use the key interface of Key Vault to sign the JWT used to obtain the access token from Azure AD. This same pattern could be leveraged for other third-party APIs that support certificate-based authentication.

Here is a Python code sample of this pattern.

Well folks I’m going to keep it short and sweet. Hopefully this brief blog post has helped to show you the value of Key Vault and provide some options to you for moving away from secret-based credentials for your non-human access to APIs. Additionally, I really hope you get some value out of the Python code samples. I know there is a fairly significant gap in Python sample code for these types of operations, so hopefully this begins filling it.

Thanks!