Lab Guide - AKS Workload Identity
Microservices brings a lot of opportunities to the table compared to old monoliths. Unfortunately, there are things that was/is easier with those big blocks of code that becomes more complex when breaking it into smaller pieces. We're not going to tackle all of those today, but rather we'll look closer at a very specific piece of the identity puzzle.
When you're using Azure Kubernetes Service (AKS) there's different parts that involve identities. You need permissions to manage the cluster. You need to login to the apps running in the cluster. You need cross-app or cross-pod exchange of data. And you need the cluster or things running inside it to interact with the greater Azure infrastructure.
What if you deploy an app that needs to query the MS Graph for various attributes. Or what if you need to create a database in an Azure SQL instance? Yes, you can create client ids and secrets and write the necessary code. Which is bound to be inadvertently checked into source code or some other mishap. So, you want this shiny new passwordless concept that you've heard about. For a while Microsoft has provided a component called "pod identity" for use with AKS. Unfortunately there were some lower level problems with the implementation of this that halted this in a preview state. It's even been deprecated now, and the new approach is called "workload identity":
https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview
The overview covers what it is and the high level details of how it works; the short version is that we "connect" the service accounts within Kubernetes with Azure AD identities. This bridge is based on Kubernetes being an OpenID Connect issuer and Azure AD trusting the tokens through federation.
The official docs have you covered if you jump a little back and forth so the intent of this post is to create an end-to-end lab.
Now, if you've been following along from the sidelines for a while you might be thinking "didn't they launch something early last year?". Well, sort of. There's "workload identity federation":
https://learn.microsoft.com/en-us/azure/active-directory/develop/workload-identity-federation
This is the overarching federation ability that enables things like GitHub Actions to deploy ARM/Bicep/Terraform code without using passwords and secrets, and workload identity for AKS builds on top of this.
There was also a version for AKS that involved using a Service Principal on top of this:
This newest iteration gets rid of the Service Principal and greatly simplifies the enablement process on the cluster. (The whole application object vs service principal explanation/discussion is out of scope for this post.)
Alrighty, let's get cracking with this and step through things incrementally.
Btw: don't spend time copying from this post and pasting in - pull down the necessary files from here instead: https://github.com/ahelland/Bicep-Landing-Zones/tree/main/aks-workload-identity
I made a Polyglot Notebook so as long as you're using Visual Studio Code and have the extension (https://marketplace.visualstudio.com/items?itemName=ms-dotnettools.dotnet-interactive-vscode) installed you can just press play on the individual code snippets to run them. (Pre-req being you have Azure cli, PowerShell, etc. installed.)
The scripts also create a virtual network and a container registry. Not related to workload identity as such, but necessary for a complete setup. For completion there's also a sample app that plugs into the MS Graph with a workload identity.
Create an AKS cluster and a user managed identity
You will of course need a cluster to test this out. By default workload identities are not enabled so you will need to add the necessary parameters for that. (If you're using Azure cli that would be "--enable-oidc-issuer --enable-workload-identity".)
When using Bicep you need a few lines of definitions:
The workload identities are created as user managed identities:
This is the part where you need to do your initial planning around the usage. The managed identity created here is the object that grants permissions into other areas of Azure, and you might not want just one super-privileged identity that can be accessed by everyone. For this lab I have configured two - one called "aks-infra-identity" to be used by the cluster components, and one called "aks-app-identity" to be used by the workload running on the cluster.
So, execute the following lines to create a cluster, registry and vnet:
Installing cluster components
A naked cluster doesn't provide much value by itself, and doesn't really demo the concepts here either, so we need to perform some extra steps to make it all work.
First thing is to establish a federated credential which is basically linking service accounts in Kubernetes to the managed identity (for infra):
Pay attention to the "subject" and the format of it. It references a service account and its corresponding namespace. Service accounts are namespaced, so basically each namespace requires a federation like this. If you fail to do this the logs of the pod attempting to use the account (which it cannot find) you will get errors referring to the subject name.
We need ingress to get the traffic into the cluster. (Well, technically a loadbalancer also works for the purpose, but for this config we'll use ingress resources.) We'll use nginx for this purpose:
You don't have to automate creation of DNS records, but it sure makes things simpler when deploying new frontends. You can host DNS where you like and create the records through other means, but for this lab we include configuration of ExternalDNS using Azure DNS for this purpose. (The zone needs to be pre-created.)
You will notice that we need to assign permissions for the DNS zone to get it working. However, ExternalDNS does not support workload identity yet. Instead it is able to leverage the system assigned managed identity of the node pool so we configure permissions accordingly.
DNS names instead of browsing to IP is user-friendly, but modern browsers get cranky when you're not using TLS/SSL so we'd better install something for that purpose as well:
CertManager also requires access to the DNS zone to create and delete records for using the ACME protocol. It does not piggy back on ExternalDNS and works independently. CertManager already supports workload identities, so we assign the permissions to the aks-infra-identity managed identity.
This highlights what I already stated with regards to having multiple managed identities. Do you want the apps to have direct access to the DNS zone or only indirectly?
In addition to assigning the permissions this requires us to create a new federated credential. CertManager is installed in its own namespace so it needs it own service account that needs to be added to the managed identity. (Currently you can add 20 service accounts to each managed identity.)
It has of course been possible to achieve these things before either with the system managed identity (like for ExternalDNS) or injecting the clientId and clientSecret of a service principal for CertManager. As you saw there are more things required than just turning a switch so it's not like it's an automagic thing, and even if details improve upon the road to GA some assembly will probably be required. The bigger win currently is not involving secrets in any way. Which is a good thing of course.
Deploying a sample workload
The previous steps take care of the core infra. We like that, but that by itself is not the whole picture. You have workloads running in your cluster as well and that's the next part we need in our lab.
A generic web app will usually have identity in two ends:
- User signing in interactively on a front-end.
- The back-end accessing databases, generating events, and so forth.
The workload identity is intended for the back-end so that's what we will demo. Yes, there are on-behalf flows where the token from the user is passed along and added to API calls on the backend. Those are out of scope for now. We have a couple of lines of script to build a sample app and deploy to our cluster:
The C# code is pushed to Azure Container Registry and bundled into Docker images. You can modify and play with it, but it should work also out of the box.
We create a separate service account for the namespace of the app and a new federated credential, but we attach it to the aks-app-identity to make sure it is not able to interfere with things like creating DNS records. (Notice how we can create the federation before the service account has been created.)
If you deal with registering applications in the Azure Portal for use in web apps you will have noticed it's fairly easy to assign permissions to the MS Graph by browsing the API list. Without going into details of how Azure AD objects are organized this works because behind the scenes there's both an application object and a service principal where the purpose of the latter is permissions. (For a single-tenant app this seems redundant, but multi-tenant SaaS apps needs to have separate objects for the application itself, and the permissions it has been granted in different tenants.) The user-managed identity doesn't have a service principal that you can attach permissions to. Which means there are some extra steps when we want to assign permissions to the identity used for apps.
In our sample app we query the Graph for properties of the app; more specifically the tenant name. And this requires the User.Read.All application permission to the Graph. For a walkthrough of the details check this blog:
https://gotoguy.blog/2022/03/15/add-graph-application-permissions-to-managed-identity-using-graph-explorer/
But it's not like applications all by itself understands that you're using this new fancy Kubernetes trickery is it? No, it requires a little nudge along the way. The details are language specific, but the general concept is that workload identity injects a JWT into the file system of the container and the app picks up on this and attaches it to authentication requests.
For C# you override the token acquisition process with a couple of overrides:
To the rest of the app this is no different than tokens acquired by other means so once you've gotten to this point it should be smooth sailing. Because there's never any other identity-related challenges ;)
For this demo I skipped "proper" DevOps by creating a GitHub Action or Azure DevOps Pipeline, but it is of course possible to adapt as a fully automated process.
Published on:
Learn more