Loading...

Enable and Operate Open Service Mesh in AKS hybrid cluster

Enable and Operate Open Service Mesh in AKS hybrid cluster

A number of organizations adopting microservices architectures have looked to service meshes as one of the tools that help these organizations in enforcing enterprise-wide policies and rules for traffic, security, reliability, and monitoring. These are generally categorized into Security, Reliability and Observability. While a networking solution other than service mesh can provide some of these features, it won’t do it without disrupting the application code or existing infrastructure. Service Mesh provides an independent infrastructure layer that can be managed by a separate team and is least intrusive. It deploys a sidecar container to pods by design, which makes it very powerful for monitoring your services within a kubernetes cluster. 

 

Open Service Mesh (OSM) is a lightweight, extensible service mesh that is utilized to secure communication of services running in a Kubernetes environment. To achieve interoperability between service meshes, a common interface standard SMI (Service Mesh Interface) has been defined by CNCF. OSM is Microsoft's implementation of the Service Mesh Interface in an actual service mesh. It is easy to install, maintain and operate. It is also effortless to troubleshoot and less complicated to configure with SMI. 

 

The Open Service Mesh extension is a managed service mesh for Arc-enabled Kubernetes clusters that is lightweight and extensible. It can be configured with Service Mesh Interface APIs, works by injecting envoy proxy as a sidecar to each application instance, and brings a new Azure Portal experience for onboarding. 

Some of the use cases you can solve with OSM are:   

  • Service to Service communication over mutual TLS authentication   
  • Access control for service-to-service communication   
  • Traffic management for canary deployments   
  • Observability for services   
  • Traffic management capabilities like auto retries, rate limiting, circuit breaking   
  • Multi-cluster communication over mTLS.   

With the OSM extension being enabled through Azure Arc, OSM can be applied to your on-premises AKS hybrid environment to manage communication and secure your cloud-native workload.  

In this article, the OSM functionalities were applied to the communication between the bookbuyer and bookstore sample applications. First, traffic access between the bookbuyer and bookstore services was disabled, but only got enabled by applying an SMI traffic policy. Additionally, the traffic-splitting capability was also implemented by creating a second bookstore service and then weighted traffic from the bookbuyer to bookstore-v1 and bookstore-v2 respectively. 

 

Setting up OSM 

The latest OSM version (v1.2.0) used for this article requires a kubernetes cluster running v1.22.9 or higher and a Kubernetes command-line tool. It also requires a workstation that is capable of executing bash commands/scripts and has the OSM code repo available locally.  

For this work, the setup was done in an Arc-enabled Aks cluster with WSL-2 (windows subsystem for linux) enabled. To setup your own Aks hybrid cluster, see Use PowerShell to set up Kubernetes on Azure Stack HCI and Windows Server clusters - AKS-HCI | Microsoft Learn. This should have Az CLI running, but you can check using az version. 

To get things going, bring up your cluster. 

 

PS C:\Users\ekele> Get-AksHciCluster 

Status                : {ProvisioningState, Details} 

ProvisioningState     : Deployed 

KubernetesVersion     : v1.23.8 

PackageVersion        : v1.23.8 

NodePools             : linuxpool 

WindowsNodeCount      : 0 

LinuxNodeCount        : 3 

ControlPlaneNodeCount : 3 

ControlPlaneVmSize    : Standard_A2_v2 

AutoScalerEnabled     : False 

AutoScalerProfile     : 

LoadBalancer          : {VMSize, Count, Sku} 

Name                  : <ClusterName> 

 

The next step is to enable OSM kubernetes extension on the AKS hybrid cluster after connecting the cluster to Azure Arc. 

Note: In order to run some bash commands for validation purposes, the rest of the cmdlets used in this article were run in Git bash terminal on Visual Studio Code with Az cli installed. Also, ensure to download the OSM cli to be able to run osm commands. To download the latest OSM cli, see Releases · openservicemesh/osm (github.com) 

 

ea@SA18n30-3 MINGW64 ~ (master) 

$ az k8s-extension create --cluster-name <ClusterName> --resource-group <ResourceGroup> --cluster-type connectedClusters --extension-type Microsoft.openservicemesh --scope cluster --name osm 

 { 

 "aksAssignedIdentity": null, 

  "autoUpgradeMinorVersion": true, 

  "configurationProtectedSettings": {}, 

  "configurationSettings": {}, 

  "customLocationSettings": null, 

  "errorInfo": null, 

  "extensionType": "microsoft.openservicemesh", 

  "id": "/subscriptions/5807bea4-cd17-4b81-b4d8-d4e94caab82d/resourceGroups/<ResourceGroup>/providers/Microsoft.Kubernetes/connectedClusters/<ClusterName>/providers/Microsoft.KubernetesConfiguration/extensions/osm", 

  "identity": { 

    "principalId": "0c8ed1d6-358f-4f6a-8342-61d10e1a9571", 

    "tenantId": null, 

    "type": "SystemAssigned" 

  }, 

  "name": "osm", 

  "packageUri": null, 

  "provisioningState": "Succeeded", 

  "releaseTrain": "Stable", 

  "resourceGroup": "<ResourceGroup>", 

  "scope": { 

    "cluster": { 

      "releaseNamespace": "arc-osm-system" 

    }, 

  "type": "Microsoft.KubernetesConfiguration/extensions", 

  "version": "1.1.1-1" 

} 

 

Verify that the OSM k8s extension was installed correctly by running  

ea@SA18n30-3 MINGW64 ~ (master) 

$ az k8s-extension show --cluster-type connectedClusters --cluster-name <ClusterName> 

--resource-group <ResourceGroup> --name osm 

 

Get the credentials of your cluster to be sure you can run kubectl using 

ea@SA18n30-3 MINGW64 ~ (master) 

$ kubectl config current-context 

<ClusterName>-admin@<ClusterName> 

 

You can further verify that the OSM components are running on your cluster by checking the deployment, pods, and services in arc-osm-system namespace. 

ea@SA18n30-3 MINGW64 ~ (master) 

$ kubectl get deployment -n arc-osm-system 

NAME                READY   UP-TO-DATE   AVAILABLE   AGE 

osm-bootstrap       1/1     1            1           86m 

osm-controller      1/1     1            1           86m 

osm-injector        1/1     1            1           86m 

osm-metrics-agent   1/1     1            1           86m 

 

ea@SA18n30-3 MINGW64 ~ (master) 

$ kubectl get pods -n arc-osm-system 

NAME                                 READY   STATUS    RESTARTS       AGE 

osm-bootstrap-58b8986b87-wfp6g       1/1     Running   0              88m 

osm-controller-768df46759-n29ps      1/1     Running   0              88m 

osm-injector-6ddd574779-wl52s        1/1     Running   0              88m 

osm-metrics-agent-78c846d57b-v9vtd  4/4     Running   1 (87m ago)   88m 

 

ea@SA18n30-3 MINGW64 ~ (master) 

$ kubectl get services -n arc-osm-system 

NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S) AGE 

osm-bootstrap   ClusterIP   10.99.80.35     <none>        9443/TCP,9095/TCP 88m 

osm-controller   ClusterIP   10.111.193.40   <none>        15128/TCP,9092/TCP,9091/TCP   88m 

osm-injector     ClusterIP   10.100.99.169   <none>        9090/TCP 88m 

osm-validator    ClusterIP   10.108.48.217   <none>        9093/TCP 88m 

 

OSM controller configuration 

OSM deploys a MeshConfig resource osm-mesh-config as a part of its control plane in arc-osm-system namespace. The MeshConfig provides the mesh owner or operator the ability to update some of the mesh configurations based on need. The default values of the MeshConfig can be viewed using the following command. 

$ kubectl describe meshconfig osm-mesh-config -n arc-osm-system 

Name:         osm-mesh-config 

Namespace:    arc-osm-system 

Labels:       <none> 

Annotations:  <none> 

API Version:  config.openservicemesh.io/v1alpha2 

Kind:         MeshConfig 

Metadata: 

  Creation Timestamp:  2022-10-17T22:56:58Z 

  Generation:          1 

  Managed Fields: 

    API Version:  config.openservicemesh.io/v1alpha2 

    Fields Type:  FieldsV1 

    fieldsV1: 

      f:metadata: 

        f:annotations: 

{ … 

Traffic: 

    Enable Egress:                          false 

    Enable Permissive Traffic Policy Mode:  true 

… } 

It can be observed that the permissive traffic policy mode of the output is set at true. With permissive traffic policy mode enabled in your OSM mesh: 

  • The SMI traffic policy enforcement is bypassed. 
  • OSM automatically discovers services that are a part of the service mesh. 
  • OSM programs traffic policy rules on each Envoy proxy sidecar to be able to communicate with these services. 

It is important to note that values in the MeshConfig osm-mesh-config are persisted across upgrades. Changes to osm-mesh-config can be made using the kubectl patch command. In a subsequent step for this sample deployment, the permissive traffic policy mode was changed to false through this command. 

 

Deploy Bookstore Applications 

After validating the checks, the sample applications manifest (Bookbuyer, Bookstore, Bookwarehouse) is deployed that will generate the services on which OSM will be applied to. 

Kubectl create -f https://raw.githubusercontent.com/openservicemesh/osm/release-v0.8/docs/example/manifests/apps/bookbuyer.yaml 

Kubectl create -f https://raw.githubusercontent.com/openservicemesh/osm/release-v0.8/docs/example/manifests/apps/bookstore.yaml 

Kubectl create -f https://raw.githubusercontent.com/openservicemesh/osm/release-v0.8/docs/example/manifests/apps/bookwarehouse.yaml 

 

Validate Application by port-forwarding the Kubernetes service 

The components of the bookstore namespace can be checked to see the pods and services running using ‘kubectl get all -n bookstore’. Afterwards, the logs of the bookbuyer pod could be traced to ensure it is communicating with the bookstore service. 

To query that bookbuyer is reaching bookstore service, run the following commands to trail the bookbuyer logs. 

POD="$(kubectl get pods -n bookbuyer  --show-labels --selector app=bookbuyer --no-headers | grep -v 'Terminating' | awk '{print $1}' | head -n1)" 

Kubectl logs “${POD}” -n bookbuyer -c bookbuyer --tail=100 -f   

{ … Fetching http://bookstore.bookstore:14001/books-bought 

Request Headers: map[Client-App:[bookbuyer] User-Agent:[Go-http-client/1.1]] 

Identity: bookstore-v1 

Booksbought: 5258 

Server: n/a 

Date: Tue, 18 Oct 2022 04:42:48 GMT 

Status: 200 OK 

MAESTRO! THIS TEST SUCCEEDED! 

 

Fetching http://bookstore.bookstore:14001/buy-a-book/new 

Request Headers: map[] 

Identity: bookstore-v1 

Booksbought: 5259 

Server: n/a 

Date: Tue, 18 Oct 2022 04:42:49 GMT 

Status: 200 OK 

MAESTRO! THIS TEST SUCCEEDED! 

 

… } 

The status should show that everything is okay and that it is able to fetch http bookstore-v1 data. 

 

Onboard Bookstore Apps namespaces for OSM 

In order to start using OSM capabilities, the application namespaces need to be onboarded to the service mesh so that OSM can manage these namespaces. Run the command 

$ osm namespace add bookstore bookbuyer bookwarehouse 

Namespace [bookstore] successfully added to mesh [osm] 

Namespace [bookbuyer] successfully added to mesh [osm] 

Namespace [bookwarehouse] successfully added to mesh [osm] 

 

Enable proxy sidecar injection for Bookstore Apps 

The sidecar containers can be enabled using the ‘kubectl rollout restart’ command. You can check the pod details before and after to know that the sidecar proxies were correctly deployed. 

$ kubectl get pods -n bookbuyer 

NAME                         READY   STATUS    RESTARTS   AGE 

bookbuyer-56d6bcc7f8-gvzn8   1/1     Running   0          86m 

 

ea@SA18n30-3 MINGW64 ~ (master) 

$ kubectl rollout restart deployment bookbuyer -n bookbuyer 

deployment.apps/bookbuyer restarted 

 

ea@SA18n30-3 MINGW64 ~ (master) 

$ kubectl rollout restart deployment bookstore -n bookstore 

deployment.apps/bookstore restarted 

 

ea@SA18n30-3 MINGW64 ~ (master) 

$ kubectl rollout restart deployment bookwarehouse -n bookwarehouse 

deployment.apps/bookwarehouse restarted 

 

By checking the pod details of the bookbuyer after restart, the ‘Ready’ state shows that there are 2 out of 2 pods running. 

$ kubectl get pods -n bookbuyer 

NAME                         READY   STATUS    RESTARTS   AGE 

bookbuyer-6ccd656dbf-7c28x   2/2     Running   0                  115s 

To get further information, the ‘kubectl describe pod’ command will show an envoy running alongside the bookbuyer container.  

POD="$(kubectl get pods -n bookbuyer  --show-labels --selector app=bookbuyer --no-headers | grep -v 'Terminating' | awk '{print $1}' | head -n1)" 

Kubectl describe pod $POD -n bookbuyer 

   

Normal  Started    4m29s  kubelet            Started container bookbuyer 

Normal  Pulling    4m29s  kubelet            Pulling image "mcr.microsoft.com/oss/envoyproxy/envoy:v1.19.3" 

Normal  Pulled     4m26s  kubelet            Successfully pulled image "mcr.microsoft.com/oss/envoyproxy/envoy:v1.19.3" in 3.413382681s 

Normal  Created    4m26s  kubelet            Created container envoy 

Normal  Started    4m26s  kubelet            Started container envoy 

 

Verify Bookstore App functioning with sidecar injection 

Kubectl logs "${POD}" -n bookbuyer -c bookbuyer --tail=100 -f  

 

Fetching http://bookstore.bookstore:14001/books-bought 

Request Headers: map[Client-App:[bookbuyer] User-Agent:[Go-http-client/1.1]] 

Identity: bookstore-v1 

Booksbought: 415 

Server: envoy 

Date: Tue, 18 Oct 2022 04:51:14 GMT 

Status: 200 OK 

MAESTRO! THIS TEST SUCCEEDED! 

 

Fetching http://bookstore.bookstore:14001/buy-a-book/new 

Request Headers: map[] 

Identity: bookstore-v1 

Booksbought: 416 

Server: envoy 

Date: Tue, 18 Oct 2022 04:51:15 GMT 

Status: 200 OK 

MAESTRO! THIS TEST SUCCEEDED! 

 

Disable OSM Permissive Mode 

If the permissive traffic policy mode is disabled, communication between the bookbuyer and bookstore will cease. This can be done using the following command 

$ kubectl patch meshconfig osm-mesh-config -n arc-osm-system -p '{"spec":{"traffic":{"enablePermissiveTrafficPolicyMode":false}}}' --type=merge 

meshconfig.config.openservicemesh.io/osm-mesh-config patched 

 

Verify that the bookbuyer can no longer reach the bookstore  

Kubectl logs "${POD}" -n bookbuyer -c bookbuyer --tail=100 -f 

{ … 

Fetching http://bookstore.bookstore:14001/books-bought 

Request Headers: map[Client-App:[bookbuyer] User-Agent:[Go-http-client/1.1]] 

Error fetching http://bookstore.bookstore:14001/books-bought: Get "http://bookstore.bookstore:14001/books-bought": dial tcp 10.99.49.182:14001: connect: connection 

refused 

ERROR: response code for "http://bookstore.bookstore:14001/books-bought" is 0;  expected 200 

Fetching http://bookstore.bookstore:14001/buy-a-book/new 

Request Headers: map[] 

Error fetching http://bookstore.bookstore:14001/buy-a-book/new: Get "http://bookstore.bookstore:14001/buy-a-book/new": dial tcp 10.99.49.182:14001: connect: connection refused 

ERROR: response code for "http://bookstore.bookstore:14001/buy-a-book/new" is 0; 

expected 200 

… } 

 

Deploy SMI Traffic Access Policies  

To restore traffic access between the bookbuyer and bookstore, run the SMI traffic policy that will enable communication between bookbuyer and bookstore.  

$ Kubectl apply -f 01-deploy-smi-access-policy.yaml 

traffictarget.access.smi-spec.io/bookbuyer-access-bookstore created 

httproutegroup.specs.smi-spec.io/bookstore-service-routes created 

traffictarget.access.smi-spec.io/bookstore-access-bookwarehouse created 

httproutegroup.specs.smi-spec.io/bookwarehouse-service-routes created 

 

If the bookbuyer logs are checked again, traffic between bookbuyer and bookstore are restored. 

$ Kubectl logs "${POD}" -n bookbuyer -c bookbuyer --tail=100 -f  

{ … Fetching http://bookstore.bookstore:14001/books-bought 

Request Headers: map[Client-App:[bookbuyer] User-Agent:[Go-http-client/1.1]] 

Identity: bookstore-v1 

Booksbought: 552 

Server: envoy 

Date: Tue, 18 Oct 2022 04:56:07 GMT 

Status: 200 OK 

MAESTRO! THIS TEST SUCCEEDED! 

 

Fetching http://bookstore.bookstore:14001/buy-a-book/new 

Request Headers: map[] 

Identity: bookstore-v1 

Booksbought: 553 

Server: envoy 

Date: Tue, 18 Oct 2022 04:56:07 GMT 

Status: 200 OK 

MAESTRO! THIS TEST SUCCEEDED! 

…} 

 

Deploy bookstore v2 service  

To display the traffic split functionality of OSM, an additional bookstore v2 service was deployed to allow the bookbuyer reach either bookstore-v1 or bookstore-v2. The deployed bookstore-v2 service would have an SMI traffic policy created as well. 

$ Kubectl apply -f 02-deploy-bookstore-v2.yaml 

service/bookstore-v2 created 

serviceaccount/bookstore-v2 created 

deployment.apps/bookstore-v2 created 

traffictarget.access.smi-spec.io/bookbuyer-access-bookstore-v2 created   

 

Implement a SMI Traffic Split 

The SMI traffic split is applied to enable the bookbuyer to communicate with the bookstore and the bookstore-v2 service at weighted measures. Here, the weight was shared at 25% for bookstore and 75% for bookstore-v2 service.  

To see the content of the trafficsplit manifest run 

$ cat 03-deploy-smi-trafficsplit.yaml     

apiVersion: split.smi-spec.io/v1alpha2 

kind: TrafficSplit 

metadata: 

  name: bookstore-split 

  namespace: bookstore 

spec: 

  service: bookstore.bookstore 

  backends: 

  - service: bookstore 

    weight: 25 

  - service: bookstore-v2 

    weight: 75 

 

Apply the manifest by running 

$ Kubectl apply -f 03-deploy-smi-trafficsplit.yaml 

trafficsplit.split.smi-spec.io/bookstore-split created 

Trail the identity of the bookbuyer logs and see that traffic is split with 75% going to bookstore-v2 and 25% going to bookstore-v1. i.e. 

$  Kubectl logs "${POD}" -n bookbuyer -c bookbuyer --tail=100 -f | grep 'Identity:' 

Identity: bookstore-v2 

Identity: bookstore-v2 

Identity: bookstore-v2 

Identity: bookstore-v2 

Identity: bookstore-v1 

Identity: bookstore-v1 

Identity: bookstore-v2 

Identity: bookstore-v2 

Identity: bookstore-v1 

Identity: bookstore-v2 

Identity: bookstore-v2 

Identity: bookstore-v2 

 

To Uninstall OSM from your AKS hybrid cluster 

az k8s-extension delete --cluster-type connectedClusters --cluster-name $CLUSTER_NAME --resource-group $RESOURCE_GROUP --name osm  

 

Summary 

In this article, we see the capabilities of OSM been applied to a sample bookstore application and its usefulness in securing your Kubernetes cluster in a variety of ways like Ingress traffic management via policies to permit/deny external access into the mesh, Sidecar proxy injection using the envoy network proxy to ensure services participate in the mesh and obey policies that have been defined, Service-to-service access control via Layer 7 (http) access policies to allow authorized clients and disallow unauthorized clients, and Traffic splitting for canary and blue-green deployments.  

In a subsequent article, the observability benefits would be demonstrated to provide metrics, tracing, and logs. This would also enable visualizing application traffic flows, errors, latency and other metrics in tools like Grafana and Jaeger. 

 

Further reading 

Azure Arc-enabled Open Service Mesh - Azure Arc | Microsoft Learn 

Published on:

Learn more
Azure Stack Blog articles
Azure Stack Blog articles

Azure Stack Blog articles

Share post:

Related posts

Stay up to date with latest Microsoft Dynamics 365 and Power Platform news!
* Yes, I agree to the privacy policy