MAC address strategies for tenant VMs running on Stack-HCI environment

Azure Stack HCI is a hyperconverged infrastructure (HCI) cluster solution consists of windows servers (Hyper-V), Storage Spaces Direct, and Azure-inspired SDN. All clustered servers share common configurations and resources by leveraging the Windows Server Failover Clustering feature. A Windows Failover Cluster consists of multiple windows servers running in a cluster to provide high availability i.e. If one server node goes down, then another node takes over. We can create multiple VMs on the failover cluster. VMs can be connected to different tenant networks. In this case we need to make sure VM connected on network 'A' with static ip should persist same network configuration even if it gets migrated from one node to another within a cluster. This is possible by assigning static MAC/static IP for the VM. But how to get free and unique MAC to assign to VM, there are different ways to solve this problem. In this article we will discuss some of them with pros and cons. (Note: solutions discussed in this article are just to suggestions, can not be considered optimal solutions)

Before this, lets understand first things required to create a tenant VM. To create tenant VM connected to tenant network in stack-hci following steps are performed. 

1. Create a new VM with vm network adapter having static MAC assigned

2. Create a new network interface on network controller service(running on SDN) having same MAC and static IP assigned.

3. Associate created network adapter with network interface controller, to make vm gets tenant network connection.

Following powershell commands perform above steps logically.

To make sure VMs works fine even after migration in windows failover cluster, we need to assign static MAC and static IP. To get available and unique MAC is challenge here. We will see following ways to solve this problem with pros and cons.

1. Generating random MAC on node
2. Assign dedicated unique MAC address range for every Node in cluster and Generating free MAC from Node from pool of MAC address range.
3. Get free MAC from Network controller service and use same on VM

Lets discuss each approach in detailed-

1. Generating random MAC on node

This is very simple approach, where we will generate random MAC and same will be used. It includes below steps:

• Generate valid MAC on node, where we need to create VM(or new network interface).
• Set same MAC on vm network adapter as static
• Set same MAC on network interface along with static IP

Following is powershell script performs same above logical steps:

But this approach has following problems :

• There is possibility that two vms running on same cluster may got same MAC and it may create MAC conflict issue after migration.
• Some specific MAC range is not allowed to be assigned, in this case we need to keep retrying random generation, which is not good.
• Very difficult to keep track of used MAC addresses.

So this method is not reliable and not recommended.

2. Assign dedicated unique MAC address pool range for every Node in cluster and Generating free MAC from Node's MAC address range pool.

We can preassign unique dedicated MAC address pool range to every node(hyper-v) in cluster, so whenever we need to create new network interface we will first get free MAC from the pool of node where vm will be running. Same MAC will be made static on vm network adapter and network interface followed by static ip. Following are logical steps need to perform:

1. Get free MAC from Node's MAC pool range.

2. Assign same static MAC on VM network adapter 

3. Set same MAC on network interface along with static IP

But main challenge here is there is no api or command available to get free MAC from node pool. There is a workaround that to get MAC assigned from the pool, we need to create dummy network adapter with dynamic configuration and start vm for a moment to get free dynamic MAC assigned from node pool range. Then we need to stop vm and revert the change and using same MAC we need to create static one and same steps need to be followed. This is not good approach since this is just a workaround, where dummy resource need to create.

# creating new network adapter with dynamic MAC
Add-VMNetworkAdapter -VMName $vm_name -SwitchName $switch_name -Name $adapter_name 

# starting and stopping vm to get free MAC from node pool
Start-VM -VMName $vm_name; Stop-VM -VMName $vm_name -Force 

# reading assigned dynamic MAC
$mac_address = (Get-VMNetworkAdapter -VMName $vm_name -Name $adapter_name).MacAddress 

# make same MAC as static
Set-VMNetworkAdapter -VMName $vm_name -Name $adapter_name -StaticMacAddress $mac_address 

# rest steps are same for network interface creation

3. Get free MAC from Network controller service and use same on VM network adapter.

There is a dedicated centralized network controller service on every Stack-hci cluster, where we can setup global MAC address range pool at network controller service. When we create new network interface on network controller service with dynamic configuration, then it assign free MAC from the global free pool. It is very reliable solution since it is a centralized service. In this, we will follow below steps

1. Create a Network interface in Network controller service with dynamic MAC assignment config, to get assigned free MAC from global MAC range.
2. Read assigned MAC on network interface
3. Assign same MAC address to vm network adapter as static. 

The following powershell commands perform above steps (Note: please use appropriate values in place of parameters):

# Creating network interface with dynamic MAC config on network controller
$vmnicproperties = New-Object Microsoft.Windows.NetworkController.NetworkInterfaceProperties
$vmnicproperties.PrivateMacAllocationMethod = "Dynamic"
$vmnicproperties.IsPrimary = $true

$vmnicproperties.DnsSettings = New-Object Microsoft.Windows.NetworkController.NetworkInterfaceDnsSettings
$vmnicproperties.DnsSettings.DnsServers = $dns_server
$ipconfiguration = New-Object Microsoft.Windows.NetworkController.NetworkInterfaceIpConfiguration
$ipconfiguration.resourceid = $vm_name + "_IP1"
$ipconfiguration.properties = New-Object Microsoft.Windows.NetworkController.NetworkInterfaceIpConfigurationProperties
$ipconfiguration.properties.PrivateIPAddress = $ip_address
$ipconfiguration.properties.PrivateIPAllocationMethod = "Static"

$ipconfiguration.properties.Subnet = New-Object Microsoft.Windows.NetworkController.Subnet

$ipconfiguration.properties.subnet.ResourceRef = "/virtualNetworks/" + $vnet_name + "/subnets/" + $subnet_name
$vmnicproperties.IpConfigurations = @($ipconfiguration)
$NIC_name = $vm_name + "_Eth1"
New-NetworkControllerNetworkInterface -ResourceID $NIC_name -Properties $vmnicproperties -ConnectionUri $uri -Confirm:$false -force
Write-host 'NIC config created..'
Start-Sleep -Seconds 8

$nic = Get-NetworkControllerNetworkInterface -ConnectionUri $uri -ResourceId $NIC_name
# Read obtained free MAC from global pool
$mac_address = $nic.Properties.PrivateMacAddress -replace '..(?!$)', '$&-'

###### Vm creation flow starts from here. We will set static MAC here
New-VM -Name $vm_name -MemoryStartupBytes $vm_memory -BootDevice VHD -VHDPath $image_path -Path $vm_data_path -Generation $vm_generation -SwitchName $switch_name
Add-ClusterVirtualMachineRole -vmname $vm_name -Name $vm_name
Set-VMNetworkAdapter -VMName $vm_name -StaticMacAddress "$mac_address"
write-host "Mac set succssfully: $mac_address"

Conclusion:

The third/last approach discussed in this article seems to be easy and more reliable, since we are consuming MAC from centralized network controller service. We will never face problem of MAC conflicts incase VMs get migrated from one node to another.