AzureHPC Lustre Marketplace Offer

The AzureHPC Lustre Marketplace offer provides a fast and easy way to deploy a Lustre Filesystem on Azure. The features include:

• Deploy a Lustre Filesystem to an existing VNET in minutes​​
• Flexibility to select different VM SKUs and disk options to meet you IO requirements
• Import an Azure BLOB storage account using Lustre HSM​​
• Option to deploy a policy manager to keep the file system synchronised with Azure BLOB storage including automatic backup
• Dashboard to monitor Lustre performance​

Disclaimer: The Marketplace offer will create the initial Filesystem that is up to the end user to manage. Support will only be provided on a best effort basis through "Issues" on the azlustre GitHub page.

Example Sizes

When choosing VM and disk options you should pay attention to the different limits. Here are the potential bottlenecks:

• VM expected network bandwidth
• VM maximum IOPs/bandwidth to managed storage ("Max uncached disk throughput" in the Azure documentation)
• VM maximum IOPs/bandwidth to temporary storage
• Managed storage IOPs/bandwidth per disk (ensure sufficient disks to fully utilize the VM uncached disk throughput)

The VM limits are set by the VM SKU and details are provided from here. The storage limits for managed disks are available here.

Below are some example configurations with expected performance:

Improvements can be seen using SKUs with bursting capabilities in addition to enabling caching.

Deployment Walkthrough

This section provides a walkthrough to create Lustre Filesystem from the Azure Marketplace Offer. Ensure the following prerequisites have been completed before proceeding:

• The VNet/subnet must already be created in the subscription.
• A storage account and container should already exist if Hierarchical Storage Management/Policy Engine is used.

Create the marketplace solution from the portal

Select Marketplace from the Azure Portal and search for AzureHPC Lustre FS. Use the Create dropdown menu and select AzureHPC Lustre 2.12.

AzureHPC Lustre FS Basic Options

The first options screen is split into four sections:

AzureHPC Lustre FS Metadata Options

This screen is for options regarding the MDS.

AzureHPC Lustre FS Storage Options

This screen is for options regarding the OSS.

AzureHPC Lustre FS Object Storage (HSM) Options

This screen is for options regarding the HSM and whether to deploy a policy engine.

Storage SAS Key

The SAS key requires read, write, list and delete permissions. Be aware when choosing an expiry time that it will be stored on the OSS/Policy Engine VMs and used for the HSM operations. Here is an example of creating a SAS key with a 1 month expiry using the Azure CLI:

# TODO: set the account name and container name below
account_name=
container_name=

start_date=$(date +"%Y-%m-%dT%H:%M:%SZ")
expiry_date=$(date +"%Y-%m-%dT%H:%M:%SZ" --date "next month")

az storage container generate-sas \
   --account-name $account_name \
   --name $container_name \
   --permissions rwld \
   --start $start_date \
   --expiry $expiry_date \
   -o tsv

Note: HSM will stop working if the SAS key expires.

Deploy

After reviewing the options and selecting the Create button, the Lustre Filesystem will be deployed. The deployment will take a few minutes to complete.

The new resource group will contain the following resources:

• MDS VM​
• OSS VMSS​
• Policy Engine VM [if policy engine is selected]
• Dashboard​

Accessing the Lustre Filesystem

Once the deployment is complete the Lustre packages can be installed and the Filesystem can be mounted on client VMs.

Installing the Lustre client packages

The version of Lustre is 2.12.5 although later versions of the client are usually backwards compatible. The 2.12.5 packages can be downloaded from the Whamcloud download site.

Below is a script that will install the Lustre client packages on a CentOS 7.8 (tested on both the regular and hpc image).

lustre_dir=lustre-2.12.5

# set up the Lustre repository
cat << EOF >/etc/yum.repos.d/LustrePack.repo
[lustreserver]
name=lustreserver
baseurl=https://downloads.whamcloud.com/public/lustre/${lustre_dir}/el7/patchless-ldiskfs-server/
enabled=1
gpgcheck=0
[e2fs]
name=e2fs
baseurl=https://downloads.whamcloud.com/public/e2fsprogs/latest/el7/
enabled=1
gpgcheck=0
[lustreclient]
name=lustreclient
baseurl=https://downloads.whamcloud.com/public/lustre/${lustre_dir}/el7/client/
enabled=1
gpgcheck=0
EOF

# install the client RPMs if not already installed
if ! rpm -q lustre-client kmod-lustre-client; then
    yum -y install lustre-client kmod-lustre-client
fi
weak-modules --add-kernel $(uname -r)

Mounting the Lustre Filesystem

The script below will mount the filesystem. Make sure that lustre_name is the Virtual Machine name/prefix chosen at deployment time and the lustre_mount is the mount point to be created and used.

lustre_name=lustre
lustre_mount=/lustre

mkdir -p $lustre_mount
echo "${lustre_name}@tcp0:/LustreFS $lustre_mount lustre flock,defaults,_netdev 0 0" >> /etc/fstab
mount -a

Monitoring the Filesystem with the Dashboard

The dashboard can be accessed through the portal and provides the following information:

• Metrics for MDS and OSS VMs​
• CPU Usage​
• Network​
• Disk metrics​
• Disk writes​
• Consumed Uncached IOPs/Bandwidth​
• Consumed Cached IOPs/Bandwidth​

This can be used to easily identify performance bottlenecks or badly performing VMs.

Policy Engine

The policy engine VM runs RobinHood is used in addition to a lustre changelog reader from the lemur repository. These are both run as systemd services on the policy engine VM. The services are called robinhood and lustremetasync.

HSM was designed for just archive data and assumes the metadata target will persist. The lustre setup on Azure will maintain metadata in Azure BLOB storage so the filesystem can be removed and easily recreated. The reason for the changelog reader is to handle cases not covered by RobinHood and HSM:

• Creating/deleting directories in the archive
• Creating/deleting symlinks in the archive
• Updating owner/attribute changes to files and directories
• Handle moving files/directories

This is diagram shows show the Lustre is set up with HSM and the policy engine:

The RobinHood setup has the following options:

• Files are archived after 30 minutes of in-activity
• Files are released once the filesystem reaches 85% capacity

he Robinhood config files are in /etc/robinhood.d/lustre.conf on the policy engine VM.

Software Versions

The AzureHPC Lustre image uses OpenLogic CentOS 7.8 as the base image. The Lustre version is 2.12.5 and packages are from the Whamcloud download site. The HSM version packages are compiled from lemur v2.1.2. All the installation scripts are available on GitHub in the azlustre project.