Loading...

Spack in a Multi-User HPC Environment on Azure

Spack in a Multi-User HPC Environment on Azure

Spack is a package management tool designed for HPC environments.  In this article we will demonstrate how to install and configure Spack in a multi-user HPC environment on Azure with shared and user repositories.  An environment could be easily created with Azure HPC On-demand Platform (Az-HOP) or Azure CycleCloud.  This allows the system administrator to provide a collection of shared packages to all users and individual users can build packages on top of this in their own repository without rebuilding or interfering with one another. 

 

Initial setup

 

Create a user to store shared packages, for example spackuser. Log on as spackuser.  Download spack from GitHub:

 

git clone https://github.com/spack/spack.git

 

Use the development branch as it contains latest Intel packages.  To load the environment:

 

source spack/share/spack/setup-env.sh

 

Changing the number of processors to build with:

 

spack config --scope defaults add config:build_jobs:32

 

Fixes for Azure

 

Spack detects the CPU using CPUID flags. Azure platform exposes subset of the flags supported by underlying CPUs, in some cases causing spack to detect architecture incorrectly. In particular, “clzero” flag is not exposed in Azure HB VM and “pku” flag is not exposed in Azure HBv3. These flags are used to detect “zen” and “zen3” platforms. To correct spack detection on Azure, these flags need to be removed from the microarchitecture definition:

 

MICROARCHFILE=$SPACK_ROOT/lib/spack/external/archspec/json/cpu/microarchitectures.json mv $MICROARCHFILE ${MICROARCHFILE}.orig cat ${MICROARCHFILE}.orig \ | jq 'del(.microarchitectures.zen3.features[] | select (. == "pku"))' \ | jq 'del(.microarchitectures.zen.features[] | select (. == "clzero"))' \ >$MICROARCHFILE

 

Installing Compilers

 

Using System Compilers

 

At the time of writing, gcc-9.2.0 is included in the CentOS HPC image, and the following command will register it with spack:

 

spack compiler find /opt/gcc-9.2.0

 

However, when initially installing it will find a compiler in the path.  You can check installed compilers using:

 

spack compilers

 

Installing latest GCC

 

 

spack install gcc spack load gcc spack compiler find --scope site

 

First command will download sources for GCC and dependencies, and build them. Second command will set up environment variables, and the last command registers the newly built compiler. Parameter “--scope site” tells spack to store the compiler configuration in $SPACK_ROOT/etc/spack/ directory. It will be local to this instance of spack and shared between all users using it. Default is “--scope user” and it stores the settings in $HOME/.spack.

To check what versions of GCC are available to download from spack repositories, run the following command:

 

spack info gcc

 

The output will show the preferred version that will be used by default. However, you can select a version using the @ syntax:

 

spack install [email protected]

 

To use the specific compiler when installing other packages, for example to install intel-mpi-benchmarks compiled with gcc-12.1.0, the syntax is:

 

spack install intel-mpi-benchmarks%[email protected]

 

Installing MPI

 

MPI libraries can be installed from spack repository using spack install. Alternatively, existing MPI packages already installed on the system can also be configured. In this example we install Intel OneAPI MPI using spack, and set up preinstalled Mellanox HPC-X MPI.

Intel MPI

 

The latest Intel MPI can be installed from SPACK and it will correctly use InfiniBand. For example:

 

spack install intel-oneapi-mpi

 

HPCX MPI

 

HPC-X MPI with other components (UCX, HCOLL) is preinstalled in the HPC image in /opt/hpcx-<version>. To add it to spack, add the following lines (note: version may be different depending on the Azure HPC image version) to the packages section of $SPACK_ROOT/etc/spack/packages.yaml:

 

packages: hpcx-mpi: buildable: false externals: - spec: [email protected] prefix: /opt/hpcx-v2.8.3-gcc-MLNX_OFED_LINUX-5.2-2.2.3.0-redhat7.9-x86_64/ompi

 

The following code may be used to find and install all installed versions of hpcx on an Azure HPC image:

 

for hpcx_dir in $(ls -d /opt/hpcx-v* | xargs -n 1 basename); do hpcx_ver=$(echo $hpcx_dir | cut -f 2 -d - | sed 's/^v//g') cat << EOF >> $SPACK_ROOT/etc/spack/packages.yaml packages: hpcx-mpi: buildable: false externals: - spec: hpcx-mpi@$hpcx_ver prefix: /opt/$hpcx_dir/ompi EOF done

 

After the change to the packages.yaml is done, you need to install (register) HPCX in spack:

 

spack install hpcx-mpi

 

Install and run Intel MPI Benchmarks (IMB)

 

The Intel MPI benchmarks package can be used to benchmark the cluster and test the communications are working. Use spack install command to build the package with the different compilers and MPI versions, for example:

 

# Intel compiler and Intel MPI spack install intel-mpi-benchmarks%intel # GCC compiler and Intel MPI spack install intel-mpi-benchmarks%[email protected]^intel-oneapi-mpi # GCC compiler and HPCX MPI spack install intel-mpi-benchmarks%[email protected]^hpcx-mpi

 

Load the package and run:

 

spack load intel-mpi-benchmarks%[email protected]^hpcx-mpi mpirun -np 120 IMB-MPI1 Allreduce

 

Chaining Spack Installation for Users

 

For users to be able to use shared packages provided by the system administrators and to create/install their own packages, one of the ways is to create a chained spack installation. To do that, individual users will need to install a private copy of spack in their home directory and connect it with the shared (“upstream”).

 

Install local copy of spack

 

Log on to the cluster as normal user and clone the repository:

 

git clone https://github.com/spack/spack.git

 

Load the local environment and set your preferred “make -j” value:

 

source spack/share/spack/setup-env.sh spack config --scope defaults add config:build_jobs:32

 

Note the location of the shared install (make sure the directory is readable by users):

 

SPACK_SHARED_DIR=/anfhome/spackuser/spack

 

Copy Azure CPU definition file from the shared install:

 

ARCH_FILE=lib/spack/external/archspec/json/cpu/microarchitectures.json cp $SPACK_SHARED_DIR/$ARCH_FILE spack/$ARCH_FILE

 

Copy compiler configuration:

 

cp $SPACK_SHARED_DIR/etc/spack/compilers.yaml spack/etc/spack/

 

Create file spack/etc/spack/upstreams.yaml with following contents:

 

upstreams: spack-shared: install_tree: $SPACK_SHARED_DIR/opt/spack

 

The chained spack is now ready to use. You can check that all upstream compilers and packages are available:

 

spack find spack compilers

 

Creating a new package

 

Create a new package:

 

spack create https://github.com/edwardsp/helloworld-mpi/archive/bc9758923e921d63878ac984f8c282f8157d38cc.zip

 

The initial setup will be guessed by spack.  Here are updates to build:

 

from spack import * class HelloworldMpi(Package): """A simple hello world MPI application.""" homepage = "https://github.com/edwardsp/helloworld-mpi" url = "https://github.com/edwardsp/helloworld-mpi/archive" version("1.0.0", sha256="9af66c590de12276f261a93c170a76776cce0a9214c244c8848a158b46f8e607", url="https://github.com/edwardsp/helloworld-mpi/archive/bc9758923e921d63878ac984f8c282f8157d38cc.zip") depends_on('mpi') def install(self, spec, prefix): from os import mkdir from os.path import join from subprocess import check_output bin_dir = join(self.prefix, "bin") mkdir(bin_dir) check_output([spec["mpi"].mpicc, "-o", bin_dir+"/helloworld-mpi", "main.c"])

 

The example we are using here does not have any build scripts associated with it and uses the base Package class with spack.  The spack package install is creating the bin directory and running the mpicc command directly to build the executable.  Spack has support for many different build systems available and should be autodetected from the source code.  The documentation is available here.

The package can be installed, loaded and run as follows:

 

spack install helloworld-mpi@intel^intel-oneapi-mpi spack load helloworld-mpi@intel^intel-oneapi-mpi mpirun -bootstrap ssh -np 120 helloworld-mpi

 

Published on:

Learn more
Azure Global articles
Azure Global articles

Azure Global articles

Share post:

Related posts

Code AI apps on Azure - Python, Prompty & Visual Studio

Build your own custom applications with Azure AI right from your code. With Azure AI, leverage over 1,700 models, seamlessly integrating them ...

2 days ago

Network Connectivity for RISE with SAP S/4HANA Cloud Private Edition on Azure

In this article, we will explore different ways to connect to RISE with SAP S/4HANA Cloud Private Edition deployment on Azure, guiding yo...

2 days ago

Azure Landing Zones - Policy Refresh Q1 FY25

ALZ - Policy Refresh Q1 FY25 is here! As you may be aware, the ALZ team release cadence is now on quarterly basis to help customers and partne...

2 days ago

Debug Queries More Efficiently with the Improved Error Messaging in Azure Cosmos DB Data Explorer

Azure Cosmos DB Data Explorer is a web-based tool available in the Azure Portal that allows you to manage data, as well as track and fix issue...

3 days ago

Meet the Winners | Microsoft Developers Azure AI & Azure Cosmos DB Learning Hackathon

Azure Cosmos DB powers some of the world’s most popular intelligent apps like ChatGPT. In a recent hackathon, Over 9,500 developers engaged wi...

3 days ago

Introducing RBAC Authentication and more for the Azure Cosmos DB Integrated Cache

We’re excited to announce new features for the Azure Cosmos DB! The integrated cache is built into the dedicated gateway, and now there’s new ...

3 days ago

Microsoft DiskANN in Azure Cosmos DB Whitepaper

We are excited to publish a new whitepaper titled, Microsoft DiskANN in Azure Cosmos DB, where we examine the impressive capabilities of Micro...

3 days ago

Announcing Private Preview: VS Code Extension of vCore-based Azure Cosmos DB for MongoDB

Overview We’re excited to introduce a new VS Code extension for vCore-based Azure Cosmos DB for MongoDB ! This tool allows users to conn...

4 days ago

Azure Communication Services September 2024 Feature Updates

The Azure Communication Services team is excited to share several new product and feature updates released in August 2024. (You can view previ...

4 days ago
Stay up to date with latest Microsoft Dynamics 365 and Power Platform news!
* Yes, I agree to the privacy policy