Moneo: Distributed GPU System Monitoring for AI Workflows
Microsoft has introduced a new open-source GPU monitoring framework named Moneo (Latin for monitor). Moneo orchestrates metric collection (DCGMI + Prometheus DB) and visualization (Grafana) across multi-GPU/node systems. This provides useful insights into workflow and system level characterization.
GPUs are optimized for high throughput, massively parallel, workloads. Efficient use of GPUs is dependent on a few factors such as workload characteristics, system characteristics, and sometimes physical environment. GPU system level monitoring determines GPU utilization and facilitates workload characterization. This aids in exploring how to utilize the GPU more efficiently.
Monitoring GPU metrics on a single system over a period may be trivial. However, formatting and analyzing the raw metric data so that it can provide intuitive insights can prove to be a tedious task. Our goal is to pair system level characterization with application metrics to determine the efficiency of our use of the GPUs.
Given that there is already some complexity with collecting and analyzing GPU metrics on a single system, unsurprisingly, scaling the same methodology for multiple systems is difficult.
For certain deep learning models use of distributed multi-GPU systems is the only feasible way to train in a reasonable time frame. Much of the application complexity is abstracted away by high level AI frameworks, but there are still configurations and design choices that users must make that ultimately affect the throughput and behavior of model training.
Moneo’s usefulness in providing system level insights can help guide design choices to achieve the efficient use of GPU systems.
Moneo Design
Figure 1: Design
Three categories of metrics that Moneo monitors:
- Device Counters
- Compute/Memory Utilization
- Streaming multiprocessor (SM) and Memory Clock frequency
- Temperature
- Power
- ECC Counts
- Profiling Counters
- SM Activity
- Memory Dram Activity
- NVLink Activity
- PCIE Rate
- InfiniBand Network Counters
- IB TX/RX rate
Once Moneo has been launched these metrics can be viewed from the Grafana portal. See figures 2,3,4 for snapshots of the different metric views.
Figure 2: Device Counter View
Figure 3: Profiling Counter View
Figure 4: IB Counter View
Getting Started
Starting with Moneo is easy. Just clone the latest release from the Moneo Repo and follow the README for detailed setup instructions or take a look at the quick start guide. In a short period, you should be able to launch Moneo with a single command and log into the Grafana portal to start seeing results!
Moneo is also available on Azure HPC + AI Ubuntu images. Just navigate to “/opt/azurehpc/tools/Moneo”. The image has all the required dependencies installed. So, all that’s necessary is configuring and deploying Moneo.
Quick start instructions:
- Clone Moneo from Github and install ansible.
- git clone https://github.com/Azure/Moneo.git
- cd Moneo
- python3 -m pip install ansible
- Next create a host.ini config file.
-
Note: The master node can also be a worker node as well. The master node will have the Grafana and Prometheus docker containers deployed to it.
-
Note: If you have configured password less SSH already, [all:vars] section can be skipped.
-
Note: The master node must be able to ssh into itself.
-
- Now deploy Moneo
- ansible-playbook -i host.ini src/ansible/deploy.yaml
- Log into the portal by navigating to http://master-ip-or-domain:3000 and inputting your credentials
- Note: By default, username/password are set to "azure". This can be changed here "src/master/grafana/grafana.env"
- Navigating Moneo Grafana Portal
- The current view is labeled in the top left corner:
- VM instance and GPU can be selected from the drop-down menus in the top left corner:
- Various actions such as dashboard selection or data source configuration can be achieved using the left screen menu:
- Metric groups are collapsible:
- The current view is labeled in the top left corner:
Published on:
Learn moreRelated posts
Power Automate: Renew App Registration Client Secret and Update Dataverse Environment Variable Value
Last week, we learned how to create an email notification to remind us if App Registration in Azure expires. Today, we will learn how to apply...
Resolved: Fix connections issue in Power Automate Flow
I got the below connection issues in all the Power Automate flow all of a sudden after opening the Flow. And I followed the below steps and re...
Approvals in Power Apps with Power Automate, Teams, and Adaptive Cards
Table of Contents Introduction Are you looking for a simple and extensible no-code solution for handling approvals in Power Apps? The post App...
Microsoft Power Automate: A Low-Code Solution for Automating Your Processes
Table of Contents Introduction The recurring changes in technology trends are constantly reshaping the business landscape. The race for growth...
Notification For Failed Flow Runs In Power Automate
If you create flows for your organisation, or for clients on projects, you’ve no doubt improved your skills over the years. This might m...
The Developer's Approach to Power Automate
How to Maximize Your Business Efficiency with Power BI, Power Apps, Power Automate?
In today’s fast-paced world, to remain ahead of the edge as a business means ensuring you are ahead in all parameters. Like automating workflo...
Power Automate Save File To A Local Drive
Power Automate can save a file to a local drive but there is a considerable ... The post Power Automate Save File To A Local Drive appeared fi...