Modular Chassis — common chassis design for future of accelerators

A community-driven approach to infrastructure innovation is vital – not just for continued advancements in trust, efficiency, and scalability, but in service of a larger vision of empowering the ecosystem towards building for the computing needs of tomorrow.  

We are contributing a new hardware design - modular chassis (Mt. Shasta) to Open Compute Project Foundation (OCP), whose mission is “to apply the benefits of open source and open collaboration to hardware and rapidly increase the pace of innovation in, near and around the data center.” In the world of high-performance computing (HPC), artificial intelligence (AI) and video codecs, there is a need for a converged architecture that brings form factor, power and management interface into a modular design. Mt. Shasta is a modular chassis that enables multiple accelerators to fit into a single chassis supporting 48V power feed. Designed in partnership with Quanta and Molex, Mt. Shasta is designed to be fully compatible with Open Rack V3, with a modular form factor that provides support for high power devices, and hot swappable modules with flexibility in changing module-module connectivity. Mt Shasta also supports additional flexibility of a host connecting to the chassis.

Majority of the devices we deploy in data centers today are general purpose CPUs. To deploy these in the data center, we design 1RU or 2RU systems and mount them in one of our standard racks like Mt. Olympus. We have BMC based out of band management infrastructure and host CPU based in band management infrastructure for automation in our data center. Process and tools are well established to support this effort. Automation tools support deployment, health monitoring and diagnostics of hardware.

Moving forward we see a variety of domain specific devices like various accelerators are coming to datacenter. One option is to deploy vendor developed systems with necessary updates to integrate with datacenter infrastructure.  Our datacenter follows certain power, cooling and connectivity guidelines, off-the-shelf systems need customization to be compatible with these guidelines. Our automation tools also need specific interfaces both in hardware and software to reach devices for monitoring and management. These modifications are time consuming and involves significant development effort. Another option is to build Peripheral Component Interconnect Express (PCIe) cards and plug them into one of the standard servers and deploy. PCIe has power and cooling limitations and hot swap is also not supported in PCIe.

With Mt. Shasta we address these challenges of updating vendor designed systems to be deployable in our data center and limitations of modular PCIe designs. Datacenter facing hardware and software components are developed as part of the chassis. Device specific components are developed as part of module development. This helps to efficiently collaborate with partners and reduce the time it takes to deploy.

Mt. Shasta is a modular chassis that enables multiple accelerators to fit into a single chassis supporting 48V power feed. Modular form factor provides support for high power devices, and hot swappable modules with flexibility in changing module-module connectivity. Mt Shasta also supports additional flexibility of a host connecting to the chassis.

Chassis Management Module (CMM) in Mt. Shasta is connected to an external head node (host server) though PCIe cable. This root complex connection terminates to a PCIe switch in CMM. The down ports of PCIe switch are connected to device modules over cable back plane. DC-SCM module in CMM  is responsible for chassis management and module management. With head node and DC-SCM in CMM, Mt. Shasta chassis provides management interface to datacenter automation tools in the same way any other server does.

Chassis picks up 48V from power bus bar in rack and feed 48V to a power distribution board (PDB). Intermediate bus voltage converter generates 12V from 48 V and feeds to modules and FANs through cable harness. PDB can be modified to generate any other intermediate bus voltage or feed 48V directly to modules.

Cable back plane in Mt. Shasta provide connectivity (PCIe down ports and other management side band signals) across CMM and device modules. The current design supports GEN3 x4 interface to each device module. Design can be easily upgraded to support higher speeds and lane counts. The cable back plane and holding frame can be positioned closer to front of chassis or away from to support different module geometry. This can be supported with minor design changes.

It is a simple solution that can help to collaborate with ODMs and device vendors without compromising competitive advantage to create win-win scenario for all. We are working on contributing the specification to OCP, soon we will be contributing the design as well.