Understanding ExpressRoute private peering to address ExpressRoute resiliency

There are 4 ExpressRoute connectivity models between On-Prem and Azure, divided in 2 approaches: 3 ExpressRoute Service Provider models and 1 ExpressRoute Direct model.

In the ExpressRoute Service Provider models, ExpressRoute connectivity is provided to customers through service providers and can be delivered in 3 different ways. ExpressRoute Direct is a direct interconnection between Microsoft and the customer without intermediate.

The customer edge routers are located in a cloud exchange facility near to or at the peering location and are cross-connected with an ExpressRoute connectivity provider using either L2 or L3.

ExpressRoute Service Provider model #2: Ethernet Point to Point

ExpressRoute Service Provider model #3: Any-to-any connectivity

ExpressRoute Direct model

ExpressRoute Direct is a dedicated physical connection to the Microsoft backbone, established between a pair of MSEEs and customer routers, without any intermediate connectivity provider. The customer is allocated an entire MSEE port (10 Gbps or 100 Gbps) allowing the creation of multiple circuits on it.

Adam Stuart's video is another valuable resource for learning about ExpressRoute Direct.

To address ExpressRoute peering location failures, the recommended solution is to build a resilient design as outlined in this article and illustrated below:

Resiliency is achieved by deploying 2 ExpressRoute Circuits in 2 distinct ExpressRoute peering locations, thereby creating geo-redundant ExpressRoute circuits that are both connected to the same Azure ExpressRoute Gateway.

Based on this principle, environments with multiple Azure regions provide the opportunity to leverage existing ExpressRoute Circuits to achieve geo-redundancy through an ExpressRoute Bow-Tie design:

Because these designs introduce 2 parallel paths to Azure, traffic engineering mechanisms must be carefully implemented to prevent unexpected asymmetric routing.

To create an optimal geo-redundant ExpressRoute Circuit design, it is important to understand that the ExpressRoute Circuit SKU determines where an ExpressRoute circuit can connect to. Each SKU has a specific scope of supported regions (same metro/same geo/cross-geo):

See this great repo from Daniele Gaiulli to learn more about the the benefits of Expressroute bow-tie designs.

ExpressRoute and VPN can also be combined in an Active-Passive configuration to provide disaster recovery capabilities (doc).

• Both a VPN Gateway and an ExpressRoute Gateway can exist in the same GatewaySubnet.
• Transit routing between the ExpressRoute Gateway and the VPN Gateway is not possible without the use of ARS or Virtual WAN.
• ExpressRoute routes take precedence over other routes for the same prefixes.
• When different but overlapping prefixes are used, the route with the longest prefix is chosen.

During an ExpressRoute Circuit maintenance, one link out of the 2 fibers connecting the MSEEs and provider equipments remains available. AS-prepending is used by Microsoft to force traffic over the remaining link.

To prevent conflicts between Microsoft AS-prepending and On-Prem routing and to maintain On-Prem network resiliency, it is important to plan for this scenario and operate both links of an ER circuit (highlighted in yellow) as Active/Active when in nominal mode.

To prevent a single point of failure, it also recommended to terminate the primary and the secondary links of an ExpressRoute circuit on 2 separate customer routers as highlighted in orange on the above diagram.

More details in this video.

Therefore, the resiliency of the ExpressRoute Gateway is crucial to ensure end-to-end ExpressRoute resiliency. It is best practice to use AZ-redundant ExpressRoute Gateway SKUs for this purpose.

Comparison table:

AZ-redundant ExpressRoute Gateway instances are distributed across different Availability Zones:

As per Route Advertisement limits documentation, it is important to note that the max number of routes advertised from Azure to On-Prem is not the same as from On-Prem to Azure.

Custom alerts can be configured to monitor advertised routes over ExpressRoute.

It is essential to consider the ExpressRoute Circuit SKU and the specific route advertisement limit it offers while planning your network architecture.

Additional resources:

• Common ExpressRoute design errors in video.