Leverage Azure API Management to distribute API traffic to multiple backend services
This is a summarized edition of the following original articles, both of which were published in Japanese. This is a memo as of 27 January 2024.
Azure API Managementのバックエンドプール – Logico Inside (logico-jp.io)
Azure API Managementのバックエンドでサーキットブレーカーを構成する – Logico Inside (logico-jp.io)
Circuit breaker and load balanced pool features have recently been added to the API Management backend. Both are in public preview as of 27 January 2024.
Azure API Management backends | Microsoft Learn
In public preview, we must use either Bicep, ARM, or REST APIs (2023-05-01-preview
or later) to configure circuit breakers and load-balanced pools. There is no command for either CLI or PowerShell and there is no dedicated configuration page in Azure Portal.
Backend - REST API (Azure API Management) | Microsoft Learn
Load-balanced pool
Previously a backend could only contain one endpoint of a backend service. If we need to distribute traffic to multiple backend services, we need to provision and deploy an L4/L7 load balancer in front of the backend services. Load-balanced pool allows us to distribute request traffic to multiple backend services without managed/unmanaged load balancer. This feature has already been included in other API gateway products, but finally this comes to Azure API Management.
Limitation
There are some limitations as of 27 January 2024.
- Round-robin is the only load balancing option. (We hope that other load balancing methods will be implemented in GA.)
- As mentioned in the documentation, an API Management instance with Developer and Premium SKUs deployed in an internal virtual network may throw HTTP 500
BackendConnectionFailure
errors when the gateway endpoint URL and backend URL are the same. Please refer to the following URL for more details.
Azure API Management backends | Microsoft Learn - Nested load-balanced pools are not supported.
Circuit breaker
Another feature, circuit breaker, is a common pattern for preventing an application from repeatedly trying to perform an operation that’s likely to fail. Previously we had to implement a circuit breaker in backend service(s) by ourselves, but we don’t have to implement this feature.
Circuit Breaker pattern - Azure Architecture Center | Microsoft Learn
Limitation
Some limitations are found as of January 27, 2024.
- Backend circuit breaker is not supported in the API Management Consumption SKU.
- “Half-open” state of circuit breaker is not supported.
Please refer to “NOTE” in the document for more details.
Azure API Management backends | Microsoft Learn
Configuration
Preparation
The following endpoints will be used for the backend.
<URL1>/api/test
<URL2>/api/test
For the API frontend, we use the following configuration.
<APIM's URL>/suffix/foo
In this case, requests will normally be routed to the following URL, unless no change is made to the routing endpoint in inbound
section.
<URL1 or URL2>/api/test/foo
Circuit breakers
To configure a circuit breaker for backend, the following JSON is published via REST API. Needless to say, we can configure the circuit breaker with ARM and Bicep.
Regarding properties.circuitBreaker.rules.failureCondition.interval
and properties.circuitBreaker.rules.tripDuration
, both should follow ISO8601 duration specification. In case of the example above, interval
(duration to collect the number of errors) is 5 minutes, and tripDuration
(duration during which circuit breaker is tripped) is 10 minutes.
A same configuration should be made for test2
.
Load-balanced pool
We will use the two backends configured in the last section to configure the load-balanced pool. Although, properties.url
is mandatory, specified URL is meaningless. In this case, it is okay to specify http://localhost
to properties.url
. Also note that Pool
should be specified in properties.type
, instead of Single
. The following JSON is then posted via REST API. As I mentioned in the last section, we can also use ARM and Bicep.
That’s it. Now it is time to test.
Let’s try it.
In this example, the backend test2
always returns HTTP 503
(Service Unavailable). I called the API every 5 seconds to observe behaviours using the APIM request tracing feature.
Tutorial - Debug APIs in Azure API Management using request tracing | Microsoft Learn
After 6 API calls, the circuit breaker is tripped since test2
returns HTTP 503
(Service Unavailable) error 3 times.
At the 7th or 8th call, the API request would normally be routed to the backend test2
in the load-balanced pool, but the request is only routed to the backend test1 because APIM detects that the circuit breaker of the backend test2
is tripped. 10 minutes later, the circuit breaker stops tripping, and the normal state is resumed; the load-balanced pool is distributing API requests to the one of backends in the pool.
How about using these features with retry
policy?
If we use a load-balanced pool and a circuit breaker with retry
policy in backend
section, we can configure the routing of API requests in a more sophisticated way. As I wrote in the following entry, we can configure fallback more easily, similar to configuring with L7 load balancer.
If we use both features introduced to backends with retry
policy, all we need to do is slightly modify the original policy configuration in the URL above, which is for request distribution to one of the Azure OpenAI Service instances, to use load-balanced pool and specify it as a backend service. Here is the example of policy configuration.
This policy configuration meets the requirements shown below.
Conclusion
Previously backends did not have load balancing or circuit breaker functionality, but now both of them have been added to Azure API Management. We hope that both will reach GA soon and we will be able to route API requests to backends in a more sophisticated way.
Published on:
Learn moreRelated posts
PostgreSQL for your AI app's backend | Azure Database for PostgreSQL Flexible Server
Use Postgres as a managed service on Azure. As you build generative AI apps, explore advantages of Azure Database for Postgres Flexible Server...
Storage migration: Combine Azure Storage Mover and Azure Data Box
Migrating storage from on-premises can be challenging. That’s why we are on a mission to make your migrations as simple as possible. We've dev...
Loop DDoS Attacks: Understanding the Threat and Azure's Defense
In the realm of cybersecurity, Distributed Denial-of-Service (DDoS) attacks are a significant concern. The recent holiday season has unveiled ...
Azure Communication Services at Microsoft Build 2024
Join us in-person in Seattle or virtually for Microsoft Build 2024 from May 21 to 24. We're excited to share the latest updates from Azure Com...
Azure Developer CLI (azd) – May 2024 Release
This post announces the May release of the Azure Developer CLI (`azd`), which includes a new demo mode, various .NET Aspire enhancements, and ...
Join us at Build 2024: Get the latest on Azure Cosmos DB in Seattle or online!
Are you ready to dive into the future of AI and data innovation? Microsoft Build 2024 is just around the corner, taking place May 21–23 in Sea...
Announcing Data API builder General Availability for Azure Cosmos DB
Great news for developers out there! The Data API Builder for Azure Cosmos DB is now officially available, eliminating the need for writing co...
Organizing rule collections and rule collection groups in Azure Firewall Policy
Firewall Policy is the recommended method to manage Azure Firewall security and operational configurations. When using Firewall Policy, any ru...
Get Started with Azure AI Services | Open AI and Deployment Models
Table of Contents Overview - Azure AI Services Kind of Azure AI Services Responsible AI Services Limited Access Features Cognitive Acc...