Azure Application Gateway: 502 error due to backend certificate not whitelisted in the AppGW

If you are using Azure Application Gateway as Layer 7 WAF for End to End SSL connectivity , you might have come across Certificate related issues most of the times. In this article I am going to talk about one most common issue  "backend certificate not whitelisted" 

 

If you check the backend health of the application gateway you will see the error like this  "The root certificate of the server certificate used by the backend does not match the trusted root certificate added to the application gateway. Ensure that you add the correct root certificate to whitelist the backend"

 

 

The error says that Root cert is not whitelisted on the AppGW , but you might have a valid Third party certificate on the backend , and more over if you try to access the backend directly bypassing the Application Gateway you will not see any issues related to certificate in the browser.

 

If your certificate is working on browser directly hitting the app and not with AppGW then what is the exact problem?

To Answer we need to understand what happens in any SSL/TLS negotiation. During SSL negotiation , Client sends "Client Hello" and Server Responds with "Server Hello" with its Certificate to the Client. Now Clients will check the Server certificate and confirm if the certificate is issued by Trusted root or not.

 

For the server certificate to be trusted we need the Root certificate in Trusted Root Cert Store , usually if you are having certs issued by Godaddy,Digicert,Vergion like Third party Vendors you don’t have to do anything because they are automatically trusted by your client/browser.

 

If your cert is issued by Internal Root CA , you would have export the root cert and import it the Trust Root Store in the Client.

 

 

This is the exact thing what we do when import .CER file in the HTTP Settings of the Application Gateway. In Azure docs, it is clearly documented that you don’t have to import Auth certificate in HTTP settings of the backend if your backend application has Global trusted certificate. You should do this only if the backend has cert which is issued by internal CA

 

I hope we are clear till now on why we import Authenticate cert in the HTTP settings of the AppGW and when we use the option "Use Well Known CA"

 

But the actual problem arises if you are using a Third party Cert or Internal CA Cert which has Intermediate CA and then Leaf certificate

 

Most of the orgs for security reasons use Root Cert----> Intermediate Cert ------> Leaf Cert , even Microsoft follows the same for bing , check the screenshot below

 

 

 

Now lets discuss what exactly is the confusion here if we have multiple Chain Cert

 

When you have single chain certificate , then there will be no confusion with appgw , if your root CA is Global trusted just select "Use Trusted Root CA" option in HTTPsettings

If you root CA is Internal CA , then import that Top root cert in .cer format and upload it in the HTTP settings

Below is what happens during SSL negotiation when you have single chain cert and root in the AppGW. Server will send its Certificate and because AppGW will already have its Root Cert, it verifies the backend server certificate and finds that it was issued by the Root cert which it is Trusting and they it starts connecting on HTTPs further for probing. 

 

 

But when we have multiple chain certificate and if your backend application/server sends only the leaf the certificate , AppGW will not be able to trust the cert up to the top level domain root. when the backend server cert hits AppGW after Server Hello , AppGW tries to check who issued the certificate and it finds that it was issued by Intermediate certificate but then it does not have information about Intermediate cert, like who issued the cert and what is the root certificate of that intermediate certificate. This causes SSL/TLS negoatiation failure and AppGW marks the backend as unhealthy because it is not able to initiate the probe. 

 

Now you may ask why it works when you browse the backend directly through browser. 

 

Most of the browsers are thick clients , so it may work in the new browsers but reverse proxies  like Application Gateway wont behave like our browsers they only trust the certificates if the backend sends the complete chain.

 

here is what happens in in Multiple chain certificate. 

 

 

AppGW is a PaaS instance , by default you wont get access to the Applicaiton Gateway. Now how do we find if my application/backendserver is sending the complete chain to AppGW?

 

You can find this by running openssl from either windows client or Linux client which is present in the same network/subnet of the backend application.

 

We should get one Linux machine which is in the same subnet/VNET of the backend application and run the following commands. what we are doing is actually trying to simulate the Linux box as AppGW as if that machine is trying probe to the backend server as AppGW. 

 

here is the sample command you need to run, from the machine that can connect to the backend server/application. 

 

OpenSSL  s_client -connect 10.0.0.4:443 -servername www.example.com -showcerts

 

If the output doesn't show the complete chain of the certificate being returned, export the certificate again with the complete chain, including the root certificate. Configure that certificate on your backend server.

 

For details on this Openssl command you can refer to  Troubleshoot backend health issues in Azure Application Gateway | Microsoft Docs  , Look for the sub topic "Trusted root certificate mismatch"