Custom DNS server issues while connecting from Databricks to Event Hub

Scenario:

You have an Event hub which receives events from Databricks.  Suddenly, you observe that after receiving few messages, you encounter below error:

AMQPConnectionError: Unable to open authentication session on connection b'EHProducer-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'.

Please confirm target hostname exists: b'xxxxxxx.servicebus.windows.net'

You start Wondering!  May be your Event Hub namespace doesn’t exist or someone has deleted accidentally.

You navigate to Azure portal and check Event Hub namespace exists and even you were continuously receiving messages in Event Hubs.

Then what could be the reason for the above error to be encountered?

Root Cause:

Connectivity Issue prevailed while connecting Data Bricks to Event Hub. This happens while you have a custom DNS Server in Databricks.

It is basically used in all PAAS services where Custom DNS Server has been used.

IP address 168.63.129.16 is a virtual public IP address that is used to facilitate a communication channel to Azure platform resources.

More about this Special IP address: -

This special public IP address is owned by Microsoft and will not change.

As mentioned in this article we recommend that you allow this IP address in any local (in the VM) firewall policies (outbound direction). The communication between this special IP address and the resources is safe because only the internal Azure platform can source a message from this IP address. If this address is blocked, unexpected behavior can occur in a variety of scenarios. 168.63.129.16 is a virtual IP of the host node and as such it is not subject to user defined routes.

Wire Server IP plays a important role in DHCP and DNS. This IP is also used for DNS resolution. Any Azure resources like Storage Account, Cosmos DB, Event Hubs or VM, VMSS and App Services will have a FQDN (Fully Qualified Domain Namespace). Each of these FQDN will be hosting the Domain names and A records at the backend server. These can be probed once you forward the request to the Wire Server IP 168.63.129.16

It acts as any cast IP in the Azure backbone. Custom DNS Server (10.2.0.4) doesn’t own any of the FQDN’s that is managed by Azure. So basically, any Azure IAAS or PAAS services will need this Wire Server IP to connect from Custom DNS server (10.2.0.4) to connect with any IAAS or PAAS services.

How to Mitigate this behavior-

So, in this scenario, Databricks, which is integrated with VNET had a Custom DNS Server. DNS server will forward the requests to Wire Server IP 168.63.129.16 to Azure PAAS resolution, in this case is Event hub from which Databricks is receiving the Events.

Steps needed to add 168.63.129.16 your DNS servers:

In the Azure Portal, please open the VNET where the Databricks service is injected in.

• Open the tab DNS Server.
• Add the IP 168.63.129.16 in the IP address (Add DNS server)
• Click on Save Button.