Monitoring traffic flows in Azure Firewall using Virtual Network Flow Logs

Azure Firewall is a managed service designed to protect your Azure Virtual Network resources, providing advanced threat protection and advanced logs and metrics that are essential tools for monitoring and managing your network security. By leveraging both logs and metrics, you can ensure the overall health and efficiency of your firewall, maintain an audit trail of configuration changes, and comply with security and auditing requirements.

 

In this blog post we will show you a different approach to enhance the monitoring experience of Azure Firewall by using Virtual Network Flow Logs and Traffic Analytics. This combination provides a comprehensive view of traffic flows within your network, offering deeper insights for analysis and investigation, helping to identify traffic deviation that may indicate a security issue and identify applications that are consuming Azure Firewall the most.

 

What is Virtual Network Flow Logs and Traffic Analytics?

Both Virtual Network Flow logs and Traffic Analytics are features of Azure Network Watcher that collects information about network traffic and enriches raw flow logs to provide insights into network traffic patterns, including source and destination IP addresses ports protocols and the volume of traffic. To learn more about both features, check out the product documentation.

 

Improving Azure Firewall Monitoring

Azure Firewall has a powerful set of structured logs that provide detailed insights into the traffic and operations of your firewall. These logs include information on application rules, network rules, IDPS and threat intelligence, allowing you to monitor and analyze the firewall’s performance and security posture.

 

To gain even further insights, enabling Virtual Network Flow Logs at the subnet level of Azure Firewall provides a comprehensive view of all traffic passing through your Azure Firewall. This enhanced visibility allows you to identify top talkers, detect undesired traffic, and uncover potential security issues that may require further investigation and mitigation. By leveraging these insights, we can ensure a more secure and efficient network environment.

 

Follow the steps below to learn how to enable Virtual Network Flow Log.

1. Go to Network Watcher > Flow Logs (under Logs) and create a new flow log using the type “Virtual Network”
2. Select the “AzureFirewallSubnet” as your target resource and fill out the remaining of the required fields, such as Location, Storage Account and Retention Days
3. In the Analytics tab, don’t forget to enable traffic analytics, selecting your Log Analytics Workspace and defining the processing interval (Every 1 hour or 10 minutes). Note: 1h is recommended for most scenarios, since 10 minutes processing has a significant cost impact.

 

This is what your configuration will look like once the new flow log is created:

 

Now, give it some time for the Virtual Network Flow Log to start collecting the traffic from the AzureFirewallSubnet and for Traffic Analytics to process the data collected. Once you have data in the table “NTANetAnalytics” it’s time to start using your KQL skills to learn more about the traffic going through your Azure Firewall. You can also use the KQL query below as an example:

 

NTANetAnalytics | where TimeGenerated > ago(8h) | extend TotalBytes = BytesDestToSrc + BytesSrcToDest | project TimeGenerated, SubType, FlowType, SrcVm,SrcIp, DestVm,DestIp, DestPublicIps, DestPort, BytesDestToSrc, BytesSrcToDest, TotalBytes

 

In the example above we are seeing the “raw” data and depending on the amount of traffic it may not be very helpful if you are trying to identify what workloads are consuming your Azure Firewall the most. For identifying what IP addresses are sending data through Azure Firewall the most, you can use the query below with the specific time frame you want to investigate. Here you will also find logs from the underneath Azure Firewall instances. Remember that we have enabled the Flow Logs for the entire AzureFirewallSubnet, so it is expected to see logs of the data sent by each one of the active instances.

 

NTANetAnalytics | where TimeGenerated > ago(8h) | where SrcIp == "10.10.0.132" or SrcIp == "10.10.0.4" or SrcIp == "10.10.0.5" | summarize TotalBytes=sum(BytesDestToSrc+BytesSrcToDest) by SrcIp, DestIp | render columnchart

 

In the dashboard we are seeing the IP address 10.10.0.5 transferring around 55GB to the IP 10.10.0.132 and another 55GB to the 10.10.0.4. One thing to take into consideration is that Azure Firewall as any other NVA is receiving the data from the source and sending it to the destination, so the numbers displayed are approximately the double of the data transferred between the VMs (27.5GB to 10.10.0.4 and 27.5GB to 10.10.0.132). Let’s say this was not an expected volume of data transferred, possibly leading to data exfiltration or some other technical issue caused by a user or an application. So, just by using this simple KQL query we can identify some deviation and start a deeper investigation to identify the root cause of that. One other example where this same KQL query can be useful is when you need to identify what are the workloads consuming Azure Firewall within a time frame to split the costs of Azure Firewall between different LOBs.

 

In the previous examples we have used the source IP addresses (SrcIp), but you can also use the name of the VMs (SrcVm) if it is easier to identify the workloads running on each virtual machine.

 

NTANetAnalytics | where TimeGenerated > ago(3d) | summarize TotalBytes=sum(BytesDestToSrc + BytesSrcToDest) by SrcVm | top 10 by TotalBytes | render columnchart

 

The query above is an example of a KQL query to find the Top10 VMs based on the total data transferred in the last 3 days. Note that the TotalBytes matches with the example above where the IP 10.10.0.5 had transferred 55GB to the 10.10.0.4 and another 55GB to the 10.10.0.132.

 

The use of Network Watcher features like Virtual Network Flow Logs and Traffic Analytics offers you a great experience monitoring all your networks in Azure. The goal in this blog is to show you a different approach to enhance the monitoring of all the traffic going through your Azure Firewall and we have only shown you how to use the KQL queries to gain a deeper view that traffic. You should also consider exploring all the other views available at Network Watcher > Traffic Analytics, such as the Traffic Analytics Workbook as seen in the picture below.

 

Conclusion

Using Virtual Network Flow Logs and Traffic Analytics to monitor the traffic going through Azure Firewall is a game-changer for network security and performance management. These tools provide invaluable insights into traffic behavior, enabling you to identify patterns and deviations that could indicate potential security compromises. By leveraging these capabilities, you can proactively address threats before they escalate, ensuring your network remains secure. Moreover, Virtual Network Flow Logs and Traffic Analytics help you pinpoint the workloads that demand the most from your Azure Firewall. This information is crucial for optimizing resource allocation and enhancing overall network efficiency. Understanding which workloads are the most demanding allows you to make informed decisions about scaling and resource management, ultimately leading to a more robust and resilient network infrastructure.

 

Learn More

• What is Azure Firewall? | Microsoft Learn
• Overview of Azure Firewall logs and metrics | Microsoft Learn
• Azure Structured Firewall Logs | Microsoft Learn
• Azure Network Watcher overview | Microsoft Learn
• Virtual network flow logs - Azure Network Watcher | Microsoft Learn
• Traffic analytics - Azure Network Watcher | Microsoft Learn
• Network traffic observability with virtual network flow logs - Microsoft Community Hub