Loading...

Experiencing Data Latency for Log Analytics - 06/20 - Mitigating

Experiencing Data Latency for Log Analytics - 06/20 - Mitigating
Update: Thursday, 23 June 2022 07:12 UTC

Root cause has been isolated to high load on backend component which was impacting Log Analytics. To address this issue, we have rolled out configuration fix, Paused and restarted the responsible backend services along with scaling out backend services to catch up with latency to mitigate the issue. We have applied a series of mitigation steps and we are continuing to monitor the service health to validate mitigation in West Europe. Additional engineering teams are being engaged to verify if additional mitigation workstreams are needed and issue is mitigated for UK South. The next update will be provided in 6 hours or as events warrant. Some customers may continue to experience issues with data access, missed or delayed log search alerts, data gaps and data latency and issues with Azure Sentinel.

Due to the impact to Log Analytics, customers using Azure Automation to delete/link automation accounts to workspaces will be impacted. Customers using Azure Automation for update management solution may see their machines in a 'Not Assessed' state and automation account not linked to Log Analytics. Customers using automation for change tracking solution may not see their data updated in Log Analytics Workspace.

As our mitigation steps were successfully applied and backend services returning healthy. We are currently in the process of catching up with ingestion backlogs.

We have completed the recovery for most data types. We expect to finish recovering for some of the remaining data types by 12:30 UTC on 23 Jun 2022.

  • Work Around: None
  • Next Update: Before 06/23 19:30 UTC
-Anmol

Update: Wednesday, 22 June 2022 22:47 UTC

Root cause has been isolated to high load on backend component which was impacting Log Analytics. To address this issue, we have rolled out configuration fix, Paused and restarted the responsible backend services along with scaling out backend services to catch up with latency to mitigate the issue. We have applied a series of mitigation steps and we are continuing to monitor the service health to validate mitigation in West Europe. Additional engineering teams are being engaged to verify if additional mitigation workstreams are needed and issue is mitigated for UK South. The next update will be provided in 6 hours or as events warrant. Some customers may continue to experience issues with data access, missed or delayed log search alerts, data gaps and data latency and issues with Azure Sentinel.

Due to the impact to Log Analytics, customers using Azure Automation to delete/link automation accounts to workspaces will be impacted. Customers using Azure Automation for update management solution may see their machines in a 'Not Assessed' state and automation account not linked to Log Analytics. Customers using automation for change tracking solution may not see their data updated in Log Analytics Workspace.

As our mitigation steps were successfully applied and backend services returning healthy. We are currently in the process of catching up with ingestion backlogs

  • Work Around:
  • Next Update: Before 06/23 05:00 UTC
-Arish Balasubramani

Update: Wednesday, 22 June 2022 11:34 UTC

Root cause has been isolated to high load on backend component which was impacting Log Analytics. To address this issue, we have rolled out configuration fix, Paused and restarted the responsible backend services along with scaling out backend services to catch up with latency to mitigate the issue. We have applied a series of mitigation steps and we are continuing to monitor the service health to validate mitigation in West Europe. Additional engineering teams are being engaged to verify if additional mitigation workstreams are needed and issue is mitigated for UK South. The next update will be provided in 6 hours or as events warrant. Some customers may continue to experience issues with data access, missed or delayed log search alerts, data gaps and data latency and issues with Azure Sentinel.

Due to the impact to Log Analytics, customers using Azure Automation to delete/link automation accounts to workspaces will be impacted. Customers using Azure Automation for update management solution may see their machines in a 'Not Assessed' state and automation account not linked to Log Analytics. Customers using automation for change tracking solution may not see their data updated in Log Analytics Workspace.
  • Work Around: None
  • Next Update: Before 06/22 18:00 UTC
-Sai Kumar

Update: Wednesday, 22 June 2022 06:29 UTC

Root cause has been isolated to high load on backend component which was impacting Log Analytics. To address this issue, we have rolled out configuration fix, Paused and restarted the responsible backend services along with scaling out backend services to catch up with latency to mitigate the issue. We are currently monitoring service health to confirm mitigation as we see signs of recovery for West Europe. And mitigation steps are in progress for UK South. The next update will be provided in 5 hours or as events warrant. Some customers may continue to experience issues with data access, missed or delayed log search alerts, data gaps and data latency and issues with Azure Sentinel.
  • Work Around: None
  • Next Update: Before 06/22 11:30 UTC
-Deepika

Update: Wednesday, 22 June 2022 02:55 UTC

Root cause has been isolated to high load on backend component which was impacting Log Analytics. To address this issue, we have rolled out configuration fix, Paused and restarted the responsible backend services along with scaling out backend services to catch up with latency to mitigate the issue. We are currently monitoring service health to confirm mitigation as we see signs of recovery for West Europe. And mitigation steps are in progress for UK South. The next update will be provided in 3 hours or as events warrant. Some customers may continue to experience issues with data access, missed or delayed log search alerts, data gaps and data latency and issues with Azure Sentinel.
  • Work Around: NA
  • Next Update: Before 06/22 06:00 UTC
-Arish Balasubramani

Update: Tuesday, 21 June 2022 21:56 UTC

Root cause has been isolated to high load on backend component which was impacting Log Analytics. To address this issue, we are rolling out the fix that help with the load, paused a responsible backend service from creating more load in order to mitigate the issue. The next update will be provided in 3 hours or as events warrant. Some customers may continue to experience issues with data access, missed or delayed log search alerts, data gaps and data latency and issues with Azure Sentinel.
  • Work Around: NA
  • Next Update: Before 06/22 02:00 UTC
-Arish Balasubramani

Update: Tuesday, 21 June 2022 15:10 UTC

Root cause has been isolated to high load on backend component which was impacting Log Analytics. To address this issue we are rolling out the fix that help with the load in order to mitigate the issue. The next update will be provided in 3 hours or as events warrant. Some customers may continue to experience issues with data access, missed or delayed log search alerts, data gaps and data latency and issues with Azure Sentinel.
  • Work Around: NA
  • Next Update: Before 06/21 18:30 UTC
-Deepika

Update: Tuesday, 21 June 2022 09:02 UTC

We continue to investigate issues within Log Analytics. Root cause is not fully understood at this time. Some customers may continue to experience issues with data access, missed or delayed log search alerts, data gaps and data latency and issues with Azure Sentinel. Our initial findings indicate that the problem began at 06/20 16:00 UTC. We currently have no estimate for resolution, Adding Cores Capacity to WEU allowing the service to scale out did not help.
  • Work Around: NA
  • Next Update: Before 06/21 14:30 UTC
-Deepika

Update: Tuesday, 21 June 2022 05:25 UTC

We continue to investigate issues within Log Analytics. Root cause is not fully understood at this time. Some customers may continue to experience issues with data access, missed or delayed log search alerts, data gaps and data latency and issues with Azure Sentinel. Our initial findings indicate that the problem began at 06/20 16:00 UTC. We currently have no estimate for resolution, Adding Cores Capacity to WEU allowing the service to scale out did not help.
  • Work Around: NA
  • Next Update: Before 06/21 08:30 UTC
-Deepika

Update: Monday, 20 June 2022 20:40 UTC

We continue to investigate issues within Log Analytics. Root cause is not fully understood at this time. Some customers may continue to experience issues with data access, missed or delayed log search alerts, data gaps and data latency and issues with Azure Sentinel. Our initial findings indicate that the problem began at 06/20 17:15 PM UTC. We currently have no estimate for resolution, Adding Cores Capacity to WEU allowing the service to scale out did not help.
  • Work Around: NA
  • Next Update: Before 06/20 23:00 UTC
-Arish Balasubramani

Published on:

Learn more
Azure Monitor Status articles
Azure Monitor Status articles

Azure Monitor Status articles

Share post:

Related posts

Azure Monitor Status Blog discontinuation notice

As part of our efforts to improve the Azure Monitor service experience and consolidate it with other Azure-wide services, from now on, we’ll p...

4 years ago

Name update for Agent and associated resources in Azure Monitor - Container Insights

What is changing?   When customers enable Container Insights, Azure Monitor deploys a containerized collection agent. This agent is ...

4 years ago

Name update for Agent and associated resources in Azure Monitor - Container Insights

What is changing?   When customers enable Container Insights, Azure Monitor deploys a containerized collection agent. This agent is ...

4 years ago

Experiencing Data Latency for Azure Log Analytics - 07/15 - Resolved

Final Update: Friday, 15 July 2022 09:07 UTCWe've confirmed that all systems are back to normal with no customer impact as of 7/15, 09:00 UTC....

4 years ago

Experiencing Data Latency for Azure Log Analytics - 07/15 - Resolved

Final Update: Friday, 15 July 2022 09:07 UTCWe've confirmed that all systems are back to normal with no customer impact as of 7/15, 09:00 UTC....

4 years ago

Experiencing Data Latency for Azure Log Analytics - 07/15 - Investigating

Update: Friday, 15 July 2022 00:49 UTCWe continue to investigate issues within Log Analytics. Root cause is not fully understood at this time....

4 years ago

Experiencing Data Access Issue in Azure portal for Log Analytics - 07/13 - Resolved

Final Update: Wednesday, 13 July 2022 13:00 UTCWe've confirmed that all systems are back to normal with no customer impact as of 7/13, 12:30 U...

4 years ago

Experiencing Data Access Issue in Azure portal for Log Analytics - 07/13 - Investigating

Initial Update: Wednesday, 13 July 2022 12:07 UTCWe are aware of issues within Log Analytics and are actively investigating. Some custome...

4 years ago

Experiencing Data Access Issue in Azure portal for Log Analytics - 07/13 - Resolved

Final Update: Wednesday, 13 July 2022 07:07 UTCWe've confirmed that all systems are back to normal with no customer impact as of 7/13, 05:20 U...

4 years ago

Experiencing Data Access Issue in Azure portal for Log Analytics - 07/13 - Resolved

Final Update: Wednesday, 13 July 2022 04:14 UTCWe've confirmed that all systems are back to normal with no customer impact as of 07/13 02:35 U...

4 years ago

Newsletter

Get the latest Dynamics 365 and Power Platform content in your inbox

A curated digest of community blogs, product news, videos, and podcasts — delivered without the noise.

Weekly updates Unsubscribe anytime Fresh community picks
We use your email only for the newsletter and you can unsubscribe at any time.
By subscribing, you agree to the privacy policy.