Combatting Risky Sign-ins in Azure Active Directory

It is almost inevitable your organization will be targeted with malicious sign-in attempts to cloud apps. It is often the case an employee uses the same password for their work account as they do for their personal accounts. Password leaks from organizations other than yours pose a threat if your employees are using the same password for the same apps, even if their email/UPN is different. As more apps are moving to the cloud it becomes increasingly more imperative that these malicious sign-in attempts don’t go unnoticed and correct action is taken against them. You can check here for Azure identity and access best practices. 

 

Understanding Risk Levels  

 

Azure AD Identity Protection is a service built-in to Azure AD for organizations using Azure AD P2 licenses. For those without the Azure AD P2 license Azure AD Identity Protection works with limited capabilities. Azure AD Identity Protection can detect risks such as anonymous IP address use, atypical travel, malware linked IP address, unfamiliar sign-in properties, leaked credentials, password spray, and more.  

For each risky sign-in Identity Protection assigns a risk level; low, medium, or high. The higher the risk level the higher the confidence Identity Protection has that this user or sign-in is compromised. There are four different reports Identity Protection generates: 

 

• Risky users – users at risk, risk history of users.  
• Risky workload identities – risk levels of service principals.  
• Risky sign-ins – sign-in aggregate risk levels, sign-in information (device, application, location, etc.), detection type.  
• Risk detections – risk detections over the last 90 days with detection type and other details.  

These reports can be found by going to the Security tab in Azure Active Directory. Figure 1 shows the Risky sign-ins report. Clicking on one of the sign-ins will give more details about the risk level and why the sign-in was determined risky. In Figure 2 the sign-in has a risk level of Low and was categorized as risky because of the use of an anonymous IP address. 

 

Figure 1: Risky sign-ins report

 

Figure 2: Details of risky sign-in

 

The Risky users report shows the users which have been categorized with a risk level. Figure 3 shows the Risky users report and Figure 4 shows the risky sign-ins associated with the user. Once a user is clicked on in the Risky users report the user password can be reset, user can be marked as compromised, user can be blocked, and user risk can be dismissed. Marking a user as compromised will move it’s risk level to High and Azure AD will treat it as such. This is especially useful if you have any automation or Conditional Access Policies which use user risk level as a condition. Conversely, dismissing user risk will remove any risk level associated with the user. Both actions cannot be reversed. 

 

Figure 3: Risky users report

 

 

Figure 4: Risky sign-ins associate with user

  

The risk levels that Identity Protection sets on users and sign-ins can be used to take automated actions. One way to do this is to create Conditional Access Policies with conditions based on user risk and sign-in risk. For example, you can force a password reset when the user signing in has a user risk level of High. You can read more recommendations from Microsoft about Conditional Access Policies based on risk levels here.  

  

Investigating Risky Sign-ins

  

When investigating risky sign-ins there are certain attributes to look out for to determine the true risk level. It is crucial to understand if the sign-in has any unusual attributes based on previous sign-ins from that user. Looking at previous sign-ins you can determine if the sign-in has similar values for the application being signed into, location, device, IP address, and user agent. If the device is Azure AD joined or Azure AD hybrid joined , you will be able to see the specific device used. You should ask yourself questions such as: 

 

• Has this user signed in from this location and IP before?  
• Has the user signed into this application before?  
• Is the device registered in Azure AD?

• Is the device compliant? 

 

Investigating the details of the sign-in will give you a clearer indication regarding the true risk level of the sign-in, and thus what action to take. Imagine two different successful sign-ins from the same user based in London, England. Both sign-ins have been identified as Low risk due to the use of Anonymous IP addresses. 

 

Attribute	Scenario 1	Scenario 2
Application	Outlook Mobile	Azure Portal
Device	iPhone	Windows 10
Location	Madrid, Spain	Connecticut, United States
User Agent	Mozilla/5.0 (iPhone; CPU iPhone OS 16_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148	Mozilla/5.0 (Windows NT 10.0; Microsoft Windows 10.0.9200; en-US) PowerShell/7.1.0
Azure AD status	Azure AD registered	Not joined
Device Compliance	Compliant	N/A

  

Starting with Scenario 1 above we can see that the user signed into Outlook Mobile from an iPhone. The user agent gives us further information on the operating system version. There are many online tools to parse user agents into a more readable format. The device is also Azure AD registered and compliant. The use of an anonymous IP address was probably because the user was using a VPN on their mobile when the sign-in to Outlook Mobile happened. This is most likely not a compromised sign-in but the user would still need to be reached out to confirm. 

 

Scenario 2 paints a different picture. This time the sign-in happens to the Azure Portal. Unless administration activities in the cloud are part of the users job, it is suspicious to see sign-ins to the Azure Portal. The device is also not joined to Azure AD. The user agent shows the sign-in happened using PowerShell. Further investigation would need to be done here before determining the true risk level of this sign-in, and if this user is really compromised. In the event that the user has been compromised then the users sessions should be revoked and password reset. If the threat actor has changed any MFA configurations (phone number, app registration, secondary email, etc.) then make sure that these have been removed and the user themselves re-enrolls to MFA. 

  

Monitoring Risky Sign-ins

 

If you collect Azure AD sign-in logs into a Log Analytics workspace or Microsoft Sentinel then you can query the logs to aid your investigations and hunt for threats. The SigninLogs table in Sentinel stores data regarding risk levels of sign-ins. The following query can be run to see which risk detection types were the most common: 

 

 

SigninLogs  | where RiskLevelDuringSignIn != "none"  | mv-expand todynamic(RiskEventTypes)  | summarize ["Type Count"] = count() by tostring(RiskEventTypes)  | sort by ['Type Count'] desc  | render columnchart with(title="Count of Risk Detection Type") 

 

 

To categorize the risky sign-ins based on level and if the sign-in was successful you can run the following query: 

 

 

SigninLogs  | where RiskLevelDuringSignIn != "none"  | extend ResultType = iif(ResultType == "0", "Successfull Logon", "Failed Logon")  | summarize Count = count() by ResultType, RiskLevelDuringSignIn  | render columnchart with(title = "Risk Levels Count per Successfull and Failed Sign-ins") 

 

 

Your organization will face malicious sign-in attempts. Utilizing Azure AD Identity Protection can help you combat these sign-ins by assigning risk-levels to sign-ins and users. You can then automate remediation actions and enhance your investigations with the additional risk details captured by Identity Protection. 

 

Learn more about Microsoft identity: 

• Get to know Microsoft Entra – a comprehensive identity and access product family 
• Return to the Microsoft Entra (Azure AD) blog home  
• Join the conversation on Twitter and LinkedIn 
• Share product suggestions on the Entra (Azure AD) forum