Exclusive News : Dynamics 365 Sales Duplicate Lead Detection with !! MERGE !!

Dynamics 365 Sales Duplicate Lead Detection with !! MERGE !!:

EXCLUSIVE NEWS : This article is talking about the new feature of Dynamics 365 Sales Module : Manage duplicate leads that is on preview at the time of writing this.
BUT, Thanks to a close relationship that the Dynamics Chronicles community has with the Microsoft product teams, we were able to obtain even more exclusive information which is not yet published on any preview. You will discover the MERGE functionality of this new Duplicate Leads feature. 
We warmly thank Microsoft for this privilege !!

Have you ever faced difficulties to maintain data clean within Dataverse !?

I am pretty sure you did ! Though, if you didn't, let's imagine.

We are working for a famous watch manufacturer and they are using Dynamics 365 to centralize data and manage their customers/potential customers. A Lead can be created through different source : for instance, within a Boutique by a salesperson or directly by the customer itself on the website by showing his interest on a specific product.

Within this very website, a customer might log-in and creates a request by clicking on 'I want more information about this watch'. As he is logged-in it will only result to the creation of a Lead in the Dataverse with an existing Contact that populate Contact lookup. What if this customer click again on this 'I want more information about this watch' without logging in the website !? It would result in the creation of a new Contact and a new Lead that actually already existed in the system.

This duplicate Lead might have a slightly different name because of misspelling and a bunch of fields value different from the original one. Which means that we are not sure to identify this duplicate as this very Lead may have a bunch of different information from the original.

My example sounds extreme to you !?

Well, that's pretty much what happened to me in a project and you can multiply this by thousands of Lead. As you can imagine those thousands duplicates are not easy to handle and raise plenty of complex question.

Indeed, in that situation, we had to develop a logic in order to solve this issue. We started wondering : Which one to keep and according to which criteria ? Within this record that we keep, which value to keep and according to which criteria for each field ? After implementing the logic, how to make it retroactive ? How to test it properly in order to don't take any risk of loosing data that might certainly be strategic for the client ? Finally, what to do with those Duplicate, delete them or deactivate them ? 

As you see, it is extremely time consuming and pretty complex to prevent those duplicates to appear in Dataverse. Although data hygiene is super strategic from business perspective so let's focus on how to prevent tones of duplicate distort reports and to pollute the Dataverse !

Microsoft is about to release a new feature called 'Manage duplicate lead' that I am curious to explore through this article. This is available on Preview and Microsoft does the honor to us to give an access to the newly developed Merge feature within 'Manage duplicate lead' that is not in preview yet. 

We will first see what is this new feature and how it could help us to maintain data clean. Then we will see how it looks in practice with a specific part dedicated to Merge feature. Finally we will conclude and see whether or not this new feature is a good answer to data hygiene challenges.

Don't forget to click on the static image if you want to see an animated GIF that will demonstrate the features. 

I - Manage duplicate leads

This new feature was released in order to help Dynamics 365 users to maintain data clean. It allows us to identify easily duplicate leads in order to remove them.

The rules applied in order to identify the duplicate leads are settled and can not be changed. Here are the rules :

• Same email address
  • if there is a same email address in two different Lead, then those leads are identified as duplicates
• Same phone number
  • if there is a same phone number in two different Lead, then those leads are identified as duplicates
• Same name AND company name
  • if there is a same Full Name AND a same Company Name, then those leads are identified as duplicates
  • interesting enough, it takes in consideration the misspelling cases. Maxime Bonnot from Elca Informatique and Maxime Bonnote from Elca Informatique must be identified as duplicates.
• Same name AND same email domain
  • if there is a same Full Name AND a same email domain within the email address, then those leads are identified as duplicates
  • for instance, Maxime BonnOT with [email protected] as email address and Maxime BonnET with [email protected] will be identified as duplicates

Out of those rules, we can surely be optimistic regarding the enhancement of duplicate detection for Leads. Let's test it and see if this is as promising in reality as in the Microsoft documentation.

II - In Practice

First we will see how to enable this feature only available in preview. Then we will experience it.

1 - Enable the feature

First, enable the feature in Preview.

Go in Sales Hub within your tenant > switch Area from Sales to App Settings > Select Duplicate Lead Detection on the left menu > enable the Feature in Preview (As Below) :

Now let's try.

2 - Duplicate Lead

I am doing this demo with Sample Data provided within a demo environment. I will create a duplicate Lead in Sales Hub and make different test in order to see how good a duplicate can be found.

Let's test every rule listed in part I of this article.

a - Same email address

I created a new Lead with my name and picked an Email Address in one of the Lead already existing in Sample Data. Then I selected my new Lead in the view and clicked on 'View Duplicate' in the top menu bar :

As you can see, it shows you the record newly created and its duplicates according to the Email Address in that particular case. 

We can manually compare the fields and decide to delete one or the other. Though there is no merge feature allowing us to choose what field to keep in what Lead. We only have the possibility to Delete the duplicate.

b - Same Phone Number

I created a new Lead with my name and picked a Phone Number in one of the Lead already existing in Sample Data. Then I selected my new Lead in the view and clicked on 'View Duplicate' in the top menu bar :

As you can see, as the Phone Number was the same, a duplicate was found.

c - Same Name and Company Name (including misspelling)

I created a new Lead with the same Name than another existing Lead but with a misspelling in the Last Name. I also picked a Company Name in an existing Lead. Then I selected my new Lead in the view and clicked on 'View Duplicate' in the top menu bar :

As you can see, there is a real improvement on detection of similar records with misspelling.

For the duplicate detection rule, Maria Campbell and Maria Cambell (without P) are similar. And so it is for the company name Fabrikam. Inc. (Sample) and Fabrikam Inc. 

Definitely an interesting enhancement that will give us more confidence towards Dynamics 365 to detect duplicate considering misspelling.

d - Same Name and Email Domain

I created a new Lead with the same name than another existing Lead but with a misspelling in the Last Name. I also picked an Email Address domain in the existing Lead. Then I selected my new Lead in the view and clicked on 'View Duplicate' in the top menu bar :

Here too, for the duplicate detection rule, Nancy Anderson (sample) and Nancy Enderson with the same domain @example.com are considered as similar.

As you see, we have the ability to find easily duplicates considering misspelling on names. We can therefore decide which one is the original and delete duplicate. Although there is no way we can Merge those records and decide what values to keep within those duplicates.

Fortunately, this is something that Microsoft has consider and we have the honor to have some views on how it will look like with Merge in Duplicate Lead window.

Let's see how it looks !

III - EXCLUSIVITY : Merge feature in Duplicate Lead window

As mentioned previously, Microsoft is currently developing the merge feature and they nicely provided some screenshot giving us an overview on how this very feature will look like.

It would be just as in part II but including this time additionally to the Delete button, a Not Duplicate button and a Merge button.

After selecting one Lead, the button 'View Duplicate' appears in command bar if duplicates exist. After clicking on it, the Duplicate Lead Detection window appear as below :

As you can see in the screen shot, after selecting two records in the Duplicate Lead Detection window, two additional button appear :

• Not Duplicate : which would allow us to signal to the system that those are actually not duplicates
• Merge : which would allow us to merge those records by selecting what values to keep from the duplicate to the main record.

Let's now see how it looks if we click on Merge after selecting those two records :

In addition of Delete button that we saw in part II, we have two checkbox appearing :

• Merge records by choosing fields with data : with this option, there is no need to select what field value to keep manually, if the field value is populated on all records, then it will keep the value of the Primary record. If it is only populated in one record, it will keep this value only. 
• View fields with conflicting data : it hides all fields with the exact same data on records and ease the selection by considering only fields with different data on duplicates

We can also ignore this two checkbox and simply select which of the duplicates is the Primary record. Then we can select one by one field which value to keep from which record in order to keep all selected values in one unique record and delete useless duplicates.

Let's see how it looks :

After selecting the Primary record, this very record appears on the left. Then we have the chance to select for each field, what value to keep from which record.

This Merge feature within Duplicate Lead Detection window is without doubt an huge improvement that might ease the user experience and the data hygiene by giving the capability to each user to detect easily and clean data with Merge feature.

Conclusion

As we all know, maintain a good data hygiene in a system is not an easy task. It gets even worse if you multiply integration with websites or other systems like ERP. 

Really quickly you end up having a huge amount of duplicates and there come the question : how do we manage this ? 

Maintaining data clean became increasingly strategic for organizations. It involves data reports in order to make quick and efficient decision out of what we analyze through data but also better performance with employees able to centralize accurately information about a specific customer in one unique record. 

With Duplicate Lead Detection, Microsoft propose a new feature helping end users to identify duplicates. As it also takes misspelling in consideration, that's a considerable improvement in order to make sure that the Lead I am dealing with is unique. We can ensure that all the information are centralized and if it is not the case, pick up manually necessary information before deleting the duplicate record.

Moreover, after identifying duplicates, end users will be able to merge those duplicates and select for each field what value to keep. This is a fantastic improvement that I am looking forward to use. Indeed detect a duplicate is one part of the job. The other part is to decide what value to keep in the primary record before deleting the duplicate. There is often important information which can be strategic to keep. Therefore, I am looking forward to see this feature operational in order to make our job easier.

Duplicate Lead Detection is definitely a big step towards good data hygiene. It is for now limited for Lead but I am confident that after some time, it will be extended to all entities.

Thank you to Microsoft Team who gave me the possibility to see how the final feature will look like with Merge included.

Dynamics 365 Sales Duplicate Lead Detection with !! MERGE !!