How to – Advanced Properties of Linked Services

Author(s): Sean Mirabile is a Program Manager in Azure Synapse Customer Success Engineering (CSE) team. 

Summary

Azure Data Factory and Synapse Analytics Pipelines have a wealth of linked service connection types that allow them to connect and interact with many services and data stores.  The Workspace UI provides the most important properties that are needed for the connection. However, at times we need more control that the UI doesn’t offer.

In this tutorial, we will focus on customer scenario to see how you can find all the available properties for the Linked Service you are working with, then how to incorporate that property into your Linked Service directly from within the UI. 

Scenario

Question: Do we have an option to select the cluster policy while creating the Databricks linked service? 

Answer: Yes, by leveraging the Advanced tab within any Linked service, you can see all the remaining properties that the service exposes in JSON format. The visual experience is working to add first-class support for these properties. You can add/edit/delete these properties in JSON via the Advanced Editor.

How To

Find Available Properties accessible through the Advanced editor
A great way to find out the available properties that the service exposes for Linked Services is to reference the Azure SDK for .Net class for LinkedService.

The Derived section has a list of all the available Linked Service connections. Select More.. to see the full list. Then select the Linked Service of interest.  

Example 

Environment

• Azure Databricks Workspace with an Azure Databricks Cluster Policy named 'ver' with the ID = D062DA0F37000397 specifying spark_version as 7.3.x-scala2.12
  • Notebook with sample code
• Azure Data Factory with an existing Azure Databricks Linked Service named 'AzureDatabricks1'
  • New Job Cluster: Cluster Version 10.4x-scala2.12 
  • Pipeline with Databricks Notebook Activity

Note: To follow along and get started, you can follow the steps in this article on how to create the linked service, a sample notebook, and a pipeline; then follow the steps detailed in the example below.  

Finding Available Properties

We want to see if we can specify the cluster policy while creating the Databricks Linked Service. Let's navigate to the Azure Databricks Linked Service  .Net class to see the available properties (typeProperties). 

Under the Properties section, there is an option for policyId, which we can reference in the Advanced Editor of our Azure Databricks Linked Service.  

How to modify the Linked Service to include the property and value

Within Azure Data Factory, from within the Linked services panel, select to view the Azure Databricks Linked Service’s code view

Modify the JSON object by adding the property: policyId within typeProperties. Select Apply.

Note: the cluster policy is enforcing the spark_version equal to 7.3.x-scala2.12. However, our Linked Service has a type property, newClusterVersion that differs. This is intentional to test the Linked Service is utilizing the new property. 

This change is reflected within the UI. Open your Linked Service and expand the Advanced dropdown. 

Testing the Linked Service with new Type Property

In this example, when we run the pipeline we are expecting the Notebook activity to fail because we are specifying a policy that requires the Linked Service to only use a specific spark version and our existing Linked Service violates this policy.  

After I ran the Pipeline, the error details shows a cluster validation error.

How to edit a Linked Service with Advanced options specified

If you want to edit any of the properties the UI displays, then you will need to either make the change within the code view or edit the JSON object within the Advanced section.  

For this example, to fix the Linked Service, navigate back to the Linked Service code view and change the property: newClusterVersion, to match our Cluster Policy, then Publish the change. 

To confirm, after running the Pipeline successfully, from the Azure Databricks Notebook activity's output, navigate to Notebook Run Page Url -->View Cluster --> Configuration, the Policy specified is the same policyId that the Linked Service specified using the Advanced editor to include the type property, policyId. 

Conclusion

Though we focused on a single scenario, this concept can be expanded across all Linked Service Connectors. To find all of the available Type Properties for your desired connector, please remember to reference the Derived section of the Azure SDK for .Net class for LinkedService.

Bonus Content: Make the Linked Service Dynamic
A Linked Service can have parameters, which allow you to make any of these properties dynamic. Instead of having one Linked Service per policyId, I might want a single Linked Service that has a parameter for the property, or any other of the available properties.

Edit the Linked Service and add a new Parameter with the name policyID. 
We can use @linkedService().<ParameterName> as the value of the item we want to make dynamic. In this case, update the Type Property: policyID from the Advanced Editor to reference the Parameter.  

Note: All of the objects that reference the Linked Service will be updated and now include the parameterizable Linked service properties that are defined. 

Our team publishes blog(s) regularly and you can find all these blogs here: https://aka.ms/synapsecseblog  

For deeper level understanding of Synapse implementation best practices, please refer our Success By Design (SBD) site: https://aka.ms/Synapse-Success-By-Design