Connection References with ALM – Mind the Gap!

Published Thursday, April 1, 2021

If you are creating Cloud Flows in Solutions today, you are using Connection References. Although they are listed as ‘Preview’ – there really is not an alternative as when you create a new Cloud Flow – a connection reference is automatically created for you.

Connection References are essentially a ‘pointer’ to an actual connection. You include the Connection Reference in your solution so that it can be deployed, and then once imported, you can wire up the Connection Reference to point to a real connection. This means that you can deploy solutions with Flows that do not actually know the details of the connection, and without the need to edit the flows after deployment.

Here is how it all fits together:

The one thing that Connection References brings us is it avoids having to edit a flow after deployment to ‘fix’ connections. Previously, if you had 10 flows, then you would previously have to fix each of the flows. With Connection References, you only have to ‘fix’ the Connection References used by the flows.

You can find all the connection references that do not have an associated connection using the following query:

<fetch>
  <entity name="connectionreference" >
    <attribute name="connectorid" />
    <attribute name="connectionreferenceid" />
    <attribute name="statecode" />
    <attribute name="connectionid" />
    <filter>
      <condition attribute="connectionid" operator="null" />
    </filter>
  </entity>
</fetch>

When you import a solution with Connection Reference into a target environment using the new solution import experience, you will be prompted to link to an existing or create a new connection for any newly imported connection references. If they have previously been imported, then they are simply re-used.

However, we want to automate our deployments...

Editing Connection References and Turning on Flows using a Service Principle

So, what about in an ALM automatic deployment scenario?

At this time, however, importing a solution using a Service Principle in ALM (e.g. using the Power Platform Build Tools) leaves your flows turned off since the connection references are not linked to connections.

You can easily connect your connection references and then turn on a flow programmatically (see at the end of this post for the full PowerShell script):

# Set the connection on a connection reference:
Set-CrmRecord -conn $conn -EntityLogicalName connectionreference -Id $connectionreferenceid -Fields @{"connectionid" = $connectorid }

# Turn on a flow
Set-CrmRecordState -conn $conn -EntityLogicalName workflow -Id $flow.workflowid -StateCode Activated -StatusCode Activated

…but if you try to do this using a Service Principal, you will get an error similar to:

Flow client error returned with status code "BadRequest" and details "{"error":{"code":"BapListServicePlansFailed","message":"{\"error\":{\"code\":\"MissingUserDetails\",\"message\":\"The user details for tenant id … and principal id …' doesn't exist

Suggested Solution

My current approach to this (until we have official support in the Power Platform Build Tools) is something like this. Imagine the scenario where a feature branch introduced a new Flow, where there previously had been none – let us run through how this works with Connection References.

1. Adding a new Cloud Flow to the solution

When you add a new Cloud Flow to a Solution, the Connection References that it uses are also added automatically. If you are adding an existing Flow that was created in a different solution, you will need to remember to add the Connection References it uses manually.
Key Point: Connection References do not include any Connection details – they are only a placeholder that will point to an actual Connection via the connectionid attribute.

2. Solution Unpacked into a new branch

The solution is unpacked and committed to the feature branch.
The Feature branch eventually results in a Pull Request that is then merged into a release branch.
The Connection Reference shows up in the PR unpacked solution:
The new Flow also shows up in the pull request unpacked solution. Notice that the connection reference is referenced via the connectionReferenceLogicalName setting in the Flow json.

3. Build & Release

When the Pull Request is merged, the Build Pipeline will run automatically.
When the CI Build has run, the Flow will be packed up into the solution.zip – so you can then deploy it to your target environments.

4. Connection Reference - Set Connections

Once the release has completed – the solution will be deployed.
Key Point: At this stage, the Flow(s) are turned off because the Connection Reference is not yet wired up to a Connection.
Of course, if you were importing this solution by hand, you would be prompted to connect the unconnected connection references.

This is what the Flow and Connection Reference will look like in the solution explorer:

Note: Connection References always show the Status of 'Off' - even if they are wired to a connection!
The Owner of the Connection Reference and Flow is the Application User SPN that is used by the Power Platform Build Tools
If you try and turn on a flow that uses any connection other than the Current Environment Dataverse connector, you’ll get a message similar to:

Failed to activate component(s). Flow client error returned with status code "BadRequest" and details "{"error":{"code":"XrmConnectionReferenceMissingConnection","message":"Connection Id missing for connection reference logical name 's365_sharedoffice365_67cb4'."}}".

5. Turning on Flows

At this time there is no way of editing a connection reference from inside a managed solution, so you need to create a new solution and add the Managed Connection References to them.
Once inside a new solution, you can edit the Connection References and create a new or select an existing connection.
This will only need to be done once on the first deployment. Once the connection is created and linked to the connection reference it will remain after further deployments.
If you have already created the connection, you can programmatically set the connection on the connection reference if you needed using the following (You will need to be impersonating an actual user rather than using the SPN - see below).
```
# Set the connection on a connection reference:
Set-CrmRecord -conn $conn -EntityLogicalName connectionreference -Id $connectionreferenceid -Fields @{"connectionid" = $connectorid }
```
Note: Interestingly, you can actually turn on a Cloud Flow that only uses the Current Environment connector without actually connecting your connection references – this is done automatically for you. For the purposes of the scenario let’s assume that we also have other connectors is use such as the Office 365 Connector.

6. Key Point - Turning flows back on after subsequent deployments

The challenge now is that subsequently, ALM automated deployments to this solution using the Service Principle will turn the flows off again. The connection references will stay connected, but the flows will be off. Furthermore, as mentioned above you can't use the Service Principle to edit connection references or turn flows on so we need to impersonate a real user (I hope this will be fixed in the future). To do this, we can use the Power Apps Admin Powershell scriptlets to get the user who created the connections in use (manually above) and impersonate this user to turn the flows on.

Here is the full powershell script that you can add to your build or release pipeline:

$connectionString = 'AuthType=ClientSecret;url=$(BuildToolsUrl);ClientId=$(BuildToolsApplicationId);ClientSecret=$(BuildToolsClientSecret)'
# Login to PowerApps for the Admin commands
Write-Host "Login to PowerApps for the Admin commands"
Install-Module  Microsoft.PowerApps.Administration.PowerShell -RequiredVersion "2.0.105" -Force -Scope CurrentUser
Add-PowerAppsAccount -TenantID '$(BuildToolsTenantId)' -ApplicationId '$(BuildToolsApplicationId)' -ClientSecret '$(BuildToolsClientSecret)' -Endpoint "prod"

# Login to PowerApps for the Xrm.Data commands
Write-Host "Login to PowerApps for the Xrm.Data commands"
Install-Module  Microsoft.Xrm.Data.PowerShell -RequiredVersion "2.8.14" -Force -Scope CurrentUser
$conn = Get-CrmConnection -ConnectionString $connectionString

# Get the Orgid
$org = (Get-CrmRecords -conn $conn -EntityLogicalName organization).CrmRecords[0]
$orgid =$org.organizationid

# Get connection references in the solution that are connected
Write-Host "Get Connected Connection References"
$connectionrefFetch = @"
<fetch>
    <entity name='connectionreference' >
    <attribute name="connectionreferenceid" />
    <attribute name="connectionid" />
    <filter><condition attribute='connectionid' operator='not-null' /></filter>
    <link-entity name='solutioncomponent' from='objectid' to='connectionreferenceid' >
        <link-entity name='solution' from='solutionid' to='solutionid' >
        <filter>
            <condition attribute='uniquename' operator='eq' value='$(BuildToolsSolutionName)' />
        </filter>
        </link-entity>
    </link-entity>
    </entity>
</fetch>
"@;
$connectionsrefs = (Get-CrmRecordsByFetch  -conn $conn -Fetch $connectionrefFetch -Verbose).CrmRecords

# If there are no connection refeferences that are connected then exit
if ($connectionsrefs.Count -eq 0)
{
    Write-Host "##vso[task.logissue type=warning]No Connection References that are connected in the solution '$(BuildToolsSolutionName)'"
    Write-Output "No Connection References that are connected in the solution '$(BuildToolsSolutionName)'"
    exit(0)
}

$existingconnectionreferences = (ConvertTo-Json ($connectionsrefs | Select-Object -Property connectionreferenceid, connectionid)) -replace "`n|`r",""
Write-Host "##vso[task.setvariable variable=CONNECTION_REFS]$existingconnectionreferences"
Write-Host "Connection References:$existingconnectionreferences"

# Get the first connection reference connector that is not null and load it to find who it was created by
$connections = Get-AdminPowerAppConnection -EnvironmentName $conn.EnvironmentId  -Filter $connectionsrefs[0].connectionid
$user = Get-CrmRecords -conn $conn -EntityLogicalName systemuser -FilterAttribute azureactivedirectoryobjectid -FilterOperator eq -FilterValue $connections[0].CreatedBy.id 

# Create a new Connection to impersonate the creator of the connection reference
$impersonatedconn = Get-CrmConnection -ConnectionString $connectionString
$impersonatedconn.OrganizationWebProxyClient.CallerId = $user.CrmRecords[0].systemuserid

# Get the flows that are turned off
Write-Host "Get Flows that are turned off"
$fetchFlows = @"
<fetch>
    <entity name='workflow'>
    <attribute name='category' />
    <attribute name='name' />
    <attribute name='statecode' />
    <filter>
        <condition attribute='category' operator='eq' value='5' />
        <condition attribute='statecode' operator='eq' value='0' />
    </filter>
    <link-entity name='solutioncomponent' from='objectid' to='workflowid'>
        <link-entity name='solution' from='solutionid' to='solutionid'>
        <filter>
            <condition attribute='uniquename' operator='eq' value='$(BuildToolsSolutionName)' />
        </filter>
        </link-entity>
    </link-entity>
    </entity>
</fetch>
"@;

$flows = (Get-CrmRecordsByFetch  -conn $conn -Fetch $fetchFlows -Verbose).CrmRecords
if ($flows.Count -eq 0)
{
    Write-Host "##vso[task.logissue type=warning]No Flows that are turned off in '$(BuildToolsSolutionName)."
    Write-Output "No Flows that are turned off in '$(BuildToolsSolutionName)'"
    exit(0)
}

# Turn on flows
foreach ($flow in $flows){
    Write-Output "Turning on Flow:$(($flow).name)"
    Set-CrmRecordState -conn $impersonatedconn -EntityLogicalName workflow -Id $flow.workflowid -StateCode Activated -StatusCode Activated -Verbose -Debug
}

Managing connection details

Since your pipeline will want to run on release pipelines as well as branch environments, I use variable groups to define the connection details.

Something like this.

Note: The name is in the format, branch-environment-<BRANCH NAME>

So then in a YAML pipeline, you can bring in the details you want to use for the specific branch using:

variables:
- group: branch-environment-${{ variables['Build.SourceBranchName'] }}

When you use the script in a Release pipeline, you can simply add the right Variable Group for environments you are deploying to:

Summary

When you first deploy your solution with connection references, they must be connected (manually through the Solution Explorer, or programmatically by updating connectionid) before the flows that use them can be turned on.
This connection reference connecting cannot be done by a service principle - the deployment script will need to impersonate a non-application user.
One approach is to use the user that created the connection references to get the user to impersonate - this way you don't need to manually specify the user for each environment. If you have multiple users involved in connection reference authentication, you will likely need to impersonate the user for each connection.
After each subsequent deployment, you will need to turn on the flows again. This also needs to be performed using impersonation.
You can setup variable groups that will be dynamically picked using the current branch (for build pipelines) or the release environment.
I hope at some point, this kind of operation will be supported by the Power Platform Build Tools out of the box.

Continue to website...

Develop 1 - Dynamics 365 Architecture Services