Implementing Consumer Data Right (CDR) Solutions on Azure

Published Thursday, July 28, 2022

Introduction

“Consumer Data Right”, or CDR, is an Australian Government initiative introduced in Nov 2017 to the Banking industry (aka “Open Banking”) and completed in Nov 2021. Today CDR is being rolled-out within the Energy sector during 2022/2023, and then later the Telecommunications and Open Finance sectors, with a long-term view to apply economy-wide.

CDR brings together 3 key parties to operate in a single managed data exchange marketplace;

Data Holders - organizations that provide services to consumers and hold data about those services
Consumers - individuals utilizing a service from a Data Holder
Data Recipients - 3^rd party organizations that seek to offer new “value-add” services based on Consumer data

CDR has been designed to support Australian Consumers with greater control over their data - and empower them to choose to share their data from their current service provider (Data Holder) to other trusted 3^rd party providers (Data Recipient) for the purpose and duration for which they have consented.

This new and exciting CDR legislation brings the true ownership of data back into the hands of consumers!

CDR will create significant new opportunities for market participants and consumers alike - but also comes with several technical challenges that must be met by both Data Holders and Data Recipients.

The purpose of this blog is to outline how the Microsoft Azure Cloud Platform can help CDR participants in developing and hosting their CDR solutions in the Azure Cloud to not only meet the technical challenges imposed by the CDR legislation – but also be ready to take advantage of the business possibilities that operating within the CDR marketplace could bring.

CDR Marketplace Participants

To facilitate CDR a number of key parties are required to seamlessly operate together to ensure security, data exchange, monitoring and performance are maintained at all times, and standards/rules continue to be aligned to government requirements;

Participant	Role
Consumer	The end customer who has a service with an existing Data-Holder. The customer investigates a service from a provider and gives Consent for their data to be shared between their current Holder and their selected Recipient.
Accredited Data Holder (ADH)	The providers who currently hold consumer data (aka ‘givers’) Legislated to participate in the CDR eco-system On an authenticated request are required to share data with accredited data recipients (ADR) as and when directed by the Consumer A Data-Holder can optionally also be a Data-Recipient
Accredited Data Recipient (ADR)	The providers who receive a Consumer’s data after the consumer has given their consent (aka ‘receivers’) Optional to participate in the CDR eco-system, and are open to service any CDR Data Holder across any industry ADR’s can use consumer data for the purpose that the consumer consented.
Australian Competition and Consumer Commission (ACCC)	Manage a verified list of Data Holder and Data Recipient participants. Manage the CDR system and ensure all parties maintain standards, compliance and API data exchange endpoint performance

History and Success of CDR in Australia

The success of CDR relies on 4 main factors;

Performant and standardized implementation of CDR by Data Holder participants. It’s worth keeping in mind that many Data Holders are required by law to participate – and are mandated by the CDR legislation to provide a CDR system adhering to government standards
Growing ecosystem of 3^rd party Data Recipient participants that are looking to leverage data from the mandated Data Holders, and ultimately offer new value-added services that Consumers want to use.
Consumers that wish to share their data and access new/differentiated services offered by Recipients
Robust data governance process (rules/standards) that are secure, monitored and managed by Government

And so far early signs are positive! CDR has been successfully implemented in Open Banking (2021). As at writing;

There are 112 Data Holder brands and 31 accredited Data Recipients
CDR covers 99.18% of all household deposits provided by the CDR Holders (currently in Open Banking)

Today CDR is being implemented in the Energy Sector – with the initial holders (Origin, AGL, Energy Australia) required to have Production systems operational by Nov 2022, with remaining mandated Energy providers operational by Nov 2023. In future additional CDR industries will be added, including Telecommunications (Telco) and Open Finance (Insurance and Superannuation) – with timelines yet to be announced by Treasury.

CDR Technology, Standards and Rules

Data interchange systems, especially those distributed across multiple private organisations, can be complicated. To ensure CDR operates effectively it’s critical that participants adhere to an agreed set of data interchange standards (driven via API’s), end-to-end security and data processing rules.

In CDR these are defined in the “Standards” and “Rules” set down and maintained by the Data Standards Body (DSB) – and enforced by the ACCC. The ACCC publish a public list of CDR participants with their compliance status, and provide an availability/performance dashboard. The DSB is part of the Australian Government Treasury Department – which own CDR as an initiative.

DSB Policy	Description
Standards	All data interchange is based on defined standards built within secure API services developed/managed by each participant. There are dozens of API’s defined in the CDR Standards that support a functionally valid CDR service. CDR Standards are not static and will change over time. CDR participants are expected to align their systems with any modified or new Standards in order to remain compliant. The CDR API standards, including the agreed “future dated obligations” are well documented. Additionally the DSB also define a number of non-functional requirements, such as availability or performance requirements.
Rules	Rules define how the consent process works, how data is exchanged, the roles of the participants, how participants can become accredited, dispute resolution, privacy/safeguards, industry specific guidance, and more. There are hundreds of rules defined that must be implemented to support a functionally valid CDR system. The CDR Rules are not static and can change over time. CDR participants are expected to align their processes with any modified or new Rules in order to remain compliant. Breaches of the CDR Rules by any participant can attract financial penalties The CDR rules are well defined, along with a “developer-friendly” set of customer experience (CX) guidelines, which provide examples of how rules can be implemented by participants.

Consumer “Consent” Process of Data Interchange

Consent is central to CDR. It is the process by which a Consumer gives approval for an Accredited Data Recipient (ADR) to make a request against their Accredited Data Holder (ADH) to share specific parts of their data for a specific period of time.

When a consent request is received, the ADR is authorized to call an ADH’s CDR API service and request data on behalf of the Consumer. The ADH is required to provide that data within certain response timeframes and in a specific format.

There are several handshake steps involved between participants to facilitate the process. All interactions between participants is performed via CDR API’s implemented at both the ADH and ADR sides of the interchange. Lastly, the consent and data interchange process is secured, with all participants required to keep detailed audit records of what occurred, when and with who.

To illustrate how CDR operates, the below example steps though a high-level consent and data exchange process.

Consumer Consent Data Flow Diagram

Step	Process Description
1	A customer of “Acme-Co Energy” has a contracted electricity account held with the company. Acme-Co Energy hold customer account data, such as; name, address, metering plans/contacts, billing, invoicing, meter usage data, etc. The Consumer scans the market and discovers an Accredited Data Recipient (ADR) website, and a specific “value-add” service/offering they wish to use. The Consumer progresses though ADR registration process on ADR website, and consents to share their data to get access to the desired ADR service
2	The ADR redirects the consent request to Accredited Data Holder (ADH) for Consumer consent login The ADH sends the Consumer a “One-Time-Password” (OTP) on their existing customer contact channels (ie mobile, text/SMS, email etc) confirming the details of the ADR request, and seeks their consent to share data
3	The Consumer opens consent request in ADH website using the OTP The Consumer confirms on the ADH website which data they wish to share with the ADR, how long to share that data, whether its once-off or reoccurring, and what happens to the data after the consent expires (ie ADR should delete the data, anonymise data etc)
4	The ADH then returns the Consumer back to the ADR website along with confirmation and specific details of the consent which has been authorized. Both the ADH and ADR record all details of the consent and authorization for auditing purposes
5	The ADR has now been authorized by the Consumer for access to selected data sets held by the ADH – and for a specific period of time. The ADR can now request that specific data from the ADH which the Consumer consented to share.
6	The ADH responds with the data as requested by the ADR – and as consented by the Consumer. The ADH records detailed logs on transaction request/response performance for auditing purposes
7	The ADR can now provide the specific “value-add” service/offering to the Consumer using their actual up to date data, while completely independently the ADH continues to provide the Consumer their existing contracted services. The specified data will continue to be shared between the providers for the specified duration. Once this data sharing agreement expires, or the consumer no longer “consents”, then all data sharing ends.
8	The ACCC are not directly involved in the data interchange process – however they provide API services for verification of CDR participants (ie is a specific ADR accredited, etc), reviewing ADR or ADH performance data, status/availability of CDR services, etc

Conceptual Azure Reference Architecture

As CDR continues to be rolled out across industries in Australia, it will continue to mandate Data Holders in those industries participate to share data, which therefore encourages new Data Recipients to participate via new business ideas, offerings and services. This creates a growing industry of data interchange, services and innovation.

Below are Azure Reference Architectures demonstrating how CDR solutions for both the ADH and ADR providers could be deployed into Azure, along with the key principals of Azure Cloud Security to support CDR.

Accredited Data Holder (ADH) Azure Reference Architecture

For Data Holders - they can respond to CDR requirements in several ways. They may choose a single approach, or combine approaches into a single customized CDR solution (ex. part build a custom solution for back-end data/storage components, and part outsource to a SaaS provider for front end Web/API components).

Accredited Data Holder (ADH) System Implementation Options

The following ADH conceptual architecture provides a simple example to deploy a CDR solution on Azure to meet the objectives of #2 “Create new systems to deliver CDR”.

Accredited Data Holder (ADH) Azure Conceptual Reference Architecture

Accredited Data Recipient (ADR) Azure Reference Architecture

For Data Recipients – their participation is CDR is optional, but strongly encouraged to bring innovation for consumers into existing industries. Additionally, the business opportunities for ADR’s themselves is substantial; on consumer consent they will have access to high-quality, well-defined datasets offering them new business growth opportunities which would never have been possible prior to CDR.

The following ADR conceptual architecture provides a simple example of how an ADR could build/deploy a solution on Azure to meet the CDR requirements, standards, and rules.

Accredited Data Recipient (ADR) Azure Conceptual Reference Architecture

Azure Shared Responsibility, Compliance and Privacy: Security Building Blocks

Shared Responsibility

One of the reasons to use Azure for CDR applications/services is to take advantage of its wide array of security tools and capabilities. CDR developers can take advantage of multi-layered security provided by Microsoft across physical data centres, infrastructure and operations in Azure. Developers must understand the shared responsibility model and which security tasks are handled by Azure, and which by customers. Responsibilities vary depending on whether workloads are hosted on SaaS, PaaS, or IaaS.

Compliance

Deploying solutions on Azure also benefits from more than 100 compliance certifications, including IRAP certifications in Australia, and 35+ key industry specific compliance offerings including health, government, finance, education, manufacturing, and media. Microsoft engages globally with regulators, standards bodies, and non-governmental organizations to ensure Azure can achieve high levels of global certifications.

Privacy

With any deployment to Azure, the customer is the owner of the data provided for storing and hosting in Azure services, and the customer decides where that data is located across our large and ever-expanding network of 60 regions around the globe. Microsoft does not share data with advertiser-supported services, nor do we mine it for any purposes like marketing research or advertising. Microsoft process customer data only with customer agreement, and when we have that agreement, we use the data to only provide the chosen services.

Becoming an Accredited CDR Participant

CDR participants (namely ADH and ADR) cannot enter the market unless they have been “accredited” by the ACCC. This process includes passing validation checks for identity - and then confirming security, performance, business readiness, insurance, and finally a series of technical API functionality checks (called “CDR conformance tests”).

The journey for each participant includes multiple steps and is different depending on their role in CDR;

ADH Accreditation Journey - Data holder user journey | Consumer Data Right (cdr.gov.au)
ADR Accreditation Journey - Data recipient user journey | Consumer Data Right (cdr.gov.au)

CDR Participant Accreditation Process. Diagram adapted from CDR registration process journey

Conformance Test Suite (CTS) and the CDR Test Process

Each new CDR provider must pass the Conformance Test Suite (provided by ACCC) before they are made active on the CDR Register. The test pass itself is automated and takes approximately an hour. The test process confirms the technical conformance of the participants’ production-ready software using a range of simulated test scenarios.

If failures are encountered - then the errors need to be diagnosed and resolved by the participant before resubmitting their solution for retesting.

Accelerating CDR Development with Starter Code

The Government provides sample CDR solutions to accelerate the development experience for both ADH and ADR participants. These can be used to interact with each other, or developers can replace components with their own data holder or data recipient solution/code.

The starter-code provides developers the ability to see working code that conforms to the Data Standards, and has passed the Conformance Test Suite. The solutions include 400 automated integration tests demonstrating how to call the CDR APIs;

Sample source code on GitHub
Sample container images on Docker Hub

Summary – Where To Next from Here?

CDR is set to bring significant change to Australian industry, and with it great opportunities for CDR participants;

Consumers - new innovative service offerings based on data they own
Data Recipients - create a favourable environment to support generation of new business ideas
Data Holders - modernise existing data operations, and even act as Data Recipients themselves

As CDR continues to move into different industries over the coming years, the pool of Data Holders will continue to grow, and thus the ecosystem of Accredited Data Recipients will also grow in turn. Together they will deliver an ever-greater number of new innovative data products for the ultimate benefit of Australian Consumers.

As ADH and ADR participants begin their CDR journey - the Microsoft Azure Cloud Platform provides a number of benefits;

broad data centre distribution providing operational scale across the AU/NZ region
state-of-the-art cloud services providing IaaS, PaaS and SaaS offerings
skilled partners and SaaS marketplace offerings ready to deliver on the CDR requirements

Reach out to your Microsoft Account Team or Partner Team to learn more about CDR, the innovation opportunities it will provide to your organization, and how deploying CDR solutions to the Microsoft Azure Cloud can help you to maximize outcomes from this exciting initiative.

Reference: CDR Terminology

Term	Description
ACCC	The ACCC has a number of roles under CDR including: accrediting potential data recipients establishing and maintaining a Register of Accredited Persons and Data Holders monitoring compliance and taking enforcement action where necessary (under a co-regulatory model with the Office of the Australian Information Commissioner (OAIC)). The ACCC will work with Treasury, the DSB and the OAIC in the development and implementation of the CDR. providing guidance to stakeholders about their rights and obligations under the CDR as they relate to the ACCC’s role in the CDR ecosystem.
ADH	Accredited Data Holder. The ‘givers’ in a CDR system. The providers who currently hold consumer data. Required to share data with accredited data recipients when directed by the consumer. A Data-Holder can optionally also be a Data-Recipient
ADR	Accredited Data Recipient. The ‘receivers’ in a CDR system. The providers who receive a consumer’s data after the consumer has given their consent. The providers can then use it for the specific purpose the consumer has requested.
AEMO	Performs an array of gas and electricity market, operational, development and planning functions. It manages the National Electricity Market (NEM) and the Victorian gas transmission network. AEMO also facilitates electricity and gas full retail contestability, overseeing these retail markets in eastern and southern Australia. It is additionally responsible for national transmission planning for electricity and the establishment of a Short Term Trading Market (STTM) for gas
CDR	Consumer Data Right
CX	Customer Experience
DSB	Department in Treasury responsible for the creation of the technical standards for the sharing of consumer data
ECS	An independent regulator that promotes the long-term interests of Victorian consumers with respect to the price, quality and reliability of essential services
NEM	National Electricity Market. an arrangement in Australia's electricity sector for the connection of the electricity transmission grids of the eastern and southern Australia states and territories to create a cross-state wholesale electricity market.
OAIC	The OAIC is the primary complaints handler under the CDR scheme. The OAIC has a range of investigative and enforcement powers to handle privacy complaints and carry out other regulatory activities with respect to privacy.
OTP	One Time Password