Azure OpenAI and Call Center Modernization

Azure OpenAI and Call Center Modernization



Much has been written about the powerful use cases enabled by Azure OpenAI. One such example is the transformation of the modern call center as their operators evaluate the potential of using artificial intelligence and natural language processing to optimize their processes and improve customer interactions. Call centers face challenges of high call volumes, long wait times, and agent turnover. Although many tools already exist in the modern call center, Azure OpenAI deployment leveraging Large Language Model (LLM) and Generative AI has the potential to improve operational efficiency and customer satisfaction by:


  • Providing the call center agent with more precise information, more quickly. 
  • Assisting with more complex queries. 
  • Providing sentiment analysis of call center interaction to ensure customer retention. 

In this article, we describe a deployment where Azure OpenAI serves as a platform to assist human agents. Existing systems such as conversational chatbots, interactive voice response (IVR), and customer relationship management (CRM) all remain and complement the new system. 


Modern Call Center Architecture 

The fictitious organization Contoso operates a call center to support its business. It decides to use Azure OpenAI to improve its customer service and reduce the cost and time of handling customer calls. It is useful to consider AI solutions in three layers of functionality that group together tasks in an end-to-end context. 




Deployment Architecture  

These three AI functional layers form the foundation of the Contoso modern call center architecture as illustrated below:  




AI Platform: This comprises the baseline LLMs of Azure OpenAI (AOAI) and Azure AI Search. 

AI Application: This is a Web app (Azure App Service) which makes API calls to the AI Platform. This application also interfaces to the Call Center CRM to gain insight of customer profile and call history. The application is presented to AI Usage and needs to have intelligent input handling.  

AI Usage: This is the human call-center agent who uses the AI application. The agent is also holding a conversation with a caller/end-customer.  

In addition to these functional layers, there are other critical components which interface to these three layers.  

CRM: The CRM is a repository of customer profile and logs of all customer interaction. While Azure OpenAI itself is stateless, the call center operation may capture state of all customer interaction via the WebApp (AI Application) and store in the CRM based on corporate policy. 

Knowledge Base: These are data sources which include industry specifications, corporate service level agreements, configurations, and any knowledge base useful to Contoso call center operations. The data sources may be in PDF, Word, Powerpoint, Excel and a variety of other formats.   

PowerBI: Analytics may be run against the CRM, which is constantly updated. 

There are two main use cases served by the architecture:  


Use Case 1: In-Call Processing 

During a person-to-person conversation (1) between the caller (end customer) and the call-center agent, the agent uses the Web app (2) to assist with the customer query such as researching the steps to troubleshoot an issue or recommending a service offering most suitable to the customer based on his/her profile and needs. The Web app interfaces to the CRM (3) to pull customer information to have context of the caller’s profile and any previous call history. The Web app is also the AI application which takes the agent’s query as input and passes that query to Azure OpenAI (4) for a response. Depending on the agent prompt, the AI application may instruct the model to call AI Search (5) to access data sources from the knowledge base (6). The agent is in the loop to validate the accuracy of the response.  




This use case reduces the Average Handling Time (AHT), as well as improve First Call Resolution (FCR). 


Use Case 2: Post Call 

Conversations with call center agents are recorded and stored in a telephony server. After the call, the conversation is converted to text (using either an existing toolset or Azure Speech to Text AI service). The agent sends the file (1) using the Web app (2) to Azure OpenAI for summarization and for sentiment analysis (3). The agent provides a validation of the sentiment analysis and summarization, and this record is stored in the CRM (4). On a regular basis, the call center operations and business team can collect insights by using PowerBI analytics (5). 


The agent is able to quickly move onto the next customer conversation, and this post call use case reduces the average After-Call Work (ACW) time because Azure OpenAI automatically summarizes the conversation. 




Securing the Deployment Architecture 

Call centers are bound by strict standards and regulations designed to protect customer data and privacy. To honor this commitment, they must establish a robust security framework that not only spans all elements of their systems—including AI platforms, applications, and external databases—but also ensures the security of the interfaces, whether they are accessed by human operators or through automated APIs. 


Secure AI System Layer Analysis 

By analyzing the components and interfaces of the architecture, we can identify risks and threats and can take measures for protection. 










1. Call Center Agent (AI Usage) 

Overprivileged access 

Agent should not have access to records/data not relevant to the current customer/callee  


Sensitive Data Exposure 

Intentional or unintentional leakage of customer data 

2. AI Application (WebApp) 

OWASP Top 10 (e.g. Broken Access Control) 

Bad actor may bypass access control by manipulating the website URL 


Prompt Injection Attack 

User prompt may include malicious instructions 


Data authorization 


3. AI Platform (Azure OpenAI) 

OWASP Top 10 for LLM (e.g. Sensitive Data Disclosure) 

Prompt injection may cause LLM to leak sensitive data 


Model Denial of Service 

Resource heavy operations may disrupt availability of system 

4. AI Search 

Data authorization 

Search on behalf of WebApp or LLM may return data for which they are not authorized 

5. Knowledge Base 

Sensitive Data Exposure 

Intentional or unintentional leakage of customer data 

6. CRM 

Sensitive Data Exposure 

Data theft of valuable confidential customer data  




7. AI Usage to AI Application 

Adversary in the Middle Attacks 

Call Center Agent are often working from remote locations subject to open Internet based attacks  

8. AI Application to AI Platform 

Adversary in the Middle Attacks 

Although this interface is in a private Virtual Network, bad actors seek to find vulnerabilities including unencrypted links 

9. AI Platform to Grounding 

Adversary in the Middle Attacks 

Although this interface is in a private Virtual Network, bad actors seek to find vulnerabilities including unencrypted links 

10. AI Application to Customer Insight 

Adversary in the Middle Attacks 

Although this interface is in a private Virtual Network, bad actors seek to find vulnerabilities including unencrypted links 


Solutions to Protect the Architecture 




Microsoft offers specific recommendations and solutions to protect against the above-described risks. 


Elevating the security posture of Generative AI (GenAI) applications within Azure requires a holistic approach that addresses the full spectrum of data handling, compliance, and lifecycle management. A confluence of Azure Landing Zones, the Azure Well-Architected Framework, and robust security practices forms the backbone of a secure and scalable cloud solution. 


Best Practices Integration: 

  • Secure Development and Deployment: Adhere to the principles outlined in the Azure Well-Architected Framework, incorporating secure coding, configuration management, and consistent updates. Use pseudonymization for data privacy enhancement. 
  • Data in Use Protection: Leverage Azure’s Confidential Computing capabilities and Integrated Identity Management solutions like Entra ID paired with Role-Based Access Control (RBAC) for identity and access management. 
  • Audit Trails and Logging: Maintain detailed logs and leverage Anomaly Detection to identify and address unusual patterns, using tools such as Microsoft Defender for Cloud and Microsoft Sentinel. 
  • Privacy Risks Management: Proactively align practices with Cybersecurity Frameworks, like NIST, and manage supply chain risks to ensure comprehensive data protection. 
  • International Data Standards Adherence: Follow data handling standards for cross-border transfers and conduct Privacy Impact Assessments to mitigate privacy issues. 
  • Implementation of RBAC: Employ this key aspect of the Microsoft security baseline to limit sensitive data access to authorized users only, supplemented by periodic audits. 

Component-Specific Measures: 

  • AI Usage (Call Center Agent): Authenticate agents with Entra ID and Multifactor Authentication, and authorize access based on JIT and Just-Enough-Access principles. 
  • AI Application (WebApp): Validate inputs, manage authentication securely with Entra ID, and protect data at rest with encryption. 
  • Azure OpenAI (AI Platform): Adopt stringent data ingestion policies, utilize Azure Managed Identities, and implement runtime threat detection and compliance controls. 
  • AI Platform (Azure AI Search:( Ensure role-based access integration and data sensitivity assessment with Purview labeling. 
  • Knowledge Base (Grounding): Encrypt data at rest and enable Defender for Cloud and Defender for SQL for advanced threat protection. 
  • CRM: Encrypt confidential data, monitor threats, and use Purview for data labeling and loss prevention. 

Inter-Component Safeguards: 

  • AI Usage to AI Application: Protect data in transit with encryption and guard against various web attacks using Application Gateway with Web Application Firewall and Azure DDoS Protection. 
  • AI Application to AI Platform: Secure AI application-API connections with Azure API Management (APIM), utilize Azure Private Link for secure communication, and segment network infrastructure for improved security. 

Governance and Oversight: 

  • Logging and Monitoring: Monitor activity comprehensively with Azure Log Analytics. 
  • Sentinel Integration: Use Microsoft Sentinel for SIEM capabilities, establishing alerts and anomaly detection for unauthorized access and abnormal behavior. 
  • Defender for Cloud Deployment: Continuously assess Azure OpenAI and connected services' security posture and protect workloads with Microsoft Defender for Cloud. 
  • Purview for Data Governance: Discover, understand, and manage data with Microsoft Purview, ensuring policy and compliance for sensitive data protection. 

Overall, these strategic measures and implementations across components and interfaces guarantee a proactive, inclusive security framework. The adoption of Azure Landing Zone architecture fundamentals offers a firm foundation for safeguarding deployments, while conforming to the Azure security baseline ensures adherence to established security protocols. This fortified structure enhances customer and partner trust while maintaining a rigorous security stance in call center operations. 


Other Considerations 

While this Call Center OpenAI deployment is limited to enterprise data, the architecture is extensible to include Bing search results.  The concepts described here for privacy, integrity, and confidentiality may be extended to include Internet results. It is important to tailor the security considerations to the specific use case of your deployment, referencing the general guidelines provided by Azure to ensure that every aspect of your application’s infrastructure, from development to deployment, adheres to the highest standards of security and privacy. 



As we explore the potential of Azure OpenAI to revolutionize call center operations, the implementation of rigorous security measures becomes paramount. The Zero Trust framework's layered security approach and continuous adaptation to emerging threats are crucial in securing Azure OpenAI deployments. By following the best practices outlined herein, advancements in AI will be coupled with robust defenses against cyber threats, ensuring a balance of innovation and security. 



Published on:

Learn more
Azure Architecture Blog articles
Azure Architecture Blog articles

Azure Architecture Blog articles

Share post:

Related posts

General Availability of SQL FCI and AG Features SQL Server Enabled by Azure Arc

We have good news. Two business continuity features for SQL Server enabled by Azure Arc are now generally available: View Failover Cluster In...

1 day ago

Azure VMware Solution using a public IP down to the NSX-T Edge; configure SNAT, No-SNAT & DNAT

Azure VMware Solution How To Series: Configuring NSX-T SNAT, No-SNAT & DNAT rules   Overview Requirements Lab Environment NAT Rules K...

1 day ago

App attach for Azure Virtual Desktop now generally available

App attach for Azure Virtual Desktop allows IT admins to dynamically attach applications from an application package to a user session in Azur...

2 days ago

A Closer Look at Azure WAF’s Data Masking Capabilities for Azure Front Door

The Azure Web Application Firewall (WAF) on Azure Front Door offers centralized protection for your web applications against vulnerabilities a...

2 days ago

Azure Virtual Network Manager (AVNM) Mesh and Direct Connectivity are Generally Available!

Azure Virtual Network Manager's (AVNM) mesh connectivity configuration and direct connectivity option in the hub and spoke connectivity config...

2 days ago

Azure pricing: How to calculate costs of Azure products and services

In our previous blogs we explained the Azure pricing structure and how customers can estimate their project costs when migrating to Azure or b...

2 days ago

Azure Pricing: How to estimate Azure project costs

In the previous blog we explained how you can learn about Azure pricing with free services and a pay-as-you-go model. Next, lets understand ho...

2 days ago

Azure Pricing: How to navigate Azure pricing options and resources

In this blog we discussed customer pricing needs and how they match different phases of the cloud journey, and we provided various tools and r...

2 days ago

Azure pricing: How to optimize costs for your Azure workloads

In our previous blogs we explained the Azure pricing structure, how customers can calculate their costs when migrating or building in Azure, a...

2 days ago
Stay up to date with latest Microsoft Dynamics 365 and Power Platform news!
* Yes, I agree to the privacy policy