Building a Private ChatGPT Interface With Azure OpenAI

TLDR - You can test my implementation at https://privategpt.baldacchino.net. If this appears slow to first load, what is happening behind the scenes is a 'cold start' within Azure Container Apps. Cold Starts happen due to a lack of load. To reduce costs, I have configured Azure Container Apps to tear down my container environment when there is no load to zero containers. The loading is due to the container starting (Alpine Linux and NodeJS). Cold Starts are not really a thing for workloads under regular load.


Chat GPT is amazing on so many levels, and it's free. But you know what they say. If something is free, then you are the product!

As part of OpenAI's T&C, your prompts (the questions you ask) will be used to further train OpenAI's LLM (Large Language Model). Are you okay with this valuable stream of data exiting your organisation? If you are reading this and are responsible for your organisaions security posture, how are you ensuring private IP is not being leaked out of your company so that you do not become the next Samsung? In order to prevent this, you can run a hosted version of this every popular LLM using Azure Open AI. There are many benefits of Azure Open AI but the two I want to highlight are .

1. Your prompts (questions) will never be used to train the large language model. We are just hosting a version of Open AI's models on Azure on your behalf
2. Privacy, from the ability to lock down end-points and fine grain RBAC, the options are there.

See this Microsoft Learn article for more details. Azure Open AI is not free, but it is almost free and certainly is going to increase your organizations security posture. I say almost free, because the reality is that tokens for Azure Open AI results in most conversations costing less than 10 cents. A link to Azure Open AI pricing can be found here. In this blog post we will build a private ChatGPT like interface, to keep your prompts safe and secure using the Azure OpenAI service and a raft of other Azure services to provide you a private Chat GPT like offering.

The Building Blocks  We are going do this in a way that is low on administrative effort as a result we will running any servers (Iaas) that we will need to water and feed. No scaling rules, no patching etc, this needs to be simple, so here is what we are going to use.

• DNS - Name Your Server [Azure DNS / Amazon Route 53 / BIND]: A canonical name to your CDN distribution.
• Ingress - Azure Front Door: Used in two ways. A CDN (Content Delivery Network) for our static content, and a private backhaul to my local Azure region of the origin. The later is less of a concern and provides little benefit if your userbase is condensed in one geographic region.
• Compute Layer - Azure Container Apps : Our application, ChatBot UI is based on a container, the easiest way to host this is to use Azure Container Apps. I don't need to worry about understanding Kubernetes primitives such as pods and hosts. Azure Container Apps is an on-demand container platform based on Kubernetes. Our container is NodeJS application based on Alpine Linux. You can take a look at the Dockerfile to see how this all comes together.
• Container Registry - GitHub Container Registry - Chatbot UI is an open source chat UI for AI models, it can be found at ghcr.io/mckaywrigley/chatbot-ui:main, host this in your own container registry or pull from the source.

Architecture This really is a simple architecture, whilst it is 12 steps it is an architecture that has been optimised for speed and low administrative effort. It will automatically update when new container versions are released and you have fine grain controls that you can toggle around not only updates but authentication and the ability to prevent public access.

What we will build.

• Step 1: DNS Query - Resolve in my sample, https://privategpt.baldacchino.net
• Step 2: DNS Response - Return CNAME FQDN of Azure Front Door distribution
• Step 3: DNS Query - Resolve Azure Front Door distribution
• Step 4: DNS Response - Respond with A record of Azure Front Door distribution
• Step 5: Connect to Azure Front Door distribution.
• Step 6: Azure Front Door to perform GET request on Origin (Azure Container Apps)
• Step 7: Azure Container Apps to fetch container image from Container Registry (GitHub Container Registry)
• Step 8: Container Registry to provide container to Azure Container Apps and Azure Container Apps to host and scale containers based on scaling rules
• Step 9: NodeJS containerised application to speak to Azure Open AI (prompt)
• Step 10: Azure Open AI to return prompt response to Azure Container Apps
• Step 11: Azure Container Apps to respond to Azure Front Door
• Step 12: Azure Front Door to respond back to user's browser with ChatBot UI

Configuration This is a relatively simple architecture, and unlike a lot of my posts I am not going to go into great amount of detail. I will use the GUI, but of course this can be automated. If you have a question that is not addressed by the sections below, please leave a comment on this article.

• Azure Open AI : Your Azure subscription will need to be whitelisted for Azure Open AI. At the time of posting (July 2023) you will need to request access via this form and a further form for GPT 4. Once you have access deploy either GPT-35-Turbo or if you have access to GPT-4-32k go forward with this model.  Note down the deployed model name, deployment name, endpoint FQDN and access key, as you will need them when configuring your container environment variables.Azure Open AI - Note down your end-point and keysDeploy either GPT 3.5 or GPT4

• Azure Container Apps: Configure Azure Container Apps as per the images below. What is important to note is the container path ghcr.io/mckaywrigley/chatbot-ui:main (you can obtain this from the repository at https://github.com/mckaywrigley/chatbot-ui), the port mapping of TCP 80 (HTTP) to TCP 3000 (Container runtime port).I have pasted in images showing ingress configuration (perhaps you want this private) and settings for authentication with a provider (OAuth etc), perhaps you want to restrict access to authorised viewers, ther are options to do so.You will need to define environment variables (ENV) for the container. These environment variables define variables such as the Azure Open AI Model, Endpoints, Keys and so on.

  When deploying the application, the following environment variables can be set:

  Environment Variable	Default value	Description
  OPENAI_API_KEY		The default API key used for authentication with OpenAI
  OPENAI_API_HOST	https://api.openai.com	The base url, for Azure use https://<endpoint>.openai.azure.com
  OPENAI_API_TYPE	openai	The API type, options are openai or azure
  OPENAI_API_VERSION	2023-03-15-preview	Only applicable for Azure OpenAI
  AZURE_DEPLOYMENT_ID		Needed when Azure OpenAI, Ref Azure OpenAI API
  OPENAI_ORGANIZATION		Your OpenAI organization ID
  DEFAULT_MODEL	gpt-3.5-turbo	The default model to use on new conversations, for Azure use gpt-35-turbo
  NEXT_PUBLIC_DEFAULT_SYSTEM_PROMPT	see here	The default system prompt to use on new conversations
  NEXT_PUBLIC_DEFAULT_TEMPERATURE	1	The default temperature to use on new conversations

  Azure Container Apps SummaryDefine environment variables as per the table above.
  Be sure to map TCP 80 to TCP 3000
  You can validate this container is working accordingly by looking at the 'Console' log and by actually hitting the application endpoint.Logs are helpful in debugging container runtime issues
  Our application running via Azure Container Apps

• Azure Front Door: I will tell anyone who listens, a CDN (Content Delivery Network) is the best arm's length protection you can provide your application. Strip anything that's not HTTP(s) at the edge and accelerate your application by keeping these bits as close to the end-user as possible. A quick and dirty DDOS solution and prevent direct access by embedding a Layer 7 header in the CDN and validating in the application. I will be using Azure Front Door as a CDN. My origin is set to my Azure Container App URL. I am using a custom domain name (privategpt.baldacchino.net) to which I will need to move forward with DNS Zone modification by adding in a TXT record .A plus of Azure Front Door is that it will provide a HTTPS end-point, provide TLS termination and then make a call over HTTP to the origin (Azure Container Apps).Once your distribution is online, test via the distribution URL before adding in a custom domain name and routing rule. I don't have instructions now screen shots for DNS zone modification but from Azure DNS through to Route53 and beyond the process is somewhat the same.
  
  AFD configured, showing the AFD distribution FQDN along with the origin
  Configure AFD to use a private domainPerform DNS validation of the _dnsauth TXT record

Test - You Are Done I hope you are like me, testing along the way. Test after Azure Container Apps is online, test after Azure Front Door is configured and finally test after you have a private domain. Feel free to have a poke around my instance at https://privategpt.baldacchino.net, I do have API limits which you will experience if you hit this too hard and I am using GPT-35-Turbo Test via the CNAME based FQDNOur own private ChatGPTSummary
Take control of your data, you IP and build your own ChatGPT like interface using Azure Open AI and slew of other Azure services. It really isn't that hard. This quick walkthrough illustrated just how easy it is for you to do. If you are reading this and working for an organisation who is still accessing ChatGPT, I urge you to stop and think about how you can gain control of your data, so that your data doesn't become the product.

Thanks
Shane Baldacchino