Armchair Architects: How to choose a LLM partner for your AI project

Introduction

Our host, David Blank-Edelman and our armchair architects Uli Homann and Eric Charran will be discussing how to choose a Large Language Model (LLM) partner.  Architects and developers are often at a crossroads: should they use existing AI models or venture into building their own? This decision is far from trivial, as it comes with a myriad of considerations, from return on investment to the rapid pace of AI commoditization.

What are the processes and criteria you should consider when choosing a LLM or a foundational model (FM) for your AI project, and are the trade-offs between building your own model or using a hosted service? What are some of the challenges of trusting and validating the outputs of LLMs and FMs?  When choosing an LLM partner, what things should you think about?

The AI Model Spectrum

There are many models available in the large language model (LLM) space. It began with OpenAI's ChatGPT, which quickly expanded to offerings from other providers like Google, and the open-source community. With potentially hundreds of thousands of models to choose from, the architects ponder over the criteria for selecting the right model. They emphasize the importance of evaluating the partner—be it a commercial entity or an open-source project—and ensuring the model aligns with the organization's needs and responsible AI criteria.

They point out that the open-source models often come with a strong support system, including extensive documentation and active forums for discussion and troubleshooting.

Building vs. Using Pre-Built Models

One of the topics discussed was a build vs. buy scenario. The rapid commoditization of AI capabilities poses a unique challenge: if you build your own model, how do you keep up with the continuous innovation? Conversely, if you choose an existing model, how do you ensure it fits your specific requirements and ethical standards? There is an inclination to build custom models in industries like financial services, where proprietary data is paramount, and the trust required when working with large commercial models.

Trust and Transparency in AI

AI models are very complex and even their creators may not fully understand their inner workings. Users must rely on the model's track record and the quality of its outputs. The architects argue that while evaluating AI models may seem similar to assessing any other technology, the opaque nature of these models adds an additional layer of complexity.

There is also the concept of responsible AI which was discussed in a previous blog, stressing the importance of developing AI in a way that is ethical and fair. The experts discuss the criteria for responsible AI, including transparency, accountability, and the mitigation of bias. They provide examples of how these criteria have been implemented in real-world scenarios, illustrating the positive impact of responsible AI practices.

Conclusion

There are many considerations when selecting an LLM or FM. Assess your specific needs, resources, and goals when deciding on your AI strategy and stay informed about the latest developments in AI technology. When selecting one, keep in mind the importance of trust, transparency, and have a clear understanding of the model's capabilities and limitations.

Watch the episode below or watch more episodes in the Armchair Architects Series.

Resources

• How to deploy large language models with Azure AI Studio.
• Explore the model catalog in Azure AI Studio
• Training: Create a large language model deployment
• Get Certified: Azure AI Fundamentals

Related Blogs

• Armchair Architects: What Is Responsible AI?
• Armchair Architects: Are AI Charters Valuable?
• Armchair Architects: Considerations for Ethical and Responsible Use of AI in Applications
• Armchair Architects: Large Language Models (LLMs) & Vector Databases
• Armchair Architects: Artificial Intelligence, Large Language Models, and Architects (Part 1 of 2)
• Armchair Architects: Artificial Intelligence, Large Language Models, and Architects (Part 2 of 2)