Loading...

How do LLMs work with Vision AI? | OCR, Image & Video Analysis

How do LLMs work with Vision AI? | OCR, Image & Video Analysis

Combine vision and language in an AI model with the latest vision AI model in Azure Cognitive Services. Use natural language to fetch visual content in images and videos without needing metadata or location, generate automatic and detailed descriptions of images using the model’s knowledge of the world, and use a verbal description to search video content.

Cognitive Service for Vision AI combines both natural language models (LLM) with computer vision and is part of the Azure Cognitive Services suite of pre-trained AI capabilities. It can carry out a variety of vision-language tasks including automatic image classification, object detection, and image segmentation. Similar to GPT, the foundational language model, Project Florence, used in this case infuses deeper language skill with vision analytics to make training, inferencing and interacting with your image and video content simpler using natural language. 

Azure Expert, Matt McSpirit shares how to customize the model and use these capabilities in your own apps.

► QUICK LINKS:

00:00 - Introduction

00:48 - Project Florence

01:52 - Open-world recognition

03:19 - Dense captioning

04:23 - Run frame analysis

05:02 - Train a custom model

06:29 - Build custom apps

07:41 - Wrap up

► Link References:

Check out https://aka.ms/CognitiveVision

► Unfamiliar with Microsoft Mechanics?

As Microsoft's official video series for IT, you can watch and share valuable content and demos of current and upcoming tech from the people who build it at Microsoft.

• Subscribe to our YouTube: https://www.youtube.com/c/MicrosoftMechanicsSeries

• Talk with other IT Pros, join us on the Microsoft Tech Community: https://techcommunity.microsoft.com/t5/microsoft-mechanics-blog/bg-p/MicrosoftMechanicsBlog

• Watch or listen from anywhere, subscribe to our podcast: https://microsoftmechanics.libsyn.com/podcast

► Keep getting this insider knowledge, join us on social:

• Follow us on Twitter: https://twitter.com/MSFTMechanics

• Share knowledge on LinkedIn: https://www.linkedin.com/company/microsoft-mechanics/

• Enjoy us on Instagram: https://www.instagram.com/msftmechanics/

• Loosen up with us on TikTok: https://www.tiktok.com/@msftmechanics

Published on:

Learn more
Need help with this product?

We can help you with How do LLMs work with Vision AI? | OCR, Image & Video Analysis

If you want help implementing, troubleshooting, or improving this product, contact us and we’ll point you in the right direction.

Microsoft Mechanics Podcast
Microsoft Mechanics Podcast

Made for tech enthusiasts and IT professionals. Expanded coverage of your favorite technologies across Microsoft; including Office, Azure, Windows and Data Platforms. We'll even bring you broader topics such as device innovation with Surface, machine learning, and predictive analytics.

Share post:

Related posts

Manage AI Cost + ROI with Copilot and Agent 365

Take control of AI spend across Microsoft Copilot, Cowork, and agents inside and outside of Microsoft. Track your own Copilot Credit usage tas...

2 hours ago

Azure SQL Database Hyperscale - Now for any size workload

Start small and scale up to your most demanding workloads on Azure SQL Database Hyperscale. Run serverless compute that scales with demand and...

15 hours ago

New Microsoft Copilot Harness | More power, token efficiency & control for agents in Foundry

Build agents you can trust with your company's data, apps, and resources using Microsoft Foundry. Start with a fully managed prompt agent and ...

9 days ago

New Conditional Access policy plans in Microsoft Entra

Group related Conditional Access policies into policy plans and define users, apps and conditions once for every rule in the plan. Add a rule ...

21 days ago

Fight AI with AI | Multi-Agent Security Defense

Match AI-powered attacks at machine speed with Project Perception. Point red, blue, and green AI agent teams at a known threat actor, confirm ...

1 month ago

Build Automated Agents, Workflows, and Apps - New in Copilot Studio

Take a business process from prompt to production by building agents, workflows, and apps in one place with Copilot Studio. Describe the agent...

1 month ago

Create agents fast with Agent Builder in Copilot

Build a specialist AI agent without writing a line of code. Describe what you want in one prompt, and Copilot Agent Builder generates the name...

1 month ago

New Agent 365 controls for Microsoft admins

Govern every AI agent running across your organization with Agent 365. Track agents from Microsoft, Amazon, Google, and Salesforce in one regi...

2 months ago

Windows 365: Five Years In, See What's New

An AI agent works inside your Windows 365 Cloud PC from a Teams chat on your phone — finding downloaded files, editing documents, and drafting...

2 months ago

Build your first Power App from data in seconds

Build a fully working model-driven app from your existing Dataverse, SharePoint, or SQL data in under a minute — screens, navigation, and form...

4 months ago

Newsletter

Get the latest Dynamics 365 and Power Platform content in your inbox

A curated digest of community blogs, product news, videos, and podcasts — delivered without the noise.

Weekly updates Unsubscribe anytime Fresh community picks
We use your email only for the newsletter and you can unsubscribe at any time.
By subscribing, you agree to the privacy policy.