Over the past few months, we’ve been working hard to expand and improve the features and usability of the Azure Communication Services calling tools. This blog provides a quick summary of the key calling features released recently.

We will focus on:

• Raw media access
• Additional browser support for JS Calling SDK
• Call readiness tools, and
• Simulcast

Raw Media Access

Public preview of audio media access is now complemented by the availability of video media access in public preview. Developers can now access the full raw media stream for incoming and outgoing audio and video media content during a call.

The video media access API provides support for developers to get real-time access to video streams to capture, analyze, and process video content during active calls. Developers can access the incoming call video stream directly on the call object and send custom outgoing video stream during the call. This sets the foreground services to support a variety of video and audio manipulation. Outgoing video access can be captured and implemented with screen sharing, background blur, and video filters before being published to the recipient, allowing viewers to build privacy into their calling experience. In more complex scenarios, video access can be fitted with a virtual environment to support augmented reality. Spatial audio can be injected into remote incoming audio to add music to enhance a waiting room lobby.

In one example, like a contact center scenario, these streams can be used to run custom AI models for analysis such as your homegrown natural language processing (NLP) for conversation analysis or provide real-time insights and suggestions to boost agent productivity. In virtual appointments, media streams can be used to analyze sentiment when providing virtual care for patients or provide remote assistance during video calls leveraging Mixed Reality capabilities. This also opens a path for developers to leverage newer innovations with endless possibilities to enhance interaction experiences.

Learn more in the Raw media access overview and QuickStart guide.

Additional Browser Support for JS Calling SDK – Firefox and Edge updates

Azure Communication Services is committed to enabling communications on any device and on any platform. The JavaScript Calling SDK has now expanded to include support for VoIP and Video calls using Firefox on Desktop browser in public preview. In addition, it now supports Edge browser on Mac platform in General Availability. Please see the full list of supported browsers by platform in the table below here.

Call Readiness Tools to Improve the User Experience

Azure Communication Services releases call readiness features to help new customers using virtual communication experiences for the first time manage issues related to devices (e.g. properly configuring microphone, camera, and speakers), connectivity (e.g. any quality or connection issues), and permissions (e.g. enabling or re-enabling an application to access various settings). This month, we are excited to announce access to Call Volume Level as well as the ability to Detect Multiple Tabs using the ACS SDK in a browser.

• Access Call Volume Level: Developers may want the ability to check and display to end users the current microphone volume. The volume indicator API provides the capability for developers to get the volume of local and remote audio streams. The ACS calling API exposes this information using the getVolume value, which is a number ranging from 0 to 100 and is sampled every 200ms to get near real-time value of the level on the corresponding audio stream. This can be used to build volume UI elements to help end users visualize and adjust their volume up or down depending on user preferences. Learn more in our QuickStart guide.
• Detect Multiple Tabs Using Azure Communication Services SDK in a Browser: Azure Communication Services has launched the ability for developers to detect if multiple calls are running simultaneously across multiple browser tabs. This follows best practices to reduce common user errors, such as locking a user’s microphone or camera settings in another tab on the browser. In order to mitigate this undefined behavior, the isCallClientActiveInAnotherTab API can be used to detect when an application with the ACS SDK is loaded in multiple tabs. Developers can use this information to build UI elements that alert a user to close their affected tabs and proceed with the active call. Learn more in our overview.

Simulcast

In group calls with three or more users, Simulcast helps to preserve call quality and bandwidth in situations where one user with poor network conditions acts as the lowest common denominator and degrades the video experience for all users.

Simulcast is a technique by which a video feed is encoded with different qualities and sent to a selective forwarding unit (SFU). The SFU decides and sends the video feed with the optimal quality to each user based on the receiving user’s respective network. Lower resolution/bitrate content can be sent to the receiver on poor network and higher resolution/bitrate content can be sent to receivers on good network, thereby minimizing the impact of the lowest common denominator.

One common use case that demonstrates the benefits of Simulcast is a receiver on a large screen (e.g. desktop) in a call with another receiver on a small screen (e.g. mobile) will not be affected by the quality of the smaller view. Simulcast will send a higher resolution to the desktop receiver and a smaller resolution to the mobile receiver.

Azure Communication Services Simulcast - An Azure Communication Services concept document | Microsoft Learn