Zoom's CTO, Xuedong Huang, discusses our federated approach to AI and how it provides high-quality performance for meeting summaries, recaps, and next steps.
8 min read
Updated on May 06, 2024
Published on March 26, 2024
In November 2023, I shared how Zoom’s federated AI approach achieved quality nearly equal to OpenAI GPT-4 with only 6% of the inference cost. As impressive as those results were, we can now deliver even better AI quality compared to OpenAI’s GPT-4 for our most popular meeting features. Zoom AI Companion reduced relative errors by over 20% (for Zoom meeting summary’s “recaps”) and 60% (for “next steps”) in comparison to GPT-4 in our internal human-validated blind benchmarking.
In support of our training efforts to refine task completion quality, our unique federated approach to AI takes advantage of many closed- and open-source advanced large language models (LLMs) working together for better results. This is in contrast to other providers who are tied to specific LLMs. For example, Microsoft Copilot has relied on GPT-4 and Google has relied on Gemini.
This approach to AI sets Zoom AI Companion apart, providing our customers with a high-quality experience with our most popular features. As I shared in my last update, we use our proprietary Z-scorer to judge the quality of our AI-generated outputs. First, we employ a lower-cost LLM most suitable for each task. Then, our Z-scorer evaluates the quality of the initial task completion. If needed, we can use another complementary LLM to refine the task. This process results in a higher-quality output in the same way a team of people can accomplish more together than any one individual.
We’ve since improved our Z-scorer by incorporating additional quality signals from a variety of LLMs. Also, to better align with human preference, we improved federated reinforcement learning. By federating Zoom LLM in combination with a set of complementary LLMs, Zoom’s popular meeting summary delivers high-quality results and, according to our recent benchmarking, can now outperform GPT-4, which is used to power Copilot in Microsoft Teams.
Regarding AI safety, we also reduced the inherent bias in most LLMs by forming a committee composed of multiple LLMs such as Claude-3, Gemini, and GPT-4 to reduce hallucinations and improve our Zoom LLM. For example, different LLMs are unlikely to make the same hallucinated mistake, so we can derive more consistent responses and reduce the impact of outliers.
We recently benchmarked our results for the two most popular components of Meeting Summary: meeting recaps and next steps. Over half a million Zoom customers have enabled these features since we launched AI Companion in September 2023.
In our latest internal benchmark, we had human judges pick the most accurate meeting summary without revealing which AI model was used to generate each summary. As shown in the chart below, Zoom LLM outperformed GPT-4 in English for both meeting recaps and extraction of next steps in each blind test. We can reduce relative errors of meeting recaps and meeting next steps by over 20% and 60%, respectively, which directly translates into superior quality advantages.
Figure 1. Human evaluation for meeting recap and next steps in English. Zoom LLM and Anthropic Claude-3 are federated for final results that significantly outperform using OpenAI GPT-4 alone.
We also measured the quality of the overall Meeting Summary for Japanese, using Zoom LLM compared to GPT-4. As you can see in the chart below, our federated approach was able to deliver better results here.
We’re committed to bringing the benefits of Zoom AI Companion across the Zoom platform to our customers at no additional cost on eligible paid Zoom plans.* This relentless focus on AI quality drives higher customer value in Zoom Workplace and Zoom Business Services. In addition, according to a GigaOm study commissioned by Zoom (published March 26, 2024), AI Companion transcription performed at 95% accuracy, and for in-meeting questions scenarios, AI Companion delivered results that are four times faster than ChatGPT-4 web.
With these latest innovations, you can feel confident that every Zoom meeting can be accompanied by an AI-generated meeting summary that has some of the best AI quality in the industry.
*AI Companion may not be available for all regions or industry verticals.
7 min read
Zoom Workplace brings communication, employee engagement, spaces, and productivity solutions together on a single platform with Zoom AI Companion capabilities woven throughout.
4 min read
Save even more time and effort throughout your day with AI Companion.
5 min read
Zoom's CTO, Xuedong Huang, discusses our federated approach to AI and how it provides high-quality performance at a lower cost.
