──“Translating Voices and Lip Movements”: The Era, Possibilities, and Discomfort

According to a report by The Verge published on August 19 (UTC), Meta has officially launched "AI Translation (AI Voice Translation + Lip Sync)" on Facebook and Instagram Reels, allowing automatic dubbing into another language with a single toggle before posting. The translated voice mimics the original voice tone, and the lip movements are synchronized. Initially, it supports English⇄Spanish, and the translated Reels are automatically delivered according to the viewer's display language. Posts will clearly state "Translated by Meta AI," and the feature will gradually be available to public accounts on Instagram and creators with over 1,000 followers on Facebook.The Verge

This feature can be considered the "product version" of the automatic dubbing and lip-sync demo showcased at the 2024 Meta Connect. YouTube (Google) is also increasing its investment in automatic dubbing, accelerating the global competition for short video views with "built-in translation" as a premise.The Verge

What's Changing?: From Subtitles to "Crossing Borders with Voices"

Until now, cross-border viewing on Instagram and Facebook relied on automatic subtitles. AI dubbing shifts the initiative from "reading" to "listening". This is particularly beneficial for beauty, cooking, and review genres, where subtitles can easily overlap on small vertical screens, as well as for comedy and short dramas where timing is crucial. Viewers can casually watch in their native language, and creators can **"automatically generate" language-specific versions** without increasing editing workload.

How to Use: Toggle Before Posting, Final Check with Preview

In the new pre-posting menu for Reels, turn on **"Translate voices with Meta AI" and, if necessary, add "Lip-sync"**. You can preview before publishing. After publishing, the translated version is automatically selected according to the viewer's app language, and it is labeled as AI translation.The Verge

Scope of Offering: Starting Point for Global Expansion

The current focus is on English⇄Spanish as the starting point. However, media outlets are reporting Meta's comments suggesting "global creator expansion and expansion of target languages," shifting market interest to "which language will come next."TechCrunchThe Times of India

Practical Benefits for Creators

• A Significant Increase in Reach: Just by crossing English-speaking and Spanish-speaking regions, the potential audience size can double. Creators can "enter the English-speaking market through translation," where ad rates are higher.

• Reinventing Production: With a single script and performance, language-specific "different versions" can be automatically mass-produced. It's easier to expand successful patterns through test distribution.

• Lowering Collaboration Barriers: Even in cross-border duets/remixes, content can reach viewers in each creator's native language.

Concerns and Issues: Half Excitement, Half Unease

On social media, there is already a swirl of support and opposition.

• Supporters: Voices praising the reduction of subtitle burden and increased immersion, and calling the synchronization of lip movements "magical."

• Critics: Posts expressing a sense of "uncanny valley" regarding "modification of lip movements" and the quality of AI voices are prominent. One user shared their surprise, saying, "Foreign Reels suddenly started speaking English." Another thread pointed out that "AI voices worsen the overall impression of the video."Reddit

• Requests for Control: There are also voices calling for more detailed control over automatic dubbing, such as the ability to turn it on/off or fix the language.Reddit

• Notation and Transparency: Although label display is mandatory, the focus will be on operational rules, such as clear disclosure in ads or PR posts, the rights to voices, and the scope of secondary use.The Verge

Practical Tips for Improving Quality (For Distributors)

1. Scripts Should Be "Conversational Short Sentences" Rather Than "Literary": Short phrases have fewer breakdowns in voice synthesis.

2. Lower BGM: Loud BGM can reduce the readability of translation results.

3. Supplement Proper Nouns with Subtitles: This helps reduce translation inconsistencies.

4. Don't Over-rely on Lip Sync: Lip synchronization is not foolproof. Avoid long close-up shots.

5. Localize Labels and CTAs: Optimize thumbnails/titles/captions for each language.

6. Conduct A/B Testing with "Eng⇄Span" First: Pre-learn successful patterns for when Japanese support arrives.

In the Japanese Context: Preparing for the Wave of Language Expansion

Even before Japanese is officially supported, the strategy of creating new fans through "reverse exporting" from English version→Spanish translation is effective. Genres with strong language independence, such as games, anime, cooking, and travel, are particularly compatible. On the other hand, in areas where **"voice personality" is an asset**, such as talents and VTubers, the use of AI voices, agency authority, and revenue sharing should be clearly defined between agencies and creators.

Competitive Landscape: A Full-scale Battle of Platforms

YouTube is also focusing on automatic dubbing, and TikTok continues to enhance translation and voice synthesis. The initiative has shifted to **"who can create the most natural, transparent, and controllable translation experience." Meta has taken a step ahead by making Reels a **"core multilingual feature."The Verge

Summary

Meta's AI dubbing evolves cross-border viewing, which was centered on subtitles, into **"crossing borders with voices." While it changes the norms of production and distribution, issues of discomfort and rights have also become apparent. The next watershed will be the expansion of supported languages and the delicate design of viewer control**. Teams that quickly ride this wave and establish an "operational model" will become the winners in multilingual short videos.

Powered by Froala Editor