TLDR: Translate, dub, and lip-sync videos into 29+ languages. Here's an example:
https://app.vivalabs.ai/?modal=videoShared&shareID=c955c97b-...I was creating a few educational math videos for kids in Latin America last summer, and I noticed they weren't easily following subtitles. I wanted to make the video content more engaging for them, but I'm still a ways away from Spanish proficiency.
Went down the rabbit hole of audio + video AI models... and now we have the initial version of Viva Labs! Given a video, we first translate the audio to the target language using a pipeline of ASR models, translation APIs, & LLMs; users can edit these translations to fix any errors. Then, we dub the audio with similar voices or voice clones. Finally, we sync the speakers lip movements to match the dubbed audio.
Our early users have surprised us with some of the common use cases like dubbing online course content, product explainers + marketing promos, podcasts, and international newscasts.
Still a lot of work to do to create full immersion. The following areas of technical exploration we're pursuing: 1) matching the output audio's tone, prosody, & emotion with that of the input voice using speech to speech models, 2) training lip sync models with more robust lip sync for extreme poses and occluding objects, 3) using LLMs to produce translations that are more colloquial and more closely match the pacing of original audio.
Here's a sample of Mira Murati's GPT4o announcement lip dubbed to Russian: https://app.vivalabs.ai/?modal=videoShared&shareID=c955c97b-...
Looking forward to seeing what videos you dub! You can dub 3 minutes of video for free at https://app.vivalabs.ai or you can ask our twitter bot https://x.com/VivaDubs to audio dub a video on twitter.
