The DUB Master: A Deep-Learning Pipeline for Automated, Emotion-Aware Video Dubbing

The DUB Master: A Deep-Learning Pipeline for Automated, Emotion-Aware Video Dubbing

Ketan Mande, Darshan Londhe, Harsh Mishra, Mehul Parmar

Students Department of Computer Engineering,

SSBT’s College of Engineering and Technology, Jalgaon, Maharashtra, India
April 2025

Abstract—The demand for multilingual video con- tent has soared, yet manual dubbing remains costly ($40–$100/min) and slow (3–5 days per 5-min clip). The DUB Master integrates Whisper (WER 8.4%), MarianMT (BLEU 37.5), Coqui TTS (MOS 4.2/5), and MFA (sync RMS 85 ms) into one seamless pipeline, reducing costs by 90% and turnaround by 80%. We validate on a 15-hour, 12-video benchmark with 12 native raters and conduct ablation studies to quantify each module’s contribution. Deployment pilot with VidLocal demonstrated consistent performance in live settings. In contrast to commercial dubbing tools, our system offers full transparency, modular control, and the ability to be fine-tuned on custom data. The DUB Master prioritizes cultural relevance, emotional nuance, and speaker consis- tency, which are often lost in traditional or monolingual pipelines. Our integration of emotion detection into both the translation and TTS modules ensures that output not only sounds natural but also aligns contextually with on-screen expressions and narrative tone.

Index Terms—Automated Dubbing, Speech Recogni- tion, Neural Machine Translation, Voice Cloning, Lip Synchronization, AI Pipeline