Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognition

Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognition

Bandi Dixitha Dept of ECE IARE

Dr. S China Venkateshwarlu Professor Dept of ECE IARE

Dr. V Siva Nagaraju Professor Dept of ECE IARE

Abstract - In this paper, we propose a novel Multi-Stage Multi-Modal Pre-Training framework for Automatic Speech Recognition (ASR) that effectively leverages the complementary information from multiple modalities, such as audio, text, and visual context, to enhance model performance. Our approach consists of three sequential pre-training stages: (1) a Masked Audio Encoding (MAE) stage that learns robust acoustic representations by reconstructing masked segments of speech, (2) a Cross-Modal Learning Regularization (CLR) stage that aligns acoustic and visual-textual representations using a contrastive loss, thereby bridging the modality gap, and (3) a Speech Translation Mid-Training (STMT) stage that introduces a translation objective to incorporate linguistic context and improve generalization. Extensive experiments on standard ASR benchmarks demonstrate that our multi-stage framework significantly outperforms existing unimodal and bimodal pre-training methods, achieving state-of-the-art results in both low-resource and high-resource settings. This work highlights the potential of structured, multi-modal, and multi-task learning in building more robust and accurate ASR systems.

Key Words:  multi-modal pre-training, automatic speech recognition, masked audio encoding, cross-modal learning regularization, speech translation, representation learning, contrastive loss, multi-task learning.