Visual Speech Recognition Using Bidirectional LSTMs and Attention Mechanisms

This project presents an advanced Multi-Lingual Lip-Reading System designed to perform Visual Speech Recognition (VSR) by interpreting spoken words solely through visual lip movements, eliminating the need for audio input. The system uniquely addresses the linguistic diversity of India by providing simultaneous support for Kannada, Hindi, and English, complete with automatic language detection capabilities. The technical pipeline utilizes MediaPipe Face Mesh to track 31 specific lip landmarks, which are processed into a comprehensive set of 330 geometric and temporal features per frame, including velocity and acceleration derivatives. These features are analyzed by a deep learning architecture combining Bidirectional Long Short-Term Memory (BiLSTM) networks with an Attention mechanism to capture complex temporal dependencies in speech. Optimized for production environments using CUDA and cuDNN for GPU acceleration, the system achieves real-time inference speeds of 25–30 FPS with prediction accuracies ranging from 85% to 95%, making it a viable assistive technology for the deaf and hard-of-hearing community.