Lip Reading Using Visual Cues

Lip Reading Using Visual Cues

Sandesh Shinde*1, Ishita Shinde*2, Pranita Shinde*3, Mandar Shitole*4

*1,2,3,4 Student, Department of Information Technology, SVPM’s College of Engineering, Malegaon(Bk), Maharashtra, India

---------------------------------------------------------------------***---------------------------------------------------------------------

Abstract - Lip reading has emerged as a challenging and impactful area of research within speech recognition, with potential applications ranging from assistive technologies for the hearing impaired to communication in noisy environments. This project proposes a deep learning-based system designed to recognize and decode sentences from videos using only visual information. The system is lexicon-free and focuses on the classification of visemes visual representations of phonemes by leveraging a spatial-temporal convolutional neural network (CNN) in combination with a transformer-based architecture for sequence processing. A novel perplexity analysis approach is employed to convert recognized visemes into words. The model has been trained and evaluated on the real time videos, demonstrating significant improvements in word error rate (WER) and viseme classification accuracy. Furthermore, the system is robust to varying lighting conditions, making suitable for practical applications. The results indicate a promising direction for real-time automated lip reading, offering new possibilities in fields such as human-computer interaction, surveillance, and assistive communication technologies.

Key Words:  Lip Reading, Deep Learning, CNN, Transformer, Perplexity Analysis, Visual Speech Recognition.