Lip Reading using Deep Learning

Lip reading, or the task of interpreting speech by observing lip movements visually, has been a longstanding problem in computer vision and speech processing. Handcrafted feature-based traditional approaches tend to fail because they are prone to visual noise and depend on handcrafted features. This work introduces a deep learning-based method for real- time lip reading with Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks. The model is learned over a bespoke dataset gathered with webcam input and MediaPipe Face Mesh for precise mouth area extraction. The final system has the capability of real-time prediction of words spoken through video input, giving a solid ground for silent speech recognition system.