Lip Reading with xLSTMs: Enhancing Temporal Memory with Exponential Gating

Lip Reading with xLSTMs: Enhancing Temporal Memory with Exponential Gating

Naveen Poliasetty1, Madhava Rao Chelluri2, V. Sharmila3, S. Praveen4, S. Vidhya Sagar Appaji5
1B.Tech Computer Science and Engineering (CSE-AIML), Raghu Institute of Technology
2B.Tech Computer Science and Engineering (CSE-AIML), Raghu Institute of Technology
3B.Tech Computer Science and Engineering (CSE-AIML), Raghu Institute of Technology
4B.Tech Computer Science and Engineering (CSE-AIML), Raghu Institute of Technology
5Professor in Department of CSE, Raghu Engineering College

Abstract - Lip reading has emerged as a critical tool for improving human-computer interaction, accessibility, and surveillance systems. The traditional LipNet[3] model, leveraging Bidirectional Gated Recurrent Units (BiGRUs)[18], has demonstrated promising results in sentence-level lip reading. This paper proposes replacing BiGRUs[18] with Extended Long Short-Term Memory (xLSTM)[4] architectures to address key limitations in temporal modeling and memory retention. xLSTM[4] introduces exponential gating, new memory mixing techniques, and matrix-based memory structures, enhancing the capability to track complex temporal patterns and retain long-term dependencies. We present an experimental evaluation of xLSTMs[4] in a modified LipNet[3] framework, demonstrating significant improvements in accuracy, robustness to noise, and computational efficiency. The findings suggest that xLSTMs[4] are a superior alternative to traditional RNNs[24] for lip reading applications.

Key Words: Lipreading, Human-computer interaction xLSTM , Temporal modelling, Memory retention

Computational efficiency