Lip Reading with xLSTMs: Enhancing Temporal Memory with Exponential Gating

Lip reading has emerged as a critical tool for improving human-computer interaction, accessibility, and surveillance systems. The traditional LipNet[3] model, leveraging Bidirectional Gated Recurrent Units (BiGRUs)[18], has demonstrated promising results in sentence-level lip reading. This paper proposes replacing BiGRUs[18] with Extended Long Short-Term Memory (xLSTM)[4] architectures to address key limitations in temporal modeling and memory retention. xLSTM[4] introduces exponential gating, new memory mixing techniques, and matrix-based memory structures, enhancing the capability to track complex temporal patterns and retain long-term dependencies. We present an experimental evaluation of xLSTMs[4] in a modified LipNet[3] framework, demonstrating significant improvements in accuracy, robustness to noise, and computational efficiency. The findings suggest that xLSTMs[4] are a superior alternative to traditional RNNs[24] for lip reading applications.