Sign Language Detection Using AI and Machine Learning: An LSTM-Based Real-Time Action Recognition System

Abstract — Sign language serves as the primary and natural medium of communication for the Deaf and Hard of Hearing (DHH) community worldwide. Despite its linguistic richness and social significance, the automated recognition of sign language by computational systems remains a formidable research challenge. This paper presents a real-time Sign Language Detection system grounded in action recognition principles and powered by Long Short-Term Memory (LSTM) deep learning networks. The system leverages Mediapipe Holistic for accurate multi-landmark extraction across hands, face, and body, and employs a deep stacked LSTM architecture to model the temporal dynamics and sequential dependencies inherent in sign language gestures from continuous live video. A comprehensive training pipeline encompassing video acquisition, morphological preprocessing, key-point feature extraction, sequence formation, and hyperparameter-optimized LSTM training is proposed and evaluated. The system is validated on a multi-class gesture dataset under varied conditions. Experimental outcomes demonstrate a training accuracy of 96.2%, a validation accuracy of 91.5%, and a test accuracy of 87.3%, surpassing traditional static frame-based CNN methods by approximately 13 percentage points. Performance is assessed across precision, recall, F1-score, and real-time inference latency (~44 ms/frame), confirming robustness and practical usability. This research contributes a scalable, cost-effective, and deployable solution that bridges the communication gap for DHH individuals, facilitating their inclusion in educational, healthcare, and everyday social contexts.