Real Time Sign-Language to Speech and Text System

This paper described SignAI, a browser-based real- time sign language translator that converts hand gestures into text and speech without any specialised hardware or server-side processing. By combining MediaPipe’s 21- landmark hand tracker with velocity-augmented features and a stacked LSTM classifier deployed via TensorFlow.js, the system achieves a macro-averaged F1-score of 0.94 across a 10-gesture vocabulary and keeps end-to-end la- tency below 2.5 seconds on everyday CPU hardware. The literature review showed that landmark-based repre- sentations consistently beat raw-pixel approaches on both efficiency and environmental robustness, that LSTM re- mains the most practical architecture for online gesture sequence classification, and that CPU-deployable models are a practical necessity for broad accessibility. SignAI’s design reflects all three of these findings. The in-browser data collection tool and the Three.js skeleton overlay fur- ther distinguish it from prior work by directly addressing two problems the literature repeatedly flags: limited signer diversity and poor user interpretability. The planned extensions—broader multi-signer data col- lection, a Transformer-LSTM hybrid, integration of facial grammar markers, and a formal community user study— will substantially increase the system’s linguistic reach and real-world usefulness. SignAI is a practical step toward communication support that is accessible, hardware-free, and respectful of user privacy.

[1]L. Pigou, S. Dieleman, P.-J. Kindermans, and B. Schrauwen, “Sign Language Recognition Using Convolu- tional Neural Networks,” in Computer Vision – ECCV 2014 Workshops, Lecture Notes in Computer Science, vol. 8925, Springer, Cham, 2015, pp. 1–8. [2]A. Ojha, A. Pandey, S. Maurya, A. Thakur, and Dr. Dayananda P, “Sign Language to Text and Speech Transla- tion in Real Time Using Convolutional Neural Network,” International Journal of Engineering Research & Technol- ogy (IJERT), vol. 8, no. 15, NCAIT-2020. [3]J. Fink, P. Poitier, M. Andre´, L. Meurice, B. Fre´nay, A. Cleve, B. Dumas, and L. Meurant, “Sign Language-to- Text Dictionary with Lightweight Transformer Models,” in Proc. 32nd IJCAI, AI for Social Good Track, Macao, China, Aug. 2023, pp. 5968–5976. DOI: 10.24963/ijcai.2023/662. [4]S. Kumar and V. Rao, “Intelligent Sign Language Recognition Using IoT Glove System,” International Jour- nal of Internet of Things and Embedded Systems, vol. 8, no. 2, 2023. [5]R. Mehta, A. Jain, and P. Singh, “Enhancing Accessibil- ity with Real-Time Conversion of Text-to-Sign Language,” International Journal of Advanced Computer Science and Applications, vol. 15, no. 1, 2024.