Implementation of Communication through Gesture and Speech Synthesis

This paper presents a real-time, vision-based sign language gesture recognition system designed to assist communication for individuals with speech and hearing impairments. The system captures hand gestures using a camera, extracts hand landmarks through MediaPipe, and converts them into feature vectors for classification using a Random Forest model. A complete real-time gesture recognition pipeline was developed, achieving accurate hand landmark detection and classification. The system enables low-latency gesture-to-text conversion and is designed to be scalable for efficient deployment on embedded hardware platforms.. The system currently supports only a limited set of static gestures and is sensitive to variations in lighting conditions, background complexity, and occlusions. Additionally, limited dataset diversity affects the model’s generalization capability across different users and environments. Furthermore, the system does not fully support dynamic gestures or continuous sign language communication. The proposed approach successfully achieves real-time gesture recognition and displays outputs as text using OpenCV, demonstrating the effectiveness of a non-intrusive, camera-based solution that eliminates the need for wearable devices. The system is cost-effective, portable, and modular, allowing integration with speech processing and embedded platforms. Overall, the system demonstrates a practical and scalable assistive communication solution, with strong potential for real-world deployment and future advancement. Future enhancements include expanding the gesture dataset, enabling dynamic and sentence-level recognition using advanced models, improving robustness under varying conditions, and integrating mobile/cloud platforms. Additional features such as text-to-speech and speech-to-sign conversion can further enable complete bidirectional communication.

[1]                 G. Sundar, M. J. Paulraj, and S. Palanisamy, “Real-Time Recognition of American Sign Language Using MediaPipe and Machine Learning Techniques,” Procedia Computer Science, vol. 172, 2020, Article ID S1877050920300880. doi:10.1016/j.procs.2020.03.266. Full text: https://www.sciencedirect.com/science/article/pii/S1877050920300880?via%3Dihub [2]                 Shravani Jadhav, Siddhi Mhatre, Riddhi Patil, and Prof. Prachi Sorte, “Smart Hand Gloves for Deaf and Dumb People,” International Journal for Research in Applied Science & Engineering Technology (IJRASET), vol. 12, issue III, March 2024. Available: https://www.ijraset.com/best-journal/smart-hand-gloves-for-deaf- and-dumb-people.ijraset.com [3]                 H. Almshjary, A. Alghamdi, A. Barsheed, D. Alahmadi, O. Alzamzami, H. Bitar & A. Babour, “Intelligent Gloves: An IT intervention for deaf-mute people,” Journal of Intelligent Systems, vol. 32, no. 1, 2023. Available: https://www.researchgate.net/publication/368831568_Intelligent_gloves_An_IT_inter vention_for_deaf-mute_people [4]                 Anusha B. Ballary and Ganashree T. S., “Raspberry Pi based Assistive Device for Communication between Blind, Deaf, and Dumb People,” International Journal for Research Trends and Innovation (IJRTI), vol. 7, issue 9, 2022. Available: https://ijrti.org/papers/IJRTI2209032.pdf [5]                 Salunke, D., Joshi, R., Ranjan, N., Tekade, P., & Panchal, G. (2023). Sign Language Recognition System Using Customized Convolution Neural Network. Proceedings of International Conference on Data Science and Applications, Lecture Notes in Networks and Systems, 552, 825-837. Springer Nature Singapore.PDF available at: https://www.researchgate.net/publication/368396468_Sign_Language_Recognition_S ystem_using_Customized_Convolution_Neural_Network [6]                 Kaur, B., Chaudhary, A., Bano, S., Yashmita, Reddy, S. R. N., & Anand, R. (2024).Fostering inclusivity through effective communication: Real-time sign language to speech conversion system for the deaf and hard-of-hearing community. Multimedia Tools and Applications, 83, 45859–45880.Available: https://link.springer.com/article/10.1007/s11042-024-18667-3 [7]                 Gil-Martín, M., Marini, M. R., Martín-Fernández, I., Esteban-Romero, S., & Cinque, L. (2025). Hand Gesture Recognition Using MediaPipe Landmarks and Deep Learning Networks. In Proceedings of the 17th International Conference on Agents and Artificial Intelligence (ICAART 2025), pp. 24–30. DOI: 10.5220/0013053500003890. PDF available at: https://www.scitepress.org/Papers/2025/130535/130535.pdf [8]                 Rastgoo, R., Kiani, K., & Escalera, S. (2020).Hand sign language recognition using multi-view hand skeleton. Expert Systems with Applications, 150, 113336. [9]                 A Comprehensive Analysis Of Sign Language Recognition System 2025 PDF: https://rjpn.org/jetnr/papers/JETNR2505045.pdf rjpn.or [10]              A. Navendu,Word-Level Indian Sign Language Recognition using MediaPipe and LSTM,2024.PDF Available: https://www.techrxiv.org/users/798477/articles/1151368/master/file/data/Navendu_iS eS_2024-2/Navendu_iSeS_2024-2.pdf