Enhanced Sign Language Communication through Deep Learning-based Gesture and Action Recognition and Correction

Enhanced Sign Language Communication through Deep Learning-based Gesture and Action Recognition and Correction

Jayaprakash S 1, Rengalakshmanan S2, Vyshali S3 ,Satheesh N P4

1Student , Department of Artificial Intelligence and Data Science,

Bannari Amman Institute Of Technology, Sathyamangalam,

2Student , Department of Artificial Intelligence and Data Science,

Bannari Amman Institute Of Technology, Sathyamangalam,

3Student , Department of Artificial Intelligence and Data Science,

Bannari Amman Institute Of Technology, Sathyamangalam,

4 Assistant Professor, Department of Artificial Intelligence and Data Science,

Bannari Amman Institute Of Technology, Sathyamangalam.

Abstract - Sign language is a crucial form of communication for the hearing impaired, presenting unique challenges for technological interpretation. This project focuses on developing a Sign Language Detection Using Action Recognition (SLDAR) system to address these challenges. Leveraging Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and multimodal fusion techniques, our system aims to accurately detect and recognize sign language gestures in real-time. The project involves collecting and annotating a diverse dataset comprising video, audio, and text data. We then design and train a deep learning model that combines features from these modalities to improve accuracy and robustness.The project involves collecting and annotating a diverse dataset, designing a deep learning model for multimodal fusion, and implementing a user-friendly interface for real-time interaction. The

system's performance is evaluated using metrics such as accuracy, precision, and recall. Our goal is to enhance accessibility and communication for the hearing impaired through the development of this advanced sign language recognition system.

Keywords:Acoustic Features, Linguistic Context, Mean  Opinion Score (MOS), Natural Language Processing,  NLP Algorithms, Signal-to-Noise Ratio (SNR), Speech  Enhancement, Speech Quality, Telecommunication  Systems, Voice Assistants.