AI-Powered Multimodal Sign Language Recognition, Translation and Assistive Communication System

Sign language is an essential communication medium for individuals with hearing and speech impairments, yet communication barriers still exist due to the limited availability of intelligent translation tools. This paper proposes an AI-powered multimodal sign language recognition, translation, and assistive communication system designed to improve accessibility and interaction between sign language users and the general community. The system utilizes MediaPipe to detect and extract 21 hand landmarks from sign gestures captured through a standard webcam. These landmarks are converted into structured JSON feature vectors and used to train a 1D Convolutional Neural Network (1D-CNN) model for accurate recognition of predefined as well as custom user-defined signs. The proposed system supports real-time and offline sign detection, sentence formation. multi-language translation, and text-to-speech output for enhanced communication. In addition, the platform enables multimodal interaction through voice, text, image, and video inputs. For video inputs, audio is extracted and converted to text, translated if necessary, and synchronized subtitles are embedded into the video. The system also includes a motivational support module that allows trained users to provide assistance and guidance. The proposed approach offers an inclusive, scalable, and hardware-independent solution for assistive communication.