SignBridgeAI: An AI-Powered Multimodal Two-Way Communication System for Deaf, Mute and Hearing Users
Shreya Ghodake1 , Sadiya Gavandi2, Kshiti Gambhir3, Gauri Bangale4,, Prof. Nikhil Deshpande5
1 Department Of Information Technology, Sinhgad College of Engineering, Pune- 41
2 Department Of Information Technology, Sinhgad College of Engineering, Pune- 41
3 Department Of Information Technology, Sinhgad College of Engineering, Pune- 41
4 Department Of Information Technology, Sinhgad College of Engineering, Pune- 41
5 Department Of Information Technology, Sinhgad College of Engineering, Pune- 41
Email: shreyaghodake.scoe.it@gmail.com
Abstract - SignBridgeAI is an AI-powered, multimodal platform designed to bridge communication barriers between deaf, mute, and hearing individuals through Indian Sign Language (ISL). The system integrates real-time gesture recognition leveraging MediaPipe and deep neural architectures (CNN and LSTM) for precise ISL sign classification. Speech-to-text conversion via Whisper API and the generation of 3D animated avatars using Unity further facilitate accessible two-way communication. Supporting a comprehensive dataset of 293 ISL gestures with customizable features, SignBridgeAI enables robust interaction suited for diverse user needs. Experimental evaluation demonstrates high recognition accuracy (97.25% for signs, 95% for speech-to-text) and low latency performance, validating its effectiveness for real-time use. The solution advances accessibility and social inclusion for the deaf community in India, and its scalable, device-agnostic design positions it as a practical assistive technology for broader societal integration.
· Key Words: Indian Sign Language (ISL), Sign Language Recognition, AI-Powered Communication, Gesture Recognition, Deep Learning, Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM), Speech-to-Text Conversion, 3D Animated Avatars, Assistive Technology, Real-Time Translation, Multimodal Communication, Accessibility in Technology, Inclusive Communication, Human-Computer Interaction
