Integrating Indian Sign Language Recognition with Real-Time Speech Synthesis for video conferences

Integrating Indian Sign Language Recognition with Real-Time Speech Synthesis for video conferences

Biswajit Dey 1  , Rajdeep paul 2

1Department of Computer Application, Techno India University, Kolkata, India

2Department of Computer Application, Techno India University, Kolkata, India

---------------------------------------------------------------------***---------------------------------------------------------------------

Abstract - Both hearing and deaf people commonly face major communication hurdles in their daily lives. To solve, this study presents a real-time video calling system that uses ai model to recognize Indian Sign Language (ISL). Peers are connected via WebSockets, and video data is shared with the AI model for identification. Our approach captures 30 frames a second and buffers them as groups of 3 seconds that a backend AI model interprets. The application using grid fragmentation-based splitting and k-NN prediction detects the hand movements very accurately and translates these movements to textual equivalents that other peers uses. In the meantime, regular video and audio communication proceeds uninterrupted, providing a natural conversation experience. The system was tested with 15 normal ISL sentences with an average accuracy of 97.2%. Performance measurement is reflected in a frame rate of 5.3 FPS, maximizing real-time interaction and processing efficiency. Future development will be in improving gesture prediction, increasing the dataset, and implementing speech-to-text capabilities to increase accessibility further. By integrating AI-powered sign language recognition with video calling, the app acts as a bridge between the deaf and hearing communities and fosters inclusivity and accessibility in digital communication