AI Tool for Indian Sign Language Generator from Audio-Visual Content in English/Hindi and Vice-Versa
Aishwarya K A, Arpitha M, Bindu P, and G Poornima
{aishwaryaka2104, arpitham1893, bindupthumma, gpoornimabdvt}@gmail.com
Department of Computer Science and Engineering, PES Institute of Technology and Management, Shivamogga, India
Abstract—People with hearing or speech impairments often face communication barriers in their day-to-day lives.The need for accessible communication tools for individuals with hearing or speech impairments continues to grow as digital communication and technology-driven interaction become central to everyday life. Indian Sign Language (ISL), being a primary mode of com- munication for many, requires automated systems that can inter- pret gestures accurately and efficiently. Motivated by this need, our project presents an AI-driven ISL Recognition and Transla- tion System that integrates Computer Vision, Deep Learning, and Natural Language Processing (NLP) within a unified web-based platform. The system employs MediaPipe to extract reliable hand landmarks and uses a Convolutional Neural Network (CNN) to learn and classify spatial gesture patterns from webcam streams or uploaded videos. For text-to-sign translation, the framework applies NLP techniques—including tokenization, lemmatization, and tense detection—to generate meaningful ISL animations. When a direct gesture animation is not available, the system automatically breaks words into alphabet-level signs to ensure complete translation. The preprocessing pipeline includes region- of-interest extraction and normalization to maintain consistent input quality. Implemented using a Django web framework with an intuitive user interface, the system supports educational, assistive, and accessibility-focused applications. Overall, this project delivers a reliable, user-centered solution that enhances communication and promotes digital inclusivity for the hearing and speech-impaired community
Keywords: Indian Sign Language (ISL), Gesture Recognition, Convolutional Neural Network (CNN), MediaPipe, Natural Lan- guage Processing (NLP), Assistive Technology.
