A Dual-Mode Sign Language Recognition System for Education and Communication Using CNN and CNN-LSTM Architectures

A Dual-Mode Sign Language Recognition System for Education and Communication Using CNN and CNN-LSTM Architectures

Dr. (Mrs.) J. R. Pansare1, Aditya Sarwade2, Kashish Kosalge3, Prathamesh Limbole4, Atharva Birari5
1Modern Education Society’s Wadia College of Engineering, Department of Computer Engineering, SPPU, Pune, India

1jayshree.pansare@mescoepune.org
2sarwadeaditya310@gmail.com
3kashishkosalge55@gmail.com
4limboleprathamesh0@gmail.com
5apb14070@gmail.com

Abstract:

The Diversified Sign Language Recognition System is designed to enhance educational accessibility for differently-abled individuals by providing an interactive platform to learn and practice sign language. Built using Human-Computer Interaction (HCI) principles, the system features a user-friendly interface with two main modes: Learning Mode and Understanding Mode. The Learning Mode uses a pre-trained CNN model on image-based datasets to help users learn static hand signs for alphabets in an engaging way.

The Understanding Mode enables real-time recognition of dynamic gestures using a combination of CNN and LSTM models trained on extracted hand keypoints. It provides immediate text and voice feedback, helping users improve communication skills through an intuitive experience. This system not only promotes inclusivity in education but also empowers differently-abled learners with a supportive tool for mastering sign language more effectively.

Keywords:
Sign Language Recognition, Human-Computer Interaction (HCI), Differently-Abled, CNN, LSTM, Real-Time Gesture Recognition, Educational Accessibility, Keypoint Detection, Voice Feedback, Learning Mode, Understanding Mode, Interactive Learning, Inclusive Education, Deep Learning.