Real-Time Bidirectional Indian Sign Language Translation Using Deep Learning and Hand Gesture Recognition

The implemented system combines MediaPipe Hands for hand landmark extraction, a Tensor-Flow/Keras model for sign classification, a FastAPI backend with SQLAlchemy/SQLite persistence, WebSocket-based real-time communication, and a React/Vite/TypeScript frontend, to-gether with user authentication, translation history, and a usage dashboard. The following directions have been identified for future work and are explicitly distinguished from the functionality cur-rently implemented: expanding the recognition vocabulary and the sign-video library toward broader ISL lexicon cov-erage; extending recognition from isolated signs to continu-ous, sentence-level recognition; incorporating two-hand ges-ture modeling; training on larger and more diverse datasets col-lected under varied recording conditions; exploring improved sequence models, including Transformer-based approaches, for both recognition and translation, building on the direction demonstrated by continuous-translation work such as [9]; de-veloping a mobile deployment of the platform; improving the robustness of the speech-processing component; exploring per-sonalized, signer-adaptive recognition; and improving robust-ness under lighting- and background-variation conditions. Con-tinuous sentence-level Transformer-based translation, in partic-ular, is identified here as future work and is not part of the cur-rent implementation.

[1] S. Katoch, V. Singh, and U. S. Tiwary, “Indian Sign Language recognition system using SURF with SVM and CNN,” Array, vol. and LSTM,” in 2023 World Conference on Communication & Computing (WCONF), Raipur, India, 2023, pp. 1–5, doi: 10.1109/WCONF58270.2023.10235113. [2]    C. Aparna and M. Geetha, “CNN and Stacked LSTM Model for Indian Sign Language Recognition,” in Machine Learning and Metaheuristics Algorithms, and Applications (SoMMA 2019),Communications in Computer and Information Science, vol. 1203, Springer, Singapore, 2020, doi: 10.1007/978-981-15-4301-2 10. [3]    K. Navendu and V. Sahula, “Word Level Sign Language Recognition Using MediaPipe and LSTM-GRU Network,” in 2024 IEEE International Symposium on Smart Electronic Systems (iSES), New Delhi, India, 2024, pp. 13–18, doi: 10.1109/iSES63344.2024.00014. [4]    P. Rawat, P. Kumar, V. K. Tamta, and A. Kumar, “A Compre-hensive Approach to Indian Sign Language Recognition: Lever-aging LSTM and MediaPipe Holistic for Dynamic and Static Hand Gesture Recognition,” EAI Endorsed Transactions on AI and Robotics, vol. 4, 2025, doi: 10.4108/airo.8693. [5]    R. Damdoo and P. Kumar, “An integrative survey on Indian sign language recognition and translation,” IET Image Processing, vol. 19, 2025, doi: 10.1049/ipr2.70000. [6]    R. Damdoo, P. Kumar, and R. Gogoi, “End-to-end sentence-level Indian sign language translation with ISH-NEWS dataset and transformer model,” Scientific Reports, 2026, doi: 10.1038/s41598-026-60893-0.