Vision-Two-Voice: An AI-Powered Real-Time Two-Way Sign Language Detection and Communication System

Preliminary evaluation of the dataset based on custom landmarks indicates that the landmark-based classification system converges quickly via CNN-LSTM. Validation accuracy will parallel training accuracy after approximately 15–20 epochs and will stabilize at about 99.0% (training) and 97.0% (validation) through low overfitting. The results on the held-out test set indicate the system achieves an overall classification accuracy of 90.4% (as shown in Table III; with a per-class F1-score average between 87–94%) using MediaPipe landmark extraction [1],[2]. MediaPipe landmark extraction is optimised to run on a CPU and achieves very high performance operating in real-time within a standard laptop without using a GPU. The palm greeting detection system performed well in isolated tests, achieving 96.2% precision and 94.8% recall rates, thereby providing reliable results for this particular automation use case [2],[11]. From examining the confusion matrix, it appears that most misclassifications happen between gesture pairs with common or visually similar hand shapes or have intersecting motion trajectories (e.g., ISL “Thank You” and ISL “Sorry” gestures both follow a hand-to-chin arc). Adding more facial-expressions landmarks could increase the overall detection accuracy by over 93% [3],[7].

perception pipelines,” arXiv:1906.08172, 2019. [2] F. Zhang et als., “MediaPipe Hands: On-device real-time hand tracking,” arXiv:2006.10214, 2020. [3] V. Bazarevsky et als., “BlazePose: On-device real-time body pose tracking,” arXiv:2006.10204, 2020. [4] T. Starner J, Weaver, A. Pentland, “Real-time ASL recognition using video,” IEEE TPAMI; 20(12):1371-1375. 1998. [5] L. Pigou et al., “Sign Language Recognition using CNNs,” ECCVW2015;572-578. [6] N.C. Camgoz, et als., “Neural Sign Language Translation,” CVPR2018, 7784-7793 [7] R. Rastgoo, K.Kiani, S. Escalera, “Sign Language Recognition: A Deep Survey,” Expert Systems with Applications; 164(113794). 2021. [8] R. Cui, H.Liu, C.Zhang, “Recurrent CNN for Continuous Sign Language Recognition," CVPR2018, 7361-7369   [9] O. Koller, J. Forster, H. Ney, “Continuous Sign Language Recognition,” CVIU.;141,108-125.2015. [10] P.Kumar, S.Rawat, “ISL Recognition using SVM,” IJCA;178(35):1-5.2019.