Intelligent ISL Translation System Using NLP and 3D Avatar

The paper provides the design, implementation, and testing of an Intelligent Indian Sign Language Translation System using NLP and 3D avatar animation for effective and efficient communication between the hearing impaired and the rest of the world. The proposed intelligent system serves the society as it automates the process of translation from speech or texts to sign languages via an animated realistic 3D avatar along with providing two-way communication via a gesture recognition interface. Experimental evaluation shows the high quality of ISL gloss generation, natural avatars gesture rendering, and accurate sign recognition by the system. Low delay is achieved and makes the tool appropriate for real-time communication. Each system element’s impact on performance is proven by the ablation study. Feedback collected from both deaf people and hearing people proves the feasibility of the system and its usefulness and accessibility. In summary, the research presented in this paper proves the high potential of using NLP, ML, and computer graph-ics technologies in developing inclusive communication aids that could assist the Deaf population to communicate freely and easily. The application developed is scalable, accessible, user-friendly, and widely applicable in different areas of life including but not limited to education, health care, public services, and personal communication. Further works include extending ISL glossary, improving continuous sign recognition performance, introducing local dialects, utilizing wearable sensor technology for gesture detection, and deploying the system on a larger scale.

[1]     C. Vogler and D. Metaxas, A Framework for Recognizing the Simultaneous Aspects of American Sign Language, Computer Vision and Image Understanding, vol. 81, no. 3, 2001. [2]     O. Koller, J. Forster, and H. Ney, Continuous Sign Language Recog-nition: Towards Large Vocabulary Statistical Recognition Systems Handling Multiple Signers, Computer Vision and Image Understand-ing, vol. 141, 2015. [3]     S. Hochreiter and J. Schmidhuber, Long Short-Term Memory, Neural Computation, vol. 9, no. 8, 1997. [4]     P. Kumari and A. Vaish, Single Semantic Level Feature Extraction for Sign Language Recognition, Procedia Computer Science, vol. 46, 2015. [5]     J. Devlin, M. Chang, K. Lee, and K. Toutanova, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Proceedings of NAACL-HLT, 2019. [6]     Z. Cao, G. Hidalgo, T. Simon, S. Wei, and Y. Sheikh, OpenPose: Re-altime Multi-Person 2D Pose Estimation Using Part Affinity Fields, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 1, 2021. [7]     E. Efthimiou and S. Fotinea, GSLC: Creation and Annotation of a Greek Sign Language Corpus for Sign Language Recognition, Proceedings of LREC, 2007. [8]     S. Gibet, N. Courty, and J. Kamp, Gesture-Based Communication in Human-Computer Interaction, Lecture Notes in Computer Science, vol. 3881, 2006. [9]     A. Othman and A. Jemni, English-to-American Sign Language Sta-tistical Machine Translation, Proceedings of the Language Resources and Evaluation Conference, 2012. [10]    P. Naert, C. Reverdy, and S. Gibet, A Survey on the Animation of Signing Avatars, Computer Graphics Forum, vol. 39, 2020. [11]    B. Huang, Y. Zhao, and K. Liu, Sign Language Translation from Spoken Language Gloss to Sign Language Video, IEEE Transactions on Multimedia, 2021. [12]    D. Jurafsky and J. H. Martin, Speech and Language Processing, Pearson, 3rd ed., 2023. [13]    S. Camgoz, S. Hadfield, O. Koller, H. Ney, and R. Bowden, Neural Sign Language Translation, Proceedings of CVPR, 2018. [14]    T. Ko, C. Kim, and J. Kim, Neural Sign Language Production, Proceedings of BMVC, 2019. [15]    O. Koller, Quantitative Survey of the Field of Automatic Sign Language Recognition, arXiv preprint arXiv:2004.09870, 2020. [16]    A. Sreeraj and I. Sheeba, Indian Sign Language Recognition Using Machine Learning and Wrist-Worn Sensor, International Journal of Computer Science and Engineering, 2019. [17]    R. W. Picard, Affective Computing, MIT Press, 1997. [18]    S. Escalera et al., ChaLearn Looking at People Challenge 2014: Dataset and Results, Proceedings of ECCV Workshops, 2014. [19]    M. Pantic and L. J. M. Rothkrantz, Toward an Affect-Sensitive Multimodal Human-Computer Interaction, Proceedings of the IEEE, vol. 91, no. 9, 2003. [20]    N. C. Camgoz et al., Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation, Proceedings of CVPR, 2020. [21]    H. Cooper, B. Holt, and R. Bowden, Sign Language Recognition, Visual Analysis of Humans, Springer, 2011. [22]    R. Zwitserlood, M. Verlinden, J. Ros, and S. Van der Schoot, Synthetic Signing for the Deaf: eSIGN, Proceedings of the Workshop on Assistive Technologies for Vision and Hearing Impairment, 2004. [23]    T. Bickmore and R. Picard, Establishing and Maintaining Long-Term Human-Computer Relationships, ACM Transactions on Computer-Human Interaction, vol. 12, no. 2, 2005. [24]    P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, Enriching Word Vectors with Subword Information, Transactions of the ACL, vol. 5, 2017. [25]    G. Tolentino et al., Static and Dynamic Hand Gesture Recognition in Sign Language, International Journal of Advanced Trends in Computer Science and Engineering, 2019.