Real-Time Sign Language Recognition from Webcam Landmarks Using a Multi-Stream Temporal Attention Network

SignVision processes webcam frames to predict ASL words based on landmarks, a two-stream attention classifier, and a subtitles user interface. It processes at an estimated 32 frames per second on a laptop CPU. With respect to the 67 glosses for which we obtained results, it has a top-1 accuracy of 69.0% and a top-5 accuracy of 70.7% on held-out clips. With only 41 clips and a random split available, this is our preliminary guess, with a reasonable top-1 accuracy range of 26% to 54%. Our contribution is a small, inspectable, and privacy-respectful pipeline from camera frames to subtitles, together with an upfront discussion of its limitations. The difference between perfect training accuracy and low test accuracy means that data, not architecture, is limiting here. The steps that need to be taken first are improved evaluation and data collection because all of Sec. 8 relies on them.

1. Abadi, M., et al.: TensorFlow: a system for large-scale machine learning. In: Proc. USENIX Symp. Oper. Syst. Des. Implement. (OSDI), pp. 265–283 (2016) 2. Bradski, G.: The OpenCV library. Dr. Dobb's J. Softw. Tools (2000) 3. Camgoz, N.C., Koller, O., Hadfield, S., Bowden, R.: Sign language transformers: joint end-to-end sign language recognition and translation. In: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 10023–10033 (2020) 4. Carreira, J., Zisserman, A.: Quo vadis, action recognition? A new model and the Kinetics dataset. In: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pp. 6299–6308 (2017) 5. Cheok, M.J., Omar, Z., Jaward, M.H.: A review of hand gesture and sign language recognition techniques. Int. J. Mach. Learn. Cybern. 10(1), 131–153 (2019) 6. Cui, Z., Zhang, W., Li, Z., Wang, Z.: Spatial–temporal transformer for end-to-end sign language recognition. Complex Intell. Syst. (2023) 7. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997) 8. Joshi, A., et al.: iSign: a benchmark for Indian sign language processing. arXiv:2407.05404 (2024) 9. Li, D., Rodriguez, C., Yu, X., Li, H.: Word-level deep sign language recognition from video: a new large-scale dataset and methods comparison. In: Proc. IEEE Winter Conf. Appl. Comput. Vis. (WACV), pp. 1459–1469 (2020) 10. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: Int. Conf. Learn. Represent. (ICLR) (2019)