Real-Time Sign Language Translator Using MediaPipe and Bidirectional LSTM

Four design goals guided this project: operate without specialized sensors, run fast enough for live conversation, cover a meaningful signing vocabulary, and produce sentence-level rather than word-level output. All four goals were met. The delivered system records 91.4% classification accuracy across fifty ASL word classes, sustains a 28–30 FPS processing rate on an unmodified consumer laptop, and outputs full sentences aloud within approximately one second of a gesture’s completion. The technical foundation rests on five innovations: a coordinate normalization scheme that improved accuracy by 8.3 percentage points; a compact BiLSTM architecture with roughly 380,000 parameters that runs entirely on CPU; a voting filter that reduced false triggers by 73% at a latency cost of 267 ms; a fuzzy phrase-matching layer that raised sentence assembly accuracy to 65% under real recognition errors; and an asynchronous audio synthesis module that decouples speech output from video throughput. Beyond the performance numbers, the work illustrates that the combination of an open-source skeleton tracker, a modest recurrent network, and a lightweight language module is sufficient to deliver a usable, low-cost sign language translation tool on hardware that most households already own.

1. World Health Organization, World Report on Hearing, Geneva, Switzerland: WHO Press, 2021. 2. F. Zhang, V. Bazarevsky, A. Vakunov, A. Tkachenka, G. Sung, C.-L. Chang, and M. Grundmann, “MediaPipe Hands: On-device real-time hand tracking,” in Proc. ECCV Workshop on CV for AR/VR, Glasgow, UK, 2020. 3. O. Koller, J. Forster, and H. Ney, “Continuous sign language recognition: Towards large vocabulary statistical recognition systems,” Computer Vision and Image Understanding, vol. 141, pp. 108–125, Dec. 2015. 4. S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, Nov. 1997. 5. D. Li, C. Rodriguez, X. Yu, and H. Li, “Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison,” in Proc. IEEE WACV, 2020, pp. 1459–1469. 6. H. R. V. Joze and O. Koller, “MS-ASL: A large-scale data set and benchmark for understanding American Sign Language,” in Proc. BMVC, Cardiff, UK, 2019. 7. O. M. Sincan and H. Y. Keles, “AUTSL: A large-scale multi-modal Turkish sign language dataset and baseline methods,” IEEE Access, vol. 8, pp. 181340–181355, Sep. 2020. 8. A. Duarte et al., “How2Sign: A large-scale multimodal dataset for continuous American sign language,” in Proc. IEEE/CVF CVPR, Nashville, TN, USA, 2021, pp. 2735–2744. 9. L. Pigou, A. Van Den Oord, S. Dieleman, M. Van Herreweghe, and J. Dambre, “Beyond temporal pooling: Recurrence and temporal convolutions for gesture recognition in video,” IJCV, vol. 126, pp. 430–439, 2018. 10. R. Cui, H. Liu, and C. Zhang, “Recurrent convolutional neural networks for continuous sign language recognition by staged optimization,” in Proc. IEEE CVPR, 2017, pp. 7361–7369. 11. S. Jiang, B. Sun, L. Wang, Y. Bai, K. Li, and Y. Fu, “Skeleton aware multi-modal sign language recognition,” in Proc. IEEE/CVF CVPRW, Nashville, TN, USA, 2021. 12. P. Dreuw, D. Rybach, T. Deselaers, M. Zahedi, and H. Ney, “Speech recognition techniques for a sign language recognition system,” in Proc. INTERSPEECH, 2006, pp. 2513–2516.