Loading article...
Loading article...
Stream/Area:
-
Subject:
-
Paper ID:
IJSREM68837
Volume:
10 |
11 |
Nov 2026
Pages:
1-6
Publication Frequency:
Monthly (12 issues per year)
License:International Journal of Scientific Research in Engineering and Management follows the Creative Commons Attribution 4.0 (CC BY 4.0) License, allowing free use and distribution with proper attribution.

Department of Computer Science and Engineering
Don Bosco Institute of Technology

Department of Computer Science and Engineering
Don Bosco Institute of Technology

Department of Computer Science and Engineering
Don Bosco Institute of Technology

Department of Computer Science and Engineering
Don Bosco Institute of Technology

Department of Computer Science and Engineering
Don Bosco Institute of Technology
Sign language is the primary mode of communication for many people with hearing and speech impairments, yet the lack of real-time interpretation tools limits their interaction with non-signers. This paper presents Gesture2Speech, a real-time, web-based sign language interpretation system that recognizes alphabets and word gestures and converts them into text and speech. The proposed pipeline captures webcam frames in the browser, streams them over a WebSocket to a FastAPI backend, and applies two complementary recognition paths: MediaPipe hand landmarks with a Random Forest classifier for alphabets, and a YOLOv11n detector trained on 58 classes (26 letters and 32 word gestures) for words. Temporal stabilization based on consecutive-frame agreement and sliding-window majority voting suppresses frame-level noise. In the video-call mode, confirmed words are assembled into grammatical sentences by a large language model (LLM) and spoken through text-to-speech, while a reverse Text-to-Sign module plays real ASL clips from the WLASL vocabulary. On held-out validation data, the Random Forest reached 94.2% accuracy for ASL letters, and YOLOv11n reached 93.4% [email protected], 78.6% [email protected]:0.95, 87.5% precision and 91.2% recall. The results indicate that the system is a practical baseline for bidirectional, real-time sign language communication.
KEYWORDS
This paper presented Gesture2Speech, a real-time web system that recognizes sign language alphabets and word gestures and converts them into text and speech, with a reverse Text-to-Sign path for replies. The pipeline combines MediaPipe landmarks and a Random Forest for letters with a YOLOv11n detector for 58 classes, stabilized by consecutive-frame and sliding-window voting, and adds an LLM and text-to-speech stage for video calls. The Random Forest reached 94.2% accuracy for ASL letters, and YOLOv11n reached 93.4% [email protected], 78.6% [email protected]:0.95, 87.5% precision and 91.2% recall on validation data at about 5 ms per image. The system is a practical baseline for bidirectional sign communication on a fixed vocabulary, and broader evaluation, a larger vocabulary and network-ready calls are the main directions for strengthening it.
[1] Zhang, F., et al.: MediaPipe Hands: On-device Real-time Hand Tracking. arXiv:2006.10214 (2020). [2] Breiman, L.: Random Forests. Machine Learning 45(1), 5–32 (2001). [3] Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You Only Look Once: Unified, Real-Time Object Detection. IEEE CVPR, pp. 779–788 (2016). [4] Ultralytics: YOLO11 Documentation (2024). Available: https://docs.ultralytics.com [5] Li, D., Rodriguez, C., Yu, X., Li, H.: Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison. IEEE WACV, pp. 1459–1469 (2020). [6] W3C: WebRTC 1.0: Real-Time Communication Between Browsers. W3C Recommendation (2021). [7] Alsharif, B., et al.: American Sign Language Interpretation Using Deep Learning (2025). [8] Rastogi, U., et al.: Indian Sign Language Recognition Using a Combination of YOLOv10 with Swin Transformer Model (2025). [9] Shahin, N., Ismail, L.: ADAT: Time-Series-Aware Adaptive Transformer Architecture for Sign Language Translation (2025). [10] Alkhorafi, A. A., et al.: Ensemble Transformer-Based Word-Level Sign Language Recognition with Multi-Modal Input Fusion (2026).
Publication Date
11/9/2026
Peer Review Type
Double-Blind Peer Review
Review Rounds: 2
Documents
Article Views
0
Total views since publication
PDF Downloads
3
Total downloads since publication
Geographic Reach
5
Digital Preservation: This article is permanently archived on Zenodo and complies with all data preservation standards across major indexing platforms.
Plagiarism Screening
This manuscript has been checked for plagiarism using TurnitinPlagiarismChecker software.
Conflict of Interest
The authors declare no conflict of interest related to this research work.
Funding Information
No funding information available for this research.
Related research in the same subject area
Loading similar articles...