Speech-to-Sign Language Translation Using Transformer Models

This paper presents the design and implementation of an end-to-end speech-to-sign translation system using Transformer-based models. A speech recognition module converts English audio into text, which is then translated into sign-language glosses via a sequence-to-sequence Transformer. A sign generation component maps gloss sequences into animated or video sign output. The system is evaluated on public datasets, showing high word-recognition accuracy and competitive translation quality (e.g. BLEU score improvements over baselines). Our solution leverages recent advances in pretrained speech encoders (such as Wav2Vec2 and Whisper) and neural machine translation, achieving low-latency operation suitable for real-time use. The results demonstrate substantial progress toward accessible communication for the Deaf community, highlighting the potential of deep learning in assistive language technologies.