A Multilingual Adaptive Digital Speech Rehabilitation System with Real-Time Feedback
DR. Priyanka B.G 1 Virupakshi2 Shafiqha Khanum3
Assistant Professor, Dept. of CSE Dept. of CSE Dept. of CSE
Nayana N4 Sukanya5
Dept. of CSE Dept. of CSE
PES Institute of Technology and Management, Shivamogga, Karnataka, India
Emails: priyankabg@pestrust.edu.in, virukannada2018@gmail.com, shafiqhakhanum53@gmail.com, nayanagowda264@gmail.com,
sukanyahiremath802005@gmail.com
Abstract— Aphasia is a neurogenic communication disorder that disrupts an individual’s ability to speak, understand, read, or write, commonly occurring after a stroke or traumatic brain injury. Traditional speech therapy, although effective, is often time consuming, expensive, and limited by linguistic and geographical accessibility. This paper presents a scalable and multilingual AI-driven speech therapy system integrating automatic speech recognition (ASR), text-to-speech (TTS), adaptive feedback, and an avatar-based phoneme–viseme synchronization module for English, Hindi, and Kannada. The system utilizes multilingual Wav2Vec2 embeddings, phoneme-level speech analysis, and ani mated lip-sync feedback generated using Rhubarb Lip Sync and Media Pipe to deliver multimodal and inclusive rehabilitation. Experimental results demonstrate ASR accuracy between 89% and 93% across the supported languages, along with high user engagement reflected by an average patient satisfaction rating of 4.6 out of 5 and positive therapist feedback. The modular architecture enables deployment in resource-constrained environments while maintaining clinical relevance through integration with standardized assessment metrics such as the Western Aphasia Battery-Aphasia Quotient. Overall, the proposed framework provides an accessible, cost-effective, and technologically enhanced approach to speech therapy for individuals with aphasia.
Keywords—Aphasia, speech therapy, artificial intelligence, multilingual ASR, adaptive feedback, phoneme–viseme mapping, avatar lip sync, Kannada, personalized assessment
