A Real-Time Conversational AI Voice Agent with Intelligent Speech Processing

The proposed real-time AI voice assistant successfully combines speech recognition, intelligent response generation, and noise reduction into a single system. It overcomes many limitations of traditional voice-based systems by providing fast, reliable, and real-time communication. By using technologies such as FastAPI, LiveKit, Silero VAD, and the Google Gemini API, the system is able to process voice input accurately and generate responses with minimal delay. This makes it suitable for various applications, including virtual assistants, customer support services, and interactive voice-based platforms. The system performs effectively even in noisy environments, improving the overall user experience. Its scalable and flexible design also provides a strong foundation for future improvements and the addition of more advanced features.

[1]      X. Yu, Y. Ju, X. Yuan, L. Yang and H. Zhang, "A Multi-Speaker Speech Processing System for Dialect- Rich Environments with LLM-Based Optimization," 2025 9th IEEE International Conference on Network Intelligence and Digital Content (IC-NIDC), Beijing, China, 2025, pp. 213-217, doi: 10.1109/IC- NIDC67200.2025.11390222. [2]      H. Jafarzadeh Asl, M. Ghazvini Nejad, A. Edraki, M. Asgharian and V. Partovi Nia, "Tiny Noise-Robust Voice Activity Detector for Voice Assistants," 2025 IEEE 35th International Workshop on Machine Learning for Signal Processing (MLSP), Istanbul,     Turkiye,          2025,          pp.          1-6,           doi:10.1109/MLSP62443.2025.11204268.  keywords: {Voice activity detection;Accuracy;Personal voice assistants;Transforms;Noise robustness;Background noise;Human voice;Noise measurement;Internet of Things;Smart glasses;Noise Robustness;SpeechRecognition;Voice activity Detection},I. S. Jacobs and C. P. Bean, “Fine particles, thin films and exchange anisotropy,” in Magnetism, vol. III, G. T. Rado and H. Suhl, Eds. New York: Academic, 1963, pp. 271–350. [3]      M. Yamano, Z. Song and J. Hoshino, "Augmenting the Social Presence of Interactive Characters Using Real- time Speech Recognition," 2022 Nicograph International (NicoInt), Tokyo, Japan, 2022, pp. 85-88, doi: 10.1109/NicoInt55861.2022.00024. [4]      K. N., R. V., S. S. S. and D. R., "Intelligent Personal Assistant - Implementing Voice Commands enabling Speech Recognition," 2020 International Conference on System, Computation, Automation and Networking (ICSCAN), Pondicherry, India, 2020, pp. 1-5, doi: 10.1109/ICSCAN49426.2020.9262279. [5]      R. Manimegalai, M. Shobana, C. Rajesh, C. Parthasarathy, A. V. Sriharsha and T. D. Kumar, "Adaptive Emotion-Aware Speech Synthesis on Edge Devices: A Lightweight Dynamic Model Adaptation Framework using LoRA and OnDevice Fine-Tuning," 2026 International Conference on Electronics and Renewable Systems (ICEARS), Tuticorin, India, 2026, pp. 1513-1520, doi: 10.1109/ICEARS67481.2026.11416528.   keywords: {Training;Adaptation models;Renewable energy sources;Quantization (signal);Speech enhancement;Real-time systems;Physiology;Speech synthesis;Reliability;Robots;Speech Synthesis;Edge Devices;Low-Rank Adaptation;Lightweight Neural Adapters;On-Device Learning}