GestOS: A Real-Time Multimodal Hand Gesture and Voice Based Human Computer Interaction System.

Hand gesture recognition and voice-based interaction are becoming natural alternatives to traditional input devices in human-computer interaction (HCI). However, many existing systems face high latency, limited scalability, or depend on a single interaction method. This paper presents GestOS, a real- time multimodal HCI system that combines hand gesture recognition and voice command processing to control a computer without physical input devices. The system uses MediaPipe for efficient hand landmark detection and applies a rule-based gesture classification method for low-latency performance. At the same time, a grammar- constrained voice recognition module is created using Windows Speech API to ensure high accuracy and few false positives. A multiprocessing structure is used to get past Python’s Global Interpreter Lock (GIL), allowing simultaneous execution of vision, voice, and command- processing modules. Experimental evaluation shows that the system provides real-time responsiveness with low latency and dependable gesture recognition in controlled settings. This approach offers a cost-effective, scalable, and practical solution for next-generation HCI applications, such as accessibility systems, smart environments, and touchless computing interfaces