Gesture-Based Eye and Hand Controlled Virtual Mouse Pointer Using Machine Learning

This research presents the design and implementation of a Gesture-Based Eye and Hand Controlled Virtual Mouse Pointer as an innovative touchless Human–Computer Interaction system using computer vision and machine learning techniques. The proposed framework successfully demonstrates that conventional input devices such as the mouse can be partially replaced with a multimodal interaction approach that combines hand gesture recognition and eye blink detection for real-time computer control. By utilizing a standard webcam along with open-source technologies including OpenCV, MediaPipe, Python, NumPy, and PyAutoGUI, the system provides a cost-effective and accessible alternative without requiring specialized hardware. The developed system effectively enables cursor movement through hand gesture tracking while using eye blink detection for mouse click operations, thereby improving interaction efficiency and reducing accidental activations commonly found in gesture-only systems. Experimental evaluation indicates that the framework achieves high gesture recognition accuracy, reliable blink detection performance, and low response latency under moderate environmental conditions. These results confirm the practical feasibility of the proposed system for real-time applications. One of the key strengths of this research lies in its accessibility and versatility. The proposed solution can significantly benefit individuals with physical disabilities, users with limited motor control, and environments where touch-based interaction is inconvenient or unhygienic, such as healthcare institutions, laboratories, educational spaces, and industrial settings. The integration of multimodal interaction enhances usability by reducing dependency on a single control mechanism. However, certain limitations remain, including sensitivity to lighting conditions, background complexity, reduced precision for high-detail tasks, and possible user fatigue during prolonged usage. Despite these challenges, the system establishes a strong foundation for future advancements in touchless interaction technology. Future improvements may involve deep learning-based gesture classification, adaptive calibration techniques, improved eye-tracking precision, sensor fusion, and compatibility with augmented or virtual reality systems. Overall, this research demonstrates that multimodal gesture- and eye-controlled interfaces represent a promising step toward intelligent, inclusive, and next-generation Human–Computer Interaction technologies.

  1. Pavlovic, V. I., Sharma, R., & Huang, T. S., “Visual interpretation of hand gestures for human– computer interaction: A review,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 677–695, 1997.   2. Mitra, S., & Acharya, T., “Gesture recognition: A survey,” IEEE Transactions on Systems, Man, and Cybernetics, vol. 37, no. 3, pp. 311–324, 2007.   3. Turk, M., & Pentland, A., “Face recognition using eigenfaces,” Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 586–591, 1991.   4. Viola, P., & Jones, M., “Rapid object detection using a boosted cascade of simple features,” IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 511–518, 2001.   5. Dalal, N., & Triggs, B., “Histograms of oriented gradients for human detection,” IEEE Conference on Computer Vision and Pattern Recognition, pp. 886–893, 2005.   6. Zhang, Z., “A flexible new technique for camera calibration,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 11, pp. 1330–1334, 2000.   7. Shotton, J., et al., “Real-time human pose recognition in parts from single depth images,” IEEE Conference on Computer Vision and Pattern Recognition, pp. 1297–1304, 2011.   8. Starner, T., & Pentland, A., “Real-time American Sign Language recognition from video using hidden Markov models,” International Symposium on Computer Vision, pp. 265–270, 1995.   9. Wachs, J. P., Kölsch, M., Stern, H., & Edan, Y., “Vision-based hand-gesture applications,” Communications of the ACM, vol. 54, no. 2, pp. 60–71, 2011.   10. Szeliski, R., Computer Vision: Algorithms and Applications, Springer, 2010.