Air Keys: Virtual Keyboard and Mouse Using Computer Vision

In this paper, the Air Keys system was proposed. It is a virtual mouse and keyboard in real time to enable touchless human computer interaction using gestures of the hands and computer vision. The mouse and keyboard functionality was implemented using the hand landmarks offered by MediaPipe in a gesture recognition approach based on rules, using only a simple webcam without requiring any hardware. The approach was tested for 90% gesture recognition accuracy in real time on CPU alone hardware. The proposed approach is a low-cost clean and usable alternative for conventional input devices and thus applicable, for example, in the field of public computing, medical computing, or assistive computing. Future directions could include interaction using two-hand gestures, better low-light environments, and input fusion.

[1]    Mallik, B., Rahim, M.A., Miah, A.S.M., Yun, K.S., Shin, J.: Virtual keyboard: A real-time hand gesture recognition-based character input system using LSTM and MediaPipe Holistic. Computer Systems Science and Engineering 48(2), 555–570 (2024). [2]    Jo, B.J., Kim, S.K.: A study on the classification of hand gesture for mobile virtual reality with MediaPipe. International Journal of Intelligent Systems and Applications in Engineering 12(21s), 3555–3559 (2024).   [3]    Rahim, M.A., Shin, J., Hashem, M.A., Akash, H.S., Hossain, M.I., Miah, A.S.M.: Advanced touchless HCI: A gesture recognition-based multilingual virtual keyboard for character input. Journal of Tianjin University Science and Technology 57(11), 1–20 (2024).   [4]    Wang, Q., Xie, Z.: HandMouse: A gesture-based virtual mouse system. International Journal of Advanced Computer Science and Applications (IJACSA) 15(11), 657–668 (2024).   [5]    Yadav, V.J., Reddy, Y.K., Kumar, P.D., Ramya, C., Kumar, M.S.: Gesture controlled virtual mouse using AI. In: IEEE Conference Proceedings, pp. 1–5 (2024).   [6]    Kavuri, M., Madhumitha, T.S., Nagesh, Pratheek, B.K., Trisha, G.: Hands-free computing: A gesture-controlled virtual AI mouse. Journal of Emerging Technologies and Innovative Research 12(6), 545–550 (2025).   [7]    Shen, J., Boldu, R., Kalla, A., Glueck, M., Surale, H.B., Karlson, A.: RingGesture: A ring-based mid-air gesture typing system powered by a deep-learning word prediction framework. arXiv preprint arXiv:2410.18100 (2024).   [8]    Linardakis, M., Varlamis, I., Papadopoulos, G.Th.: Survey on hand gesture recognition from visual input. IEEE Access 13, 135373–135377 (2025).   [9]    Bisht, D., Pal, A., Banerjea, S.: VKM: A virtual keyboard and mouse solution towards a lightweight computing system. In: 2024 20th International Conference on Distributed Computing in Smart Systems and the Internet of Things (DCOSS-IoT), pp. 254–258. IEEE (2024).   [10]              Meng, Y., Jiang, H., Duan, N., Wen, H.: Real-time hand gesture monitoring model based on MediaPipe’s registerable system. Sensors 24(19), 6262 (2024).