Hand-Gesture Based Mathematic Learning Using AI
Manasi Patil∗, Darshan Gurav†, Tushar Bari‡, Yash Afre§, Ms. Priyanka Medhe¶
∗†‡§UG Students, ¶Associate Professor,
Department of Computer Engineering, SSBT’s College of Engineering and Technology, Jalgaon, Maharashtra, India
Abstract—In this paper, we present an AI-driven, gesture- controlled system that allows users to interact with and solve computer-generated mathematical problems without the need for a mouse, keyboard, or physical effort. Dubbed ”MathVision,” the system leverages contemporary machine learning techniques, including OpenCV for gesture detection and Google Generative AI (Gemini) technology for interpreting mathematical equations, within a streamlined and intuitive interface. A webcam captures hand movements, enabling real-time recognition of gestures for operations such as addition, subtraction, multiplication, and division. These movements are converted into usable formats that users can scroll through, delete, or confirm via specific gestures. The system utilizes a trained model built using tools like cvzone.HandTrackingModule to detect and map hand and finger gestures, incorporating custom mathematical layers and gesture-based input to write expressions.
Designed as an educational software product, MathVision enhances accessibility and interactivity in learning environ- ments—especially for students who face challenges with tradi- tional input methods or find conventional mathematical tools less engaging. Users can either solve exercises manually or input solutions via typing, offering flexibility in problem-solving. Built using a technology stack that includes Python, Streamlit, NumPy, and PIL, the system supports real-time execution and immediate visual feedback. The integration of gesture recognition with AI- based computation not only simplifies human-computer inter- action but also transforms visual manipulations into immediate mathematical responses.
By making mathematical learning more engaging, inclusive, and intuitive, MathVision has the potential to revolutionize mathematical instruction and bridge the gap between human intuition and artificial intelligence in education across all levels.
Index Terms—Generative Artificial Intelligence, Gesture Recognition, Human Computer Interaction, Computer Vision.
