ScribbleAI using OpenCV and MediaPipe

ScribbleAI using OpenCV and MediaPipe

Prof. Pradnya Kulkarni1, Siddharth Lahoti2,

Siddharth Marne3, Kumar Rakshit Singh4, Khan Almaaz Ateeq5

(Computer engineering, Sinhgad Academy of Engineering, Pune)

ABSTRACT

Human-computer interaction is evolving rapidly with the integration of vision-based systems, enabling intuitive and touch-free interfaces. One such advancement is hand tracking, which allows users to control or interact with digital environments through natural gestures. This project introduces ScribbleAI, a creative and interactive tool that enables users to draw or write virtually in real time using only hand movements. Utilizing OpenCV for image processing and MediaPipe for precise hand landmark detection, the system captures finger motions to translate them into on-screen drawings without the need for external devices like a stylus or touchscreen.

What makes ScribbleAI unique is its combination of computer vision, gesture recognition, and real-time drawing, providing a fun, educational, and accessible platform. The system identifies the index fingertip to act as a pen, allowing dynamic freehand sketching by tracking its trajectory. The project integrates features such as virtual color palette selection, gesture-based commands for clearing the screen, and adaptive stroke thickness, all controlled through specific hand signs. These intuitive controls reduce complexity and enhance the overall user experience.

Moreover, the application supports a mode-switching mechanism where gestures can toggle between writing, erasing, and pausing. This eliminates the need for physical buttons and encourages a seamless workflow. The use of OpenCV ensures fast and efficient frame analysis, while MediaPipe’s pre-trained hand detection model provides accurate tracking, even in varied lighting conditions or backgrounds. With minimal hardware requirements—a basic webcam and a standard computing setup—the tool remains lightweight and broadly accessible.

This fusion of real-time vision processing and gesture-based control aims to make digital art and handwritten input more approachable, especially for children, hobbyists, or individuals with motor constraints. Preliminary testing demonstrates high responsiveness, minimal latency, and engaging interaction. Ultimately, ScribbleAI exemplifies how AI and computer vision can foster creativity through non-traditional input methods and build smarter, more inclusive interfaces for users of all backgrounds.

The integration of real-time hand tracking with intuitive visual feedback creates exciting possibilities in fields like education, design, and virtual collaboration. With further refinement, ScribbleAI can evolve into a versatile platform for both creative expression and functional gesture-driven applications.

Index Terms—Hand Tracking, OpenCV, Computer Vision, Gesture Recognition, Real-Time Drawing.