Math Sketch Interpretation Engine Using Latex
Ms. Saba Noor Ayesha1, P Maithri2, VVS Chaitanya3, B Surya Kiran4
1Assistant Professor, Department of Computer Science and Machine Learning,
Methodist College of Engineering and Technology Abids, Hyderabad, Telangana, 500001, India.
2,3,4Student , Department of Artificial Intelligence and Data Science,
Methodist College of Engineering and Technology Abids, Hyderabad, Telangana, 500001, India.
Abstract – The problem of recognizing handwritten mathematical expressions (HMER), which is one of the important issues in the field of computer vision and image processing, still needs to be addressed. The Optical Character Recognition (OCR) sys-tem, which is normally used for this purpose, is found to be inappropriate for the recognition of the two-dimensional relationships of the handwritings such as nested fractions, sub-scripts, and superscripts in the mathematical expressions. In this paper, a new intelligent multimodal system, namely the "Math Sketch Interpretation Engine," which can transform the user's sketches of math problems into machine-understandable LaTeX format and the results of the computations in real-time, is proposed. The proposed system utilizes multimodal encoder-decoder with cross-attention. The pro-posed system is able to recognize complex mathematical expressions with high structural accuracy by utilizing the com-parison of the location of visual symbols and meanings of symbols. In order for the proposed system to recognize all kinds of handwrittings, the proposed system utilizes a special preprocessing technique called adaptive threshold. From the experiment results, it is shown that the proposed system is able to recognize complex arithmetic and algebraic expressions. The proposed system is able to develop the "Sketch-to-Solution" tool for digital learning. [3][11][15].
Key Words: Handwritten Mathematical Expression Recognition (HMER), Image-to-LaTeX Conversion, pix2tex Architecture, CNN-based Feature Extraction
