Math Sketch Interpretation Engine Using Latex

The problem of recognizing handwritten mathematical expressions (HMER), which is one of the important issues in the field of computer vision and image processing, still needs to be addressed. The Optical Character Recognition (OCR) sys-tem, which is normally used for this purpose, is found to be inappropriate for the recognition of the two-dimensional relationships of the handwritings such as nested fractions, sub-scripts, and superscripts in the mathematical expressions. In this paper, a new intelligent multimodal system, namely the “Math Sketch Interpretation Engine,” which can transform the user’s sketches of math problems into machine-understandable LaTeX format and the results of the computations in real-time, is proposed. The proposed system utilizes multimodal encoder-decoder with cross-attention. The pro-posed system is able to recognize complex mathematical expressions with high structural accuracy by utilizing the com-parison of the location of visual symbols and meanings of symbols. In order for the proposed system to recognize all kinds of handwrittings, the proposed system utilizes a special preprocessing technique called adaptive threshold. From the experiment results, it is shown that the proposed system is able to recognize complex arithmetic and algebraic expressions. The proposed system is able to develop the “Sketch-to-Solution” tool for digital learning. [3][11][15].