Evaluating Subjective Answers with Machine Learning and Natural Language Processing
Varsha Pandagre, Deepali Hajare, Rupesh Suryawanshi, Yash Patil, Pratik Pawar , Suvam Kumar Verma
Department of Artificial Intelligence and Data Science
Dr. D .Y. Patil Institute of Engineering, Management and Research
Akurdi, Pune, India
{suryawanshirupesh25, yrajpatil2002, pawarpratik1932, vksuvam}@gmail.com
{varsha.pandagre, deepali.hajare}@dypiemr.ac.in
Abstract— Evaluating papers with subjective answers can be both challenging and exhausting when done manually. One of the biggest hurdles in analyzing subjective papers through Artificial Intelligence (AI) is the limited understanding and acceptance of data. Many existing AI approaches for scoring students’ answers rely on simplistic metrics such as word counts or specific keywords. Additionally, the dearth of curated datasets for training such AI models further complicates the process. Most previous methods in this domain have utilized predefined solution keys containing expected answers to questions. However, it’s crucial to note that these solution keys might themselves contain inaccuracies or contextually incorrect content, leading to sub-optimal evaluations of students’ responses. The proposed methodology addresses these challenges by employing a Large Language Model (LLM) to generate answers based on a relevant textbook. This LLM, equipped with the contextual knowledge from the textbook, forms responses to given questions. To assess student answers, we introduce a novel approach involving the comparison of embeddings between the student’s response and the LLM-generated answer, utilizing cosine similarity. The resulting similarity score serves as a metric for determining the quality of the student’s response. Furthermore, to implement our solution, we leverage the Langchain framework, ensuring a robust and efficient framework for the evaluation process..
Keywords– Subjective Answer Evaluation, Large Language Model (LLM), Cosine Similarity, Machine Learning, Langchain.
