Automated Answer Scoring in the Transformer and Large Language Model Era: A PRISMA-Guided Systematic Review of Datasets, Methods, Evaluation Metrics, and Trustworthiness (2021-2026)

Automated Answer Scoring has entered a new phase in which transformer encoders, sentence embeddings and LLMs coexist rather than replace one another. The most convincing evidence supports supervised transformer and hybrid systems for benchmarked scoring, while LLMs offer strong flexibility for low-resource, multilingual and feedback-rich assessment. However, reliability, fairness, explanation fidelity and domain generalization remain unresolved. Therefore, the best future AAS systems will be neither fully manual nor blindly automatic; they will be auditable human-AI assessment systems that reduce workload while preserving educational validity.

1.       Garcia-Varela, F., Lopez, M., & Sanchez, R. (2025). ChatGPT as a stable and fair tool for automated essay scoring. Education Sciences, 15(8), 946. https://doi.org/10.3390/educsci15080946 2.   Gombert, S., Drachsler, H., & Kroehen, U. (2024). From the automated assessment of student essay content to highly informative feedback: A case for trustworthy learning analytics. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-023-00387-6 3.   Grevisse, C. (2024). LLM-based automatic short answer grading in undergraduate medical education. BMC Medical Education, 24, 1060. https://doi.org/10.1186/s12909-024-06026-5 4.   Hahn, M. G., Navarro, S. M. B., de la Fuente Valentin, L., & Burgos, D. (2021). A systematic review of the effects of automatic scoring and automatic feedback in educational settings. IEEE https://doi.org/10.1109/ACCESS.2021.3100890 5.   Kortemeyer, G. (2024). Performance of the pre-trained large language model GPT-4 on automated short answer grading.https://doi.org/10.1007/s44163-024-00147-y 6.   Li, W., & Liu, H. (2024). Applying large language models for automated essay scoring for non-native Japanese. Humanities and Communications, 11,    723. 7.    Ludwig, S., Mayer, C., Hansen, C., Eilers, K., & Brandt, S. (2021). Automated essay scoring using transformer models. Psych, 3(4), 897-915. https://doi.org/10.3390/psych3040056 8.      Mardini, G. I. D., Sanuy, J. P., & Prats, M. A. (2024). A deep-learning-based grading system for reading comprehension short answers. Education and Information Technologies, 29, 1769-1798. https://doi.org/10.1007/s10639-023-11890-7 9.   Maslim, M., Wang, H. C., Putra, C. D., & Prabowo, Y. D. (2024). A trustworthy automated short-answer scoring system using a new dataset and hybrid transfer learning method. International Journal of Interactive Multimedia and Artificial Intelligence, 8(5), 37-45. https://doi.org/10.9781/ijimai.2024.02.003 10.     Memarian, B., & Doleck, T. (2024). A review of assessment for learning with artificial intelligence. Computers in Human Behavior:              Artificial             Humans, 2,             100040.