An Automated Evaluation Framework for Large Language Models in 3D Game Environments Using Structural and Behavioral Unit Testing

The deployment of Large Language Models for 3D spatial programming necessitates specialized evaluation pipelines.   By abstracting the evaluation process into Setup, Reference substitution, Structural Testing, and Behavioral Testing, we created a highly reliable programmatic grader. This framework successfully bridges the gap between text-based code generation and real-time game engine execution.   Consequently, the granular rubrics generated by our unit tests offer robust reward signals, proving highly effective for refining LLMs through Reinforcement Learning from Human Feedback (RLHF). Future work will explore automating the generation of the unit tests themselves to further scale the RLHF dataset without manual overhead

[1]             Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., ... & Lowe, R. "Training language models to follow instructions with human feedback." Advances in Neural Information Processing Systems, 35 (2022): 27730-27744. [2]             Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., ... & Zaremba, W. "Evaluating Large Language Models Trained on Code." arXiv preprint arXiv:2107.03374 (2021). [3]             Rozière, M., Lachaux, M. A., Chanussot, L., & Lample, G. "Code Llama: Open Foundation Models for Code." arXiv preprint arXiv:2308.12950 (2023). [4]             Liu, J., Xia, C. S., Wang, Y., & Zhang, L. "Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation." Advances in Neural Information Processing Systems, 36 (2023). [5]             Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., ... & Anandkumar, A. "Voyager: An Open-Ended Embodied Agent with Large Language Models." arXiv preprint arXiv:2305.16291 (2023). [6]             Zheng, L., Chiang, W. L., Ying, Y., Hao, S., ... & Xing, E. P. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." Advances in Neural Information Processing Systems, 36 (2023). [7]             Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., & Yao, S. "Reflexion: Language Agents with Verbal Reinforcement Learning." Advances in Neural Information Processing Systems, 36 (2023). [8]             Zan, D., Chen, B., Zhang, F., Lu, D., Wu, B., ... & Lou, J. G. "Large Language Models Meet NL2Code: A Survey." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (2023). [9]             Cassano, F., Gouwar, J., Nguyen, D., ... & Guha, A. "MultiPL-E: A Scalable and Polyglot Approach to Evaluating Large Language Models for Code." IEEE Transactions on Software Engineering, 49(7) (2023): 3840-3855. [10]          Ammann, P., & Offutt, J. Introduction to Software Testing, 2nd Edition. Cambridge University Press (2016).