QUALTAN: A Governed Agentic Architecture for Specification-Driven and Execution-Grounded Autonomous Software Testing

This paper presents a protocol for a controlled study of governed, execution-grounded agentic testing. Its central proposition is that LLM reasoning becomes more useful for software quality when it is constrained by specifications, canonical artifacts, deterministic execution, evidence collection, and intent-preserving repair. The protocol pre-specifies the research questions, hypotheses, conditions, metrics, statistical tests, and falsification criteria under which that proposition will be judged. The results of the study will determine whether the proposition holds.

[1] C. Augusto, A. Bertolino, G. De Angelis, F. Lonetti, and J. Morán, “Large Language Models for Software Testing: A Research Roadmap,” arXiv:2509.25043, 2025. [2] D. Huang, J. M. Zhang, M. Harman, Q. Zhang, M. Du, and S.-K. Ng, “Benchmarking LLMs for Unit Test Generation from Real-World Functions,” arXiv:2508.00408, 2025. [3] M. Schäfer, S. Nadi, A. Eghbali, and F. Tip, “An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation,” IEEE Transactions on Software Engineering, 2024. [4] C. Lemieux, J. P. Inala, S. K. Lahiri, and S. Sen, “CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models,” in Proc. 45th IEEE/ACM International Conference on Software Engineering (ICSE), 2023. [5] N. Alshahwan, J. Chheda, A. Finegenova, B. Gokkaya, M. Harman, I. Harper, A. Marginean, S. Sengupta, and E. Wang, “Automated Unit Test Improvement using Large Language Models at Meta,” in Proc. 32nd ACM Symposium on the Foundations of Software Engineering (FSE) Industry Track, 2024. [6] J. Wang, Y. Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, “Software Testing with Large Language Models: Survey, Landscape, and Vision,” IEEE Transactions on Software Engineering, 2024. [7] Y. Jia and M. Harman, “An Analysis and Survey of the Development of Mutation Testing,” IEEE Transactions on Software Engineering, vol. 37, no. 5, 2011. [8] A. Arcuri and L. Briand, “A Practical Guide for Using Statistical Tests to Assess Randomized Algorithms in Software Engineering,” in Proc. 33rd International Conference on Software Engineering (ICSE), 2011. [9] A. Vargha and H. D. Delaney, “A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong,” Journal of Educational and Behavioral Statistics, vol. 25, no. 2, 2000. [10] S. Holm, “A Simple Sequentially Rejective Multiple Test Procedure,” Scandinavian Journal of Statistics, vol. 6, no. 2, 1979.