Automatic Detection of Software Defects Using Machine Learning

This paper has presented a comprehensive survey of machine learning approaches to automatic software defect detection, spanning the full spectrum from early statistical methods to state-of-the-art deep learning architectures. Our analysis has traced the evolution of the field from simple metric-based classifiers to sophisticated graph neural networks and pre-trained transformer models capable of learning rich, multi-scale representations of program structure and semantics. The empirical evidence consistently demonstrates that deep learning approaches, and particularly graph-based and transformer-based models, achieve superior defect detection performance compared to classical ML methods. The pre-train/fine-tune paradigm has emerged as a particularly effective strategy, enabling high-accuracy detection with limited labeled training data by leveraging knowledge acquired from large unlabeled code corpora. Feature representation remains a crucial determinant of model performance, with graph-based representations encoding program structure and data flow outperforming simpler metric and token-based approaches. Despite these impressive advances, significant challenges remain before ML-based defect detection can be fully integrated into mainstream software development practice. Cross-project generalization, model interpretability, data quality, and adaptability to evolving codebases are pressing open problems that require sustained research attention. The emergence of large language models introduces both new opportunities and new challenges, particularly around reliability and uncertainty quantification. The long-term vision of this research agenda is the development of AI-assisted software development environments that seamlessly and continuously monitor code quality, flag potential defects at the point of authorship, and provide actionable repair suggestions grounded in natural language explanations. Realizing this vision will require advances not only in ML modeling but also in software engineering methodology, human-computer interaction, and empirical evaluation practice. We hope that this survey serves as a valuable reference for researchers and practitioners working toward this important goal.

[1] Akiyama, F. (1971). An example of software system debugging. Proceedings of IFIP Congress, 71, 353–359. [2] Basili, V. R., Briand, L. C., & Melo, W. L. (1996). A validation of object-oriented design metrics as quality indicators. IEEE Transactions on Software Engineering, 22(10), 751–761. [3] Catal, C., & Diri, B. (2009). A systematic review of software fault prediction studies. Expert Systems with Applications, 36(4), 7346–7354. [4] Chakraborty, S., Krishna, R., Ding, Y., & Ray, B. (2022). Deep learning based vulnerability detection: Are we there yet? IEEE Transactions on Software Engineering, 48(9), 3280–3296. [5] Dam, H. K., Pham, T., Ng, S. W., Tran, T., Grundy, J., Ghose, A., Kim, T., & Kim, C. J. (2018). A deep tree-based model for software defect prediction. arXiv preprint arXiv:1802.00921. [6] Fan, J., Li, Y., Wang, S., & Nguyen, T. N. (2020). A C/C++ code vulnerability dataset with code changes and CVE summaries. Proceedings of MSR 2020, 508–512. [7] Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., ... & Liu, T. (2020). CodeBERT: A pre-trained model for programming and natural languages. Findings of EMNLP 2020, 1536–1547. [8] Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Shujie, L., ... & Liu, T. (2021). GraphCodeBERT: Pre-training code representations with data flow. ICLR 2021. [9] Hall, T., Beecham, S., Bowes, D., Gray, D., & Counsell, S. (2012). A systematic literature review on fault prediction performance in software engineering. IEEE Transactions on Software Engineering, 38(6), 1276–1304. [10] Li, Z., Zou, D., Xu, S., Ou, X., Jin, H., Wang, S., ... & Zhong, J. (2018). VulDeePecker: A deep learning-based system for vulnerability detection. Proceedings of NDSS 2018.