AI-Driven Malware Obfuscation Detection and Defence System Using Ensemble Machine Learning and Byte-Level DNA Fingerprinting

  In this paper, MalDex AI is introduced — a web application to detect malware that is based exclusively on tackling the issue of obfuscation. It uses a combination of an RF + LSTM ensemble which can detect 100% of obfuscations on artificial data, alongside five specific tools dedicated to the identification of malware obfuscation: an entropy score, packer signatures, obfuscation types classification by name, suspicious strings scanning, and byte bigram DNA. The inclusion of a detector of attacks against the system and of an automatic MITRE ATTACK kill chain reconstructing tool sets it apart from all freely available systems. Testing was performed on FACT_Broucher_2026.pdf, and it showed a live detection with model scores matching those of a medium risk behavioural profile. Three directions will be pursued in future research. First, the models should be trained with an actual malware binary database from public resources for achieving realistic performance results. Secondly, instead of the current fake behavioural data generation approach, we can implement a true sandbox environment for malware execution using Docker containers. Finally, it is proposed to use the DNA-fingerprinting algorithm for cross file similarity searches that would allow to determine whether a newly uploaded file is related to any previous analysed samples within the same cyberattack campaign.

[1] AV-TEST Institute 2024, 'IT Security Report — Malware Statistics 2024', AV-TEST GmbH, viewed June 2026, <https://www.av-test.org>. [2] Gibert, D, Mateu, C and Planes, J 2020, 'The rise of machine learning for detection and classification of malware: Research developments, trends and challenges', Journal of Network and Computer Applications, vol. 153, p. 102526. [3] Aslan, O and Yilmaz, A.A 2021, 'A new malware classification framework based on deep learning algorithms', IEEE Access, vol. 9, pp. 87936-87951. [4] Ma, Y.W, Chen, J.L, Kuo, W.H and Chen, Y.C 2022, 'AI@nti-Malware: An intelligent framework for defending against malware attacks', Journal of Information Security and Applications, vol. 65, p. 103092. [5] Rokach, L 2019, 'Dynamic malware analysis in the modern era — A state of the art survey', ACM Computing Surveys, vol. 52, no. 5, pp. 1-48. [6] Lakshmi, T.S, Govindarajan, M and Sreenivasulu, A 2022, 'Malware visual resemblance analysis with minimum losses using Siamese neural networks', Theoretical Computer Science. [7] VirusTotal 2024, 'About VirusTotal', Google LLC, viewed June 2026, <https://www.virustotal.com/gui/about>. [8] Jamalpur, S, Navya, Y.S, Raja, P, Tagore, G and Reddy, G.R.K 2018, 'Dynamic malware analysis using Cuckoo Sandbox', Proc. 2018 ICSCCC, IEEE, pp. 1-4. [9] CrowdStrike 2024, 'Falcon Platform Overview', CrowdStrike Inc., viewed June 2026, <https://www.crowdstrike.com/products/>. [10] Maniriho, P, Mahmood, A.N and Chowdhury, M.J.M 2024, 'A Survey of Recent Advances in Deep Learning Models for Detecting Malware in Desktop and Mobile Platforms', ACM Computing Surveys, vol. 56, no. 1, pp. 1-41. [11] MITRE Corporation 2024, 'ATTACK for Enterprise — Techniques', MITRE ATTACK Knowledge Base, viewed June 2026, <https://attack.mitre.org>. [12] Cybersecurity Ventures 2024, 'Cybercrime To Cost The World $10.5 Trillion Annually By 2025', Cybersecurity Ventures Annual Report.