Machine Learning for Malware Analysis and Detection

This paper presented the design, implementation, and empirical evaluation of a hybrid CNN-Attention framework for multi-class Windows PE32 malware family classification using static IAT-based features. The system employs a five-stage data pipeline including statistical feature engineering, variance-based filtering, and SMOTE-constrained class balancing, and evaluates the proposed model against three baseline classifiers across five complementary metrics. The CNN-Attention model achieved a weighted F1-score of 0.50 and cross-validation mean accuracy of 0.579, outperforming SVM and Naive Bayes on weighted metrics. The self-attention mechanism identified CertCreateContext, AreFileApisANSI, and AddAtom as the most influential API features, providing operationally interpretable classification evidence aligned with established cybersecurity behavioural indicators. Experimental analysis confirms that class imbalance and limited dataset size are the primary constraints on minority-class generalisation, with all models showing macro F1 below 0.45. Future research directions include: (i) dataset expansion with larger and more balanced PE32 malware repositories from threat intelligence sources to provide sufficient training signal for all eight malware families; (ii) GAN-based minority sample generation as a complementary approach to SMOTE for the most underrepresented families (Miner, Keylogger); (iii) hybrid static-dynamic feature fusion combining IAT features with runtime system-call sequences for improved detection of obfuscated and packing-resistant malware; (iv) transformer-based architectures with multi-head attention for larger-scale API sequence modelling; (v) adversarial robustness evaluation testing the model against API suppression and randomisation evasion techniques; and (vi) Explainable AI (XAI) integration with SHAP or Grad-CAM visualisation to complement attention-weight interpretability with class-specific attribution maps for security forensic workflows.

[1]  M. F. Imran, "Malware Classification Using SVM and XGBoost: A Study of the Influence of Features," J. Al-Qadisiyah for Computer Science and Mathematics, vol. 17, no. 4, 2025. [2]  M. S. Akhtar and T. Feng, "Malware Analysis and Detection Using Machine Learning Algorithms," Symmetry, vol. 14, no. 11, p. 2304, 2022. [3]  S. Kumar, B. Janet, and S. Neelakantan, "Malware Classification Using Machine Learning Models," Procedia Computer Science, vol. 235, pp. 1419–1428, 2024. [4] L. El Neel, A. Copiaco, W. Obaid, and H. Mukhtar, "Comparison of Feature Extraction and Classification Techniques of PE Malware," in Proc. IEEE ICSPIS, 2022, pp. 1–6. [5]  B. Panda, S. S. Bisoyi, S. Panigrahy, and P. Mohanty, "Machine Learning Techniques for Imbalanced Multiclass Malware Classification Through Adaptive Feature Selection," PeerJ Computer Science, vol. 11, p. e2752, 2025. [6]  N. O. Matsobane and S. Mokwena, "Malware Detection Using a Random Forest Method Trained on a Balanced Synthetic Dataset," Science, Engineering and Technology, vol. 5, no. 1, 2025. [7] D. Dang, D. Troia, and M. Stamp, "Malware Classification Using Long Short-Term Memory Models," arXiv:2103.02746, 2021. [8]  S. Kumar, B. Janet, and S. Neelakantan, "IMCNN: Intelligent Malware Classification Using Deep Convolution Neural Networks as Transfer Learning and Ensemble Learning," Computer Communications, 2024. [9]  N. Emmanuella, O. E. Oluwaseyi, and F. Adamu-Fika, "Malware Classification Using a Hybrid CNN-Autoencoder and Transformer Encoded Approach," Int. J. Machine Learning, vol. 15, no. 4, 2025. [10]  L. Yao, H. Zhu, and Y. Xin, "Res2NeXt with Attention Mechanisms for Malware Classification Based on Feature Visualization," J. Information Security and Applications, vol. 95, p. 104271, 2025.