A Domain-Aware, Explainable Hybrid MLP-LSTM Framework for Scalable Network Intrusion Detection with Cross-Domain Generalization

Network Intrusion Detection Systems (NIDS) must be able to process high volume, high dimensional traffic, and be understandable by security analysts, and be capable of adapting to new networks. Current hybrid deep learning IDSs generally show high accuracy on a single benchmark dataset, and do not always provide a theoretical basis for feature selection based on the knowledge of network-security, seldom perform a generalization analysis on datasets collected under different conditions, and tend to view explainability as an after-thought tool rather than as an analysis tool. In this paper, a domain-aware, explainable hybrid Multi-Layer Perceptron-Long Short-Term Memory (MLP-LSTM) approach for scalable network intrusion detection is proposed. To handle large amounts of traffic, raw flow-level traffic records are fed into and cleaned with a distributed data processing pipeline (Apache Spark). Features are clustered into four sets based on their motivation for network security: flow-based, protocol-based, time-based and payload-based, and filtered for duplication within each set of features, not across all features in an undifferentiated correlation cut. This produces a feature set which is fed to a hybrid architecture comprising of an MLP branch to model static flow statistics and an LSTM branch to model the temporal traffic behaviour, with the two branches combined using a learned fusion layer rather than simple concatenation and provided with an attention-branch mechanism. SHapley Additive exPlanations (SHAP) are used to explain model decisions, and feature importance is aggregated by group and by attack-type, both of which are domain-aware, to generate security-analyst-relevant explanations. The group-level SHAP analysis points in the direction of further analysis, as it is found that the output of the trained model is practically entirely driven by the flow-based features, with 33 of a total 35 non-flow based features having exactly zero attribution despite high variance in the features themselves. The framework is tested in-domain and, to test generalization, without retraining in a cross-domain scenario, where the model is trained on a first intrusion detection dataset and tested in a second, separately collected dataset. For the proposed model, accuracy of 99.81% in domain and macro-F1 score of 99.82% are achieved over seven traffic categories with CICIDS2017 (Domain A). However, when evaluated without retraining on CSE-CIC-IDS2018 (Domain B), the accuracy drops to 90.46% but macro-F1 drops significantly to 32.22% with a per-class breakdown of this drop almost entirely from a complete failure to detect DDoS traffic cross-domain (0.00% recall on 36,641 true DDoS samples, all classified as DoS or Normal Traffic) while Normal Traffic is still detected well (97.61% recall). The proposed fusion mechanism is evaluated using ablation study, with attention-based fusion achieving 99.82% macro-F1, which is higher than 99.48% macro-F1 of the otherwise identical plain-concatenation hybrid, 99.44% macro-F1 of LSTM-only branch and 96.69% macro-F1 of MLP-only branch. Experiments with data volume scaling demonstrate that inference time grows sub-linearly with the number of training samples (from 10% to 100% of the data, it changes from 6.8s to 37.8s), but remains near-constant per sample (from 0.18-0.20 ms per sample), suggesting it is suitable for near-real-time deployment. The results show that the proposed attention-fusion hybrid model can be very effective for in-domain accuracy, but it is difficult to achieve an effective cross-domain generalization performance when using hybrid MLP-LSTM models for intrusion detection, which highlights the needs for testing cross-domain generalization performance in intrusion detection beyond an in-domain setting. The proposed approach introduces a network-security-based feature engineering method, an attention-based fusion method for hybrid spatial-temporal intrusion detection, a group-and-attack-type explainability analysis, and the experimental results of cross-domain generalization for hybrid MLP-LSTM intrusion detection systems.