A Self-Adaptive Autonomous Pipeline Optimization Framework for Data Quality, Drift Detection, Anomaly Monitoring, and Runtime Performance Improvement

This study presented SAAPO as a self-adaptive and autonomous framework for improving the quality, efficiency, and reliability of data engineering pipelines operating under changing data and workload conditions. By integrating adaptive preprocessing, data-quality assessment, Kolmogorov-Smirnov-based drift detection, Isolation Forest anomaly identification, runtime monitoring, health-score evaluation, and feedback-driven reconfiguration, the framework provides a closed-loop mechanism for detecting degradation and dynamically adjusting pipeline configurations. Experimental evaluation demonstrated that SAAPO can improve throughput, reduce processing latency and resource utilization, control drift, and recover pipeline health more effectively than static and monitoring-only configurations. The implemented prototype also showed the value of configuration acceptance and rollback mechanisms for preventing ineffective adaptations from being permanently applied. Future research should validate SAAPO using complete real-world streaming datasets and production-scale distributed environments such as Apache Spark, Kafka, Flink, and cloud-native data platforms. Further work may incorporate reinforcement learning, Bayesian optimization, multi-objective optimization, adaptive threshold estimation, concept-drift localization, predictive resource allocation, and explainable decision mechanisms. Evaluation across heterogeneous datasets, long-duration workloads, cloud cost constraints, and multi-node environments would further establish the framework's scalability, generalizability, and practical applicability.

[1] J. Xu, H. Wu, J. Wang, and M. Long, “Anomaly Transformer: Time series anomaly detection with association discrepancy,” in Proc. Int. Conf. Learn. Representations (ICLR), Virtual, Apr. 25–29, 2022. [2] C. Velasco-Gallego and I. Lazakis, “Real-time data-driven anomaly detection for marine machinery: A deep learning approach,” Ocean Eng., vol. 256, Art. no. 111511, 2022, doi: 10.1016/j.oceaneng.2022.111511. [3] Q. Liang, K. E. Knutsen, E. Vanem, H. Zhang, and V. Æsøy, “Unsupervised anomaly detection in marine diesel engines using transformer neural networks and residual analysis,” Int. J. Progn. Health Manag., vol. 15, pp. 1–18, 2024. [4] A. Haque and H. Soliman, “A transformer-based autoencoder with Isolation Forest and XGBoost for malfunction and intrusion detection,” Future Internet, vol. 17, Art. no. 164, 2025, doi: 10.3390/fi17040164. [5] H. Xu, G. Pang, Y. Wang, and Y. Wang, “Deep Isolation Forest for anomaly detection,” IEEE Trans. Knowl. Data Eng., vol. 35, pp. 12591–12604, 2023, doi: 10.1109/TKDE.2023.3270293. [6] J. Lu, A. Liu, F. Dong, F. Gu, J. Gama, and G. Zhang, “Learning under concept drift: A review,” IEEE Trans. Knowl. Data Eng., vol. 31, pp. 2346–2363, 2019, doi: 10.1109/TKDE.2018.2876857. [7] N. Kodakandla, “Data drift detection and mitigation: A comprehensive MLOps approach for real-time systems,” Int. J. Sci. Res. Arch., vol. 12, pp. 3127–3139, 2024, doi: 10.30574/ijsra.2024.12.1.0724. [8] A. Bifet and R. Gavaldà, “Learning from time-changing data with adaptive windowing,” in Proc. 2007 SIAM Int. Conf. Data Mining (SDM), Minneapolis, MN, USA, Apr. 26–28, 2007, pp. 443–448, doi: 10.1137/1.9781611972771.42. [9] W. M. Ashraf, T. Ansar, F. Ahmed, J. Hussain, M. M. Abbas, and V. Dua, “From drift to adaptation to the failed ML model: Transfer learning in industrial MLOps,” arXiv preprint arXiv:2602.00957, 2026, doi: 10.48550/arXiv.2602.00957. [10] R. Wang, H. Chen, and C. Guan, “Bi-AAE: A binary adversarial autoencoder deep neural network for health condition monitoring of marine diesel engines,” Ocean Eng., vol. 312, Art. no. 119275, 2024, doi: 10.1016/j.oceaneng.2024.119275.