AI-Based Data Quality Monitoring and Anomaly Detection

This paper presented an AI-based framework for continuous data quality monitoring and anomaly detection that combines statistical, tree-based, clustering, and deep learning techniques into a hybrid ensemble with human-in-the-loop feedback. Experiments on structured data with injected anomalies show the hybrid ensemble clearly outperforming both traditional rule-based validation and single-model machine learning approaches, reaching an F1-score of 0.90 against 0.59 for the statistical baseline. These findings support a broader move away from static, rule-based data quality checks and toward adaptive, learning-based monitoring systems. Future work will push the framework further in several directions: bringing in explainable AI techniques (such as SHAP-based feature attribution) to make anomaly scores easier for non-technical data stewards to interpret; extending evaluation to real-world, naturally occurring anomalies across multiple industry datasets rather than relying solely on synthetic injection; exploring self-healing pipelines that can automatically propose or apply corrective actions (such as imputation or deduplication) for high-confidence anomalies; and investigating federated or privacy-preserving variants of the framework for organizations that cannot centralize sensitive data for model training.

1. Battani, C., & Scannapieco, M. (2016). Data and Information Quality: Dimensions, Principles and Techniques. Springer. 2. Breunig, M. M., Kriegel, H. P., Ng, R. T., & Sander, J. (2000). LOF: Identifying density-based local outliers. Proceedings of the ACM SIGMOD International Conference on Management of Data, 93–104. 3. Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly detection: A survey. ACM Computing Surveys, 41(3), 1–58. 4. Ester, M., Kriegel, H. P., Sander, J., & Xu, X. (1996). A density-based algorithm for discovering clusters in large spatial databases with noise. Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, 226–231. 5. Liu, F. T., Ting, K. M., & Zhou, Z. H. (2008). Isolation forest. Proceedings of the Eighth IEEE International Conference on Data Mining, 413–422. 6. Malhotra, P., Ramakrishnan, A., Anand, G., Vig, L., Agarwal, P., & Shroff, G. (2016). LSTM-based encoder-decoder for multi-sensor anomaly detection. arXiv preprint arXiv:1607.00148. 7. Redman, T. C. (1998). The impact of poor data quality on the typical enterprise. Communications of the ACM, 41(2), 79–82. 8. Sakr, S., & Elgammal, A. (2016). Towards a comprehensive data analytics framework for smart healthcare services. Big Data Research, 4, 44–58. 9. Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., ... & Dennison, D. (2015). Hidden technical debt in machine learning systems. Advances in Neural Information Processing Systems, 28. 10. Wang, R. Y., & Strong, D. M. (1996). Beyond accuracy: What data quality means to data consumers. Journal of Management Information Systems, 12(4), 5–33.