Intelligent Multi-Source Data Integration Framework Using Machine Learning for Schema Matching and Data Transformation

This paper presents the design of an Intelligent Multi-Source Data Integration Framework Using Machine Learning for Schema Matching and Data Transformation. The proposed framework addresses the challenges of integrating heterogeneous structured datasets with inconsistent attribute names, data types, formats, and representations. By combining schema profiling, lexical and statistical analysis, structural information, and optional semantic features, the framework aims to identify corresponding attributes more intelligently than approaches based exclusively on predefined rules. The integration pipeline connects matching predictions with controlled data transformation, quality validation, and reporting.The use of confidence-based decisions and human review provides a mechanism for handling ambiguous mappings, while transformation logs support traceability and reproducibility. The proposed implementation can be developed using Python, Pandas, scikit-learn, Streamlit, and SQLite. Future work may explore transformer-based semantic matching, active learning from user corrections, incremental integration for evolving schemas, and large-scale processing of additional data formats. A systematic comparison with established matching techniques is necessary to quantify the framework's benefits, limitations, and generalization capability.

[1] D. Rodrigues and A. da Silva, “A study on machine learning techniques for the schema matching network problem,” Journal of the Brazilian Computer Society, vol. 27, article 14, 2021. DOI: 10.1186/s13173-021-00119-5. [2] C.-C. Sun and D.-R. Shen, “Mixed Hierarchical Networks for Deep Entity Matching,” Journal of Computer Science and Technology, vol. 36, pp. 822–838, 2021. DOI: 10.1007/s11390-021-1321-0. [3]    E. Krivosheev, M. Atzeni, K. Mirylenka, P. Scotton, C. Miksovic, and A. Zorin, “Business Entity Matching with Siamese Graph Convolutional Networks,” AAAI Conference on Artificial Intelligence, 2021. [4]   A. Bogatu, N. W. Paton, M. Douthwaite, S. Davie, and A. Freitas, “Cost-effective Variational Active Entity Resolution,” IEEE International Conference on Data Engineering (ICDE), 2021. DOI: 10.1109/ICDE51399.2021.00114. [5]    A. Jain, S. Sarawagi, and P. Sen, “Deep Indexed Active Learning for Matching Heterogeneous Entity Representations,” Proceedings of the VLDB Endowment, vol. 15, no. 1, pp. 31–45, 2021. DOI: 10.14778/3485450.3485455. [6]      B. Li, Y. Miao, Y. Wang, Y. Sun, and W. Wang, “Improving the Efficiency and Effectiveness for BERT-based Entity Resolution,” Proceedings of AAAI, vol. 35, no. 15, pp. 13226–13233, 2021. DOI: 10.1609/aaai.v35i15.17562. [7]    P. Wang, W. Zheng, J. Wang, and J. Pei, “Automating Entity Matching Model Development,” IEEE ICDE, 2021. DOI: 10.1109/ICDE51399.2021.00116. [8]   W. Ren, X. Lian, and K. Ghazinour, “Online Topic-Aware Entity Resolution Over Incomplete Data Streams,” SIGMOD 2021, pp. 1478–1490. DOI: 10.1145/3448016.3457238. [9]   N. Barlaug and J. A. Gulla, “Neural Networks for Entity Matching: A Survey,” ACM Transactions on Knowledge Discovery from Data, vol. 15, no. 3, 2021. DOI: 10.1145/3442200. [10]  R. Peeters and C. Bizer, “Supervised Contrastive Learning for Product Matching,” Companion Proceedings of the Web Conference        2022,                  pp.   248–251.              DOI: 10.1145/3487553.3524254.