Data Duplication Removal Using Machine Learning

This paper introduces a novel concept termed near-duplicate datasets, which represent quasi-duplicate versions of original datasets subjected to unknown modifications such as row and column insertions and deletions. These datasets hold significant importance for various data-related tasks including exploration, integration, and quality assessment. The paper proposes an innovative method to detect near-duplicate datasets, leveraging features extraction and machine learning techniques. Unlike conventional approaches relying on direct column comparisons, this method focuses on comparing metadata vectors summarizing the datasets, thus offering a distinct and effective means of detection. Additionally, the paper presents a methodology for artificially generating training data to facilitate algorithm training. Through extensive experimentation, the paper identifies optimal parameters for training data creation and evaluates the performance of multiple classifiers. Notably, the results demonstrate an accuracy rate exceeding 95%, underscoring the efficacy of the proposed approach in effectively identifying quasi-duplicate datasets, thus offering valuable insights into data quality and integrity.