Data Duplication Removal Using Machine Learning

Data Duplication Removal Using Machine Learning

Nikhil Kumar .K , Ganesh Kumar .G , Hemanth  Kumar .V  , Jivesh Kumar , Vishal Kumar  ,

Dinkar Kumar, M. Sathish Kumar

Student and Assistant Professor of1Excel Engineering College, NH-544, Salem Main Road, Sankari West, Pallakkapalayam, Pin:637 303. Komarapalayam Namakkal Dt.

3Anna University, Chennai ,Tamilnadu

Abstract

This paper introduces a novel concept termed near-duplicate datasets, which represent quasi-duplicate versions of original datasets subjected to unknown modifications such as row and column insertions and deletions. These datasets hold significant importance for various data-related tasks including exploration, integration, and quality assessment. The paper proposes an innovative method to detect near-duplicate datasets, leveraging features extraction and machine learning techniques. Unlike conventional approaches relying on direct column comparisons, this method focuses on comparing metadata vectors summarizing the datasets, thus offering a distinct and effective means of detection. Additionally, the paper presents a methodology for artificially generating training data to facilitate algorithm training. Through extensive experimentation, the paper identifies optimal parameters for training data creation and evaluates the performance of multiple classifiers. Notably, the results demonstrate an accuracy rate exceeding 95%, underscoring the efficacy of the proposed approach in effectively identifying quasi-duplicate datasets, thus offering valuable insights into data quality and integrity.