An Extensive Comparative Analysis of Document Similarity Algorithms: A Systematic Survey and Evaluation

An Extensive Comparative Analysis of Document Similarity Algorithms:

A Systematic Survey and Evaluation

Mr. Gaurabh G. Sawal, M.E. (ITE, GEC) and Dr. Nilesh B. Fal Dessai,

Head of Department (ITE, GEC)

Goa Engineering College (GEC), Farmagudi, 403401, Goa, India.

Contributing authors: gaurabh.gec@gmail.com

Abstract:

Document similarity analysis plays a crucial role in various natural language processing (NLP) tasks such as information retrieval, text classification, and recommendation systems. This research paper presents a comprehensive survey and comparison of five prominent techniques for document similarity: BERT, GloVe, USE, Doc2Vec, and LDA. The paper begins by providing an overview of each technique and their underlying principles. BERT, a transformer-based model, captures contextual word representations, while GloVe generates dense word embeddings based on global co-occurrence statistics. USE offers sentence-level embeddings that capture semantic information, and Doc2Vec extends the concept to document-level embeddings. LDA, a probabilistic topic modelling technique, assigns topics to documents based on word distributions. The survey focuses on evaluating these techniques in terms of their performance, applicability, and computational efficiency for document similarity tasks. The results of the comparative analysis highlight the strengths and limitations of each technique. BERT excels in capturing fine-grained contextual information, while GloVe is effective in capturing global semantic relationships. USE provides reliable sentence-level embeddings, and Doc2Vec captures document-level semantics. LDA offers a probabilistic approach to modelling document topics. By providing a comprehensive survey and comparison of these document similarity techniques, this research paper aims to assist researchers and practitioners in selecting the most suitable technique for their specific NLP tasks.

Keywords: BERT, GloVe, USE, Doc2Vec, LDA