Similar Question Pair

Similar Question Pair

Mr. Niraj Aware, Ms. Prachi Shirbhate, Ms. Shreya Sangale, Ms. Achal Bannore, Prof. A. R. Thakur

Information Technology, Sipna College of Engineering & Technology, Amravati

---------------------------------------------------------------------***---------------------------------------------------------------------

Abstract - Similarity analysis of question pairs is an important task in natural language processing (NLP) and information retrieval. In this study, we aim to compare the performance of a Random Forest classifier using three different techniques for feature representation: Bag of Words (BoW), Term Frequency-Inverse Document Frequency (TF-IDF), and FuzzyWuzzy. We preprocess a dataset of question pairs by tokenizing, lowercasing, and removing stop words and special characters. We use various evaluation metrics such as accuracy, precision, recall, and F1-score to compare the performance of the classifier using BoW, TF-IDF, and FuzzyWuzzy as feature representations. Our experimental results demonstrate that the Random Forest classifier performs well on all three techniques, with TF-IDF showing slightly better performance compared to BoW and FuzzyWuzzy. However, FuzzyWuzzy performs better in handling small dataset sizes and questions with spelling mistakes or word rearrangements. In conclusion, this study provides a comparative analysis of a Random Forest classifier for similarity analysis of question pairs using BoW, TF-IDF, and FuzzyWuzzy as feature representations. The findings can assist researchers and practitioners in selecting the most suitable technique for their specific NLP and information retrieval tasks, considering the strengths and weaknesses of each technique and the characteristics of their dataset.

Key Words:  Similar Questions Pair, TF-IDF, Bag-of-Words, Fuzzy-Wuzzy, Random Forests Algorithm, Information Retrieval, Question Answering, Community Question Answering.