Fake News Prediction Using NLP

This paper introduces a comprehensive and reproducible pipeline for detecting fake news, utilizing TF-IDF feature extraction along with (1,2)-gram representation and two traditional classifiers: Logistic Regression and Multinomial Naive Bayes. The model was trained and evaluated on a dataset of around 44,898 labeled English-language news articles, using a stratified 80/20 split. Remarkably, Logistic Regression achieved an impressive overall accuracy of 98.6%, with both macro-averaged precision and recall surpassing 0.97 for the fake and real classes. ROC analysis further supports this, showing an AUC of 0.993, which indicates an almost flawless ability to distinguish between the two classes across all classification thresholds. The main takeaway from the ablation studies is that when we expand TF-IDF from just unigrams to include bigrams, we see a notable and consistent boost in accuracy. This approach helps us capture those telling two-word phrases—especially the attribution structures that are typical in wire-service journalism—that unigrams simply miss. By using the class_weight='balanced' parameter in Logistic Regression, we can effectively address minor class imbalances, which leads to balanced precision and recall for both classes. This way, the classifier doesn't just achieve high accuracy by leaning too heavily on the majority class. When we compare Logistic Regression to Naive Bayes, we find that it outperforms it by 5.5 percentage points across all metrics. This reinforces the idea that the conditional independence assumption of Naive Bayes is a significant limitation when dealing with natural language text. Analyzing feature importance gives us clear insights that the model has picked up on real stylistic differences—like the vocabulary used in genuine news versus the sensational and conspiratorial language often found in fake news—rather than just random correlations tied to the dataset. Through error analysis, we’ve pinpointed three distinct failure modes, each suggesting specific areas for future improvement. This work shows that traditional NLP methods are still very effective for detecting fake news and come with practical benefits—like being interpretable, low-cost in terms of computation, quick to process (under 0.1 ms per article), and easy to deploy without needing GPU resources. The entire pipeline can be trained in less than 30 seconds on standard hardware, making it feasible for organizations that don’t have access to extensive machine learning infrastructure. Looking ahead, we expect that enhancements like contextual embeddings, metadata integration, multilingual transfer, continual learning, explainability, and adversarial training will help close the accuracy gap compared to large pre-trained language models, all while keeping these practical benefits intact.

[1] C. Castillo, M. Mendoza, and B. Poblete, "Information credibility on Twitter," in Proc. WWW, Hyderabad, India, 2011, pp. 675–684. [2] A. Zubiaga, M. Liakata, R. Procter, G. W. Hoi, and P. Tolmie, "Analysing how people orient to and spread rumours in social media by looking at conversational threads," PLOS ONE, vol. 11, no. 3, p. e0150989, 2016. [3] V. Pérez-Rosas, B. Kleinberg, A. Lefevre, and R. Mihalcea, "Automatic detection of fake news," in Proc. COLING, Santa Fe, NM, 2018, pp. 3391–3401. [4] A. Agarwal and H. Mittal, "Text classification using machine learning methods—a survey," in Proc. MODELSWARD, 2012, pp. 269–275. [5] W. Y. Wang, "Liar, liar pants on fire: A new benchmark dataset for fake news detection," in Proc. ACL, Vancouver, Canada, 2017, pp. 422–426. [6] L. Breiman, "Random forests," Machine Learning, vol. 45, no. 1, pp. 5–32, 2001. [7] Y. Kula, M. Choraś, R. Kozik, P. Ksieniewicz, and M. Woźniak, "Sentiment analysis for fake news detection by means of neural networks," in Proc. ICCS, 2020, pp. 653–666. [8] B. Horne and S. Adali, "This just in: Fake news packs a lot in title, uses simpler, repetitive content in the text body," in Proc. AAAI ICWSM Workshop, 2017. [9] Y. Kim, "Convolutional neural networks for sentence classification," in Proc. EMNLP, Doha, Qatar, 2014, pp. 1746–1751. [10] A. Vaswani et al., "Attention is all you need," in Proc. NeurIPS, Long Beach, CA, 2017, pp. 5998–6008. [11] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding," in Proc. NAACL-HLT, Minneapolis, MN, 2019, pp. 4171–4186. [12] F. Pedregosa et al., "Scikit-learn: Machine learning in Python," Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011. [13] T. Joachims, "Text categorization with support vector machines: Learning with many relevant features," in Proc. ECML, 1998, pp. 137–142. [14] G. Salton and C. Buckley, "Term-weighting approaches in automatic text retrieval," Information Processing & Management, vol. 24, no. 5, pp. 513–523, 1988. [15] I. Ahmed, P. Traore, and S. Saad, "Detection of online fake news using N-gram analysis and machine learning techniques," in Proc. ISDDC, 2017. [16] R. Socher et al., "Recursive deep models for semantic compositionality over a sentiment treebank," in Proc. EMNLP, 2013, pp. 1631–1642. [17] M. Granik and V. Mesyura, "Fake news detection using naive Bayes classifier," in Proc. UKRPROG, IEEE, Kyiv, Ukraine, 2017, pp. 1–4. [18] S. Shu, A. Sliva, S. Wang, J. Tang, and H. Liu, "Fake news detection on social media: A data mining perspective," ACM SIGKDD Explorations Newsletter, vol. 19, no. 1, pp. 22–36, 2017. [19] X. Zhou and R. Zafarani, "A survey of fake news: Fundamental theories, detection methods, and opportunities," ACM Computing Surveys, vol. 53, no. 5, pp. 1–40, 2020. [20] N. Kaliyar, A. Goswami, P. Narang, and S. Sinha, "FakeBERT: Fake news detection in social media with a BERT-based deep learning approach," Multimedia Tools and Applications, vol. 80, pp. 11765–11788, 2021.