Loading article...
Loading article...
Stream/Area:
-
Subject:
-
Paper ID:
IJSREM68504
Volume:
10 |
10 |
Oct 2026
Pages:
1-6
Publication Frequency:
Monthly (12 issues per year)
License:International Journal of Scientific Research in Engineering and Management follows the Creative Commons Attribution 4.0 (CC BY 4.0) License, allowing free use and distribution with proper attribution.
School of Computer Science & Engineering
VIT - AP University
School of Computer Science & Engineering
VIT - AP University
School of Computer Science & Engineering
VIT - AP University
School of Computer Science & Engineering
VIT - AP University
School of Computer Science & Engineering
VIT - AP University
Published misinformation detectors often report ac-curacies close to 100%, but these figures usually come from a random split of a single benchmark. We revisit this practice for two forms of misinformation, news text and images, using evaluation protocols that limit data leakage. For text, we fine-tune RoBERTa-base and DeBERTa-v3-base on three news sources derived from ISOT, McIntire and WELFake. Before splitting, we remove exact and MinHash near-duplicates and separate the overlapping datasets, and in doing so we found that WELFake already contains every ISOT and McIntire article. Over 20 training runs, the two transformers reach 0.99–1.00 macro-F1 in-domain and cannot be told apart statistically. Their performance drops to 0.78–0.95 on a news source withheld from training, to about 0.51 on LIAR claims, and below chance on COVID-19 social-media posts (ROC-AUC 0.455–0.469). On these out-of-domain sets the models are strongly overconfident (expected calibration error 0.38–0.56), and their errors change markedly with the random seed. An error analysis and a small controlled style probe indicate that the models respond to writing style rather than to factual truth. For images, we merge CASIA 2.0 tampered photographs with a dataset of AI-generated images into a 45,614-image benchmark and report preliminary single-run baselines. In our setup, every model finds AI-generated images much easier to detect than tampered photographs (ROC-AUC 0.977 versus 0.843 for ResNet50 and 0.975 versus 0.778 for ViT-B/16), although part of this gap may stem from differences between the two datasets and not only from the difficulty of the tasks. On the pooled test set, the ImageNet-pretrained ResNet50 and ViT-B/16 obtain ROC-AUC of 0.954 and 0.946, respectively. In both modalities, pooled or in-domain scores give an inflated view of reliability, and predictions can depend on cues tied to a particular data source. We therefore argue that source-disjoint evaluation, per-source reporting, multiple seeds and explicit shortcut checks should become routine.
KEYWORDS
When evaluated with attention to leakage, deep detectors for both fake news and manipulated images turn out to be far less reliable than their headline scores suggest. Trans-formers that are almost perfect on familiar news sources lose accuracy on unseen sources, fail on short claims and social-media posts, and respond to writing style rather than truth. In our preliminary single-run image baselines, detectors that perform strongly on AI-generated images are clearly weaker on traditional tampering (ROC-AUC 0.977 versus 0.843 for ResNet50 and 0.975 versus 0.778 for ViT-B/16), although this gap may partly reflect differences between the datasets. In future work, we plan to apply donor-aware splitting to CASIA with repeated seeds, run leave-one-generator-out tests, explore style-agnostic training [14] and add evidence retrieval to the text detector.
[1] H. Ahmed, I. Traore, and S. Saad, “Detection of online fake news using n-gram analysis and machine learning techniques,” in Proc. ISDDC, LNCS vol. 10618, Springer, 2017, pp. 127–138. [2] H. Ahmed, I. Traore, and S. Saad, “Detecting opinion spams and fake news using text classification,” Security and Privacy, vol. 1, no. 1, 2018. [3] P. K. Verma, P. Agrawal, I. Amorim, and R. Prodan, “WELFake: Word embedding over linguistic features for fake news detection,” IEEE Trans. Comput. Social Syst., 2021, doi: 10.1109/TCSS.2021.3068519. [4] W. Y. Wang, “‘Liar, liar pants on fire’: A new benchmark dataset for fake news detection,” in Proc. ACL, 2017. [5] P. Patwa et al., “Fighting an infodemic: COVID-19 fake news dataset,” in CONSTRAINT@AAAI, Springer, 2021. [6] Y. Liu et al., “RoBERTa: A robustly optimized BERT pretraining approach,” arXiv:1907.11692, 2019. [7] P. He, J. Gao, and W. Chen, “DeBERTaV3: Improving DeBERTa us-ing ELECTRA-style pre-training with gradient-disentangled embedding sharing,” in Proc. ICLR, 2023. [8] R. Geirhos et al., “Shortcut learning in deep neural networks,” Nature Mach. Intell., vol. 2, 2020. [9] L. Bozarth and C. Budak, “Toward a better performance evaluation framework for fake news classification,” in Proc. ICWSM, vol. 14, 2020,pp. 60–71. [10] C. Blackledge and A. Atapour-Abarghouei, “Transforming fake news: Robust generalisable news classification using transformers,” arXiv:2109.09796, 2021.
Publication Date
10/30/2026
Peer Review Type
Double-Blind Peer Review
Review Rounds: 2
Documents
Article Views
0
Total views since publication
PDF Downloads
8
Total downloads since publication
Geographic Reach
5
Digital Preservation: This article is permanently archived on Zenodo and complies with all data preservation standards across major indexing platforms.
Plagiarism Screening
This manuscript has been checked for plagiarism using TurnitinPlagiarismChecker software.
Conflict of Interest
The authors declare no conflict of interest related to this research work.
Funding Information
No funding information available for this research.
Related research in the same subject area
Loading similar articles...