TRANSFER-LEARNED CONVOLUTIONAL–RECURRENT NETWORKS FOR GENERALISABLE VIDEO DEEPFAKE DETECTION UNDER CROSS-DATASET SHIFT

The present study investigated the generalisability of video deepfake detection using a transfer-learned XceptionNet backbone and convolutional–recurrent extensions evaluated under self-, cross- and multi-domain protocols. The main conclusions are as follows: (1)     Dataset shift is the dominant limitation of learning-based deepfake detectors. Single-source single-frame models lost 20.45–34.97 percentage points when tested on other benchmarks, and the Celeb-DF→DFD pair fell to near-chance accuracy (53.41% with transfer learning), reflecting the opposite visual- and manipulation-quality characteristics of these datasets. (2)     ImageNet transfer learning with staged fine-tuning accelerated convergence and improved accuracy in 18 of 20 single-frame configurations (mean +4.97 percentage points) and in all 25 multi-domain CNN-LSTM configurations (mean +3.19 percentage points), with the largest gains in cross-dataset settings. (3)     Training-source diversity was the most effective measure for generalisation: the combined-distribution model reached 86.49% on the combined test set and 73.20% on the custom test set, and the pooled CNN-LSTM reached 81.54% on unseen blended and in-the-wild samples. (4)     Nadam was the most effective of eight optimisers for all three architectures, whereas Adadelta and Adagrad failed to converge adequately within 70 epochs. (5)     CNN-LSTM achieved the highest accuracy on all four datasets (85.2% on DFD, with precision 0.895, recall 0.797 and F1-score 0.843), reducing false negatives on DFD by 23.5% relative to the frame-level CNN, whereas CNN-GRU did not consistently outperform the baseline.

[1]      I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial nets," in Proc. 27th Int. Conf. Neural Information Processing Systems (NIPS), vol. 2, 2014, pp. 2672–2680. [2]      T. Karras, S. Laine, and T. Aila, "A style-based generator architecture for generative adversarial networks," in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2019, pp. 4396–4405. [3]      J. Thies, M. Zollhöfer, M. Stamminger, C. Theobalt, and M. Nießner, "Face2Face: Real-time face capture and reenactment of RGB videos," in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2016, pp. 2387–2395. [4]      "faceswap-GAN," GitHub repository. [Online]. Available: https://github.com/shaoanlu/faceswap-GAN/ [5]      T. T. Nguyen, C. M. Nguyen, D. T. Nguyen, D. T. Nguyen, and S. Nahavandi, "Deep learning for deepfakes creation and detection," arXiv preprint arXiv:1909.11573, 2019. [6]      F. Matern, C. Riess, and M. Stamminger, "Exploiting visual artifacts to expose deepfakes and face manipulations," in Proc. IEEE Winter Applications of Computer Vision Workshops (WACVW), 2019, pp. 83–92, doi: 10.1109/WACVW.2019.00020. [7]      D. Afchar, V. Nozick, J. Yamagishi, and I. Echizen, "MesoNet: A compact facial video forgery detection network," in Proc. IEEE Int. Workshop on Information Forensics and Security (WIFS), 2018, pp. 1–7. [8]      F. Chollet, "Xception: Deep learning with depthwise separable convolutions," in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2017, pp. 1800–1807. [9]      A. Rössler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nießner, "FaceForensics++: Learning to detect manipulated facial images," in Proc. IEEE/CVF Int. Conf. Computer Vision (ICCV), 2019, pp. 1–11. [10]    D. Güera and E. J. Delp, "Deepfake video detection using recurrent neural networks," in Proc. 15th IEEE Int. Conf. Advanced Video and Signal Based Surveillance (AVSS), 2018, pp. 1–6, doi: 10.1109/AVSS.2018.8639163.