IDExtractNet: An Intelligent Deep Learning Pipeline for Accurate Information Extraction from Identity Documents

IDExtractNet establishes new state-of-the-art benchmarks for identity document OCR: 95.8% end-to-end field F1 and 4.2% CER on MIDV-500 through principled three-stage in- tegration of YOLOv8 localisation, U-Net binarisation, and CRNN-BiLSTM recognition. Key findings: binarisation is the highest-leverage single intervention (+12.4 p.p. CER); staged modular design yields 11% emergent F1 gain beyond naïve component summation; and U-Net preprocessing enables zero-shot Indian document generalisation (94.7% F1) without Indic fine-tuning. The matra-aware CTC extension reduces Devanagari CER from 14.2% to 7.1%; the security detection branch recovers +11.8 p.p. F1 under hologram occlusion; and quantisation-aware training delivers 0.19 s Coral TPU inference with <1% F1 penalty. Six future directions are identified: (1) TrOCR integration [11] for holistic handwriting recovery; (2) federated fine- tuning across institutions without raw data disclosure; (3) ad- versarial training against presentation attacks and deepfake documents; (4) multilingual Indic expansion via mBERT- augmented CTC vocabularies covering Hindi, Marathi, Ben- gali, and Tamil; (5) continual learning for zero-downtime adaptation to new document formats; and (6) open-source release of both the model weights and the Pune-ID corpus to the research community.

  [1]   B. Shi, X. Bai, and C. Yao, “An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 11, pp. 2298–2304, Nov. 2017. [2]   G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics YOLOv8,” 2023. [Online]. Available: https://github.com/ultralytics/ultralytics [3]   O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional net- works for biomedical image segmentation,” in Proc. MICCAI, Cham: Springer, 2015, pp. 234–241. [4]   V. V. Arlazarov et al., “MIDV-500: A dataset for identity document analysis and recognition on mobile devices in video stream,” Comput. Opt., vol. 43, no. 5, pp. 818–824, 2019. [5]   V. V. Arlazarov et al., “MIDV-2019: Challenges of the modern mobile document OCR,” arXiv preprint arXiv:1910.04009, 2019. [6]   P. Netrapalli et al., “MRZ code extraction from visa and passport documents using deep learning,” arXiv preprint arXiv:2009.05489, 2020. [7]   R. Smith, “An overview of the Tesseract OCR engine,” in Proc. 9th Int. Conf. Document Anal. Recognit. (ICDAR), Parana, Brazil, 2007, pp. 629–633. [8]   Y. Du et al., “PP-OCR: A practical ultra lightweight OCR system,” arXiv preprint arXiv:2009.09941, 2020. [9]   X. Zhou et al., “EAST: An efficient and accurate scene text detector,” in Proc. IEEE CVPR, Honolulu, HI, USA, 2017, pp. 5551–5560. [10]   M. Liao et al., “Real-time scene text detection with differentiable binarization,” in Proc. AAAI, 2020, vol. 34, pp. 11474–11481. [11]   M. Li et al., “TrOCR: Transformer-based optical character recognition with pre-trained models,” in Proc. AAAI, 2023, vol. 37, pp. 13094– 13102. [12]   A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, “Connection- ist temporal classification: Labelling unsegmented sequence data with recurrent neural networks,” in Proc. ICML, Pittsburgh, PA, USA, 2006, pp. 369–376. [13]   M. Jaderberg, K. Simonyan, A. Vedaldi, and A. Zisserman, “Synthetic data and artificial neural networks for natural scene text recognition,” arXiv preprint arXiv:1406.2227, 2014. [14]   S. Ma et al., “DocUNet: Document image unwarping via a stacked U- Net,” in Proc. IEEE CVPR, Salt Lake City, UT, USA, 2018, pp. 4700– 4709. [15]   R. Sánchez-Rivero et al., “A joint study of deep learning-based methods for identity document binarization,” Comput. Opt., vol. 46, no. 4, pp. 612–621, 2022. [16]   M. Abuadbba et al., “DL-IDS: A deep learning-based identity document system,” Secur. Commun. Netw., Wiley, 2020. [17]   A. Patil et al., “YOLO and OCR based automated identity document verification,” Int. J. Res. Appl. Sci. Eng. Technol. (IJRASET), vol. 11, no. 3, 2023. [18]   V. I. Levenshtein, “Binary codes capable of correcting deletions, inser- tions, and reversals,” Soviet Physics Doklady, vol. 10, no. 8, pp. 707–710, 1966. [19]   B. Belval et al., “TextRecognitionDataGenerator: A flexible synthetic text image dataset generator,” 2018. [Online]. Available: https://github. com/Belval/TextRecognitionDataGenerator F. Cheng et al., “Focusing attention: Towards accurate text recognition in natural images,” in Proc. IEEE ICCV, Venice, Italy, 2017, pp. 5076–