Deep Multimodal Fusion of Medical Imaging and Clinical Data for Disease Prediction: A Review and Framework

This review confirms that multimodal deep learning represents the next frontier in disease prediction, as it effectively emulates the integrative reasoning used by human clinicians. By fusing high-dimensional medical imaging with structured clinical variables, laboratory results, and genomic profiles, these frameworks capture complementary disease markers that are often missed by single-modality systems [1]. Evidence across oncology, cardiology, and neurology benchmarks has shown that this integration yields significant performance gains, with accuracy improvements ranging from 1.2% to 27.7% over unimodal baselines [3]. Technically, the choice of fusion strategy remains a critical design decision. While Intermediate (Joint) Fusion is the most powerful for learning complex, non-linear interactions between pixels and clinical context, Late Fusion remains the most practical and robust choice for real-world clinical settings where records are often incomplete [1],[4]. Furthermore, the transition from standard CNNs to Vision Transformers (ViTs) and attention-based modules has significantly enhanced the model’s ability to handle global contextual relationships in large-scale radiological and pathological data [5], [6]. However, the path to clinical deployment is hindered by challenges related to data scarcity, demographic bias, and the "black-box" nature of deep learning [7]. This review highlights that a model’s success should not be measured solely by high AUROC or precision scores. For clinical adoption, frameworks must incorporate Explainable AI (XAI) tools like Grad-CAM and SHAP to provide transparent, verifiable evidence for their predictions [8]. Moving forward, future research must prioritize the development of privacy-preserving Federated Learning systems that allow multi-institutional collaboration without compromising sensitive patient data [1], [9]. Ultimately, the goal is to develop multimodal systems that are not just technically superior, but are also fair, explainable, and provide measurable value in improving patient outcomes and precision medicine [2].

[1] S.-C. Huang, A. Pareek, S. Seyyedi, I. Banerjee, and M. P. Lungren, “Fusion of medical imaging and electronic health records using deep learning: A systematic review and implementation guidelines,” npj Digital Medicine, vol. 3, Art. no. 136, 2020, doi: 10.1038/s41746-020-00341-z. [2] C. Cui, H. Yang, Y. Wang, S. Zhao, Z. Asad, L. A. Coburn, K. T. Wilson, B. A. Landman, and Y. Huo, “Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: A review,” Progress in Biomedical Engineering, vol. 5, no. 2, Art. no. 022001, 2023, doi: 10.1088/2516-1091/acc2fe. [3] A. Kline, H. Wang, Y. Li, S. Dennis, M. Hutch, Z. Xu, F. Wang, F. Cheng, and Y. Luo, “Multimodal machine learning in precision health: A scoping review,” npj Digital Medicine, vol. 5, Art. no. 171, 2022, doi: 10.1038/s41746-022-00712-8. [4] B. Lobato-Delgado, B. Priego-Torres, and D. Sanchez-Morillo, “Combining molecular, imaging, and clinical data analysis for predicting cancer prognosis,” Cancers, vol. 14, no. 13, Art. no. 3215, 2022, doi: 10.3390/cancers14133215. [5] T. Baltrušaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 2, pp. 423-443, 2019, doi: 10.1109/TPAMI.2018.2798607. [6] A. W. Salehi, S. Khan, G. Gupta, B. I. Alabduallah, A. Almjally, H. Alsolai, T. Siddiqui, and A. Mellit, “A study of CNN and transfer learning in medical imaging: Advantages, challenges, future scope,” Sustainability, vol. 15, no. 7, Art. no. 5930, 2023, doi: 10.3390/su15075930. [7] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and A.  Batra, “Grad-CAM: Visual explanations from deep networks via gradient-based localization,” in Proc. IEEE International Conference on Computer Vision (ICCV), 2017, pp. 618-626, doi: 10.1109/ICCV.2017.74. [8] S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” in Advances in Neural Information Processing Systems, vol. 30, 2017. [9] Y. Wang, C. Yin, and P. Zhang, “Multimodal risk prediction with physiological signals, medical images and clinical notes,” Heliyon, vol. 10, no. 5, Art. no. e26772, 2024, doi: 10.1016/j.heliyon.2024.e26772. [10] R. Wang et al., “Artificial intelligence for prediction of COVID-19 progression using CT imaging and clinical data,” European Radiology, vol. 32, no. 1, pp. 205-212, 2022, doi: 10.1007/s00330- 021-08049-8.