AI-Based Multi-Disease Detection Application Using Multimodal Deep Learning

This paper presented an integrated approach to respiratory disease screening using chest X-ray images, breathing audio and clinical notes. ResNet50, MFCC with a dense neural network, and TF-IDF with a dense neural network analyse the three modalities, and their predictions are combined by late weighted fusion with weights of 45% for image, 30% for audio and 25% for text. The system supports the detection of Asthma,  COPD,  Pulmonary  Fibrosis,  Pneumonia  and Tuberculosis while providing confidence scores, risk classification, Grad-CAM visual explanations and downloadable reports. With authentication, data storage, validation and an interactive interface, the application offers a practical framework for explainable respiratory disease screening. However, it is intended to support medical professionals and should not replace clinical diagnosis. In future, the system can be trained with larger and more diverse clinical datasets to improve reliability across patient populations. Additional respiratory information such as spirometry and oxygen-saturation measurements may be incorporated to strengthen the multimodal analysis. A separate React Native/Expo mobile interface could be developed using the existing Flask backend and trained models. Improvements to Grad-CAM and other explainable AI methods may give clearer interpretations of chest X-ray predictions, and integration with hospital information systems and electronic health records could support secure clinical use, subject to validation, privacy protection and regulatory approval. Continuous performance monitoring, periodic model updating, personalised risk assessment and multilingual clinical-text processing are further enhancements.

1.    T. Dimakatso, V. Kuthadi, R. Selvaraj, and O. Dinakenyane, “Pragmatic review on progressions in multimodal disease prediction with combination of machine learning, deep learning and electronic health records,” in Proc. 4th IEEE Int. Conf. ICT in Business Industry & Government (ICTBIG), 2024, doi: 10.1109/ICTBIG64922.2024.10911239. 2.  S. D. M. Sunil and P. V. Joshi, “Early prediction of chronic diseases through multimodal data fusion in machine learning: A survey,” in Proc. 7th Int. Conf. Intelligent Sustainable Systems (ICISS), 2025, pp. 1108–1115, doi: 10.1109/ICISS63372.2025.11076438. 3.   M. Constantinou, T. Exarchos, A. G. Vrahatis, and P. Vlamos, “COVID-19 classification on chest X-ray images using deep learning methods,” Int. J. Environ. Res. Public Health, vol. 20, no. 3, Art. no. 2035, 2023, doi: 10.3390/ijerph20032035. 4.  M. Irtaza, A. Ali, M. Gulzar, and A. Wali, “Multi-label classification of lung diseases using deep learning,” IEEE Access, vol. 12, pp. 124062–124080, 2024, doi: 10.1109/ACCESS.2024.3454537. 5.   E. Khan, M. Z. U. Rehman, F. Ahmed, F. A. Alfouzan, N. M. Alzahrani, and J. Ahmad, “Chest X-ray classification for the detection of COVID-19 using deep learning techniques,” Sensors, vol. 22, no. 3, Art. no. 1211, 2022, doi: 10.3390/s22031211. 6.  R. Sreejith, R. K. Ramasamy, W.-N. Mohd-Isa, J. Abdullah, S. M. Jamal, F. A. Hamzah, and S. Thanjappan, “RespireNet: Enhancing lung sound classification using CNN-TCN hybrid approach,” in Proc. 5th Int. Conf. Artificial Intelligence and Application Technologies (AIAT), Kyoto, Japan, Dec. 2025, pp. 1–10, doi: 10.1145/3787120.3787132. 7.  W. Zhang, X. Li, Q. Liu, X. Zheng, Y. Ge, X. Pan, and Y. Zhou, “High-accuracy lung sound classification for healthy versus unhealthy diagnosis using artificial neural network,” Frontiers in Bioengineering and Biotechnology, vol. 13, Art. no. 1583416, 2025, doi: 10.3389/fbioe.2025.1583416. 8.  I. Topaloglu, G. Ozduygu, C. Atasoy, G. Batıhan, D. Serce, G. Inanc, M. O. Güçsav, A. M. Yıldız, T. Tuncer, S. Dogan, and P. D. Barua, “Machine learning-driven lung sound analysis: Novel methodology for asthma diagnosis,” Advances in Respiratory Medicine, vol. 93, Art. no. 32, 2025, doi: 10.3390/arm93050032. 9.  B. Estay Zamorano, A. Dehghan Firoozabadi, P. Adasme, W. Montiel Piña, M. Chávez Muñoz, D. Zabala-Blanco, P. Palacios Játiva, and C. A. Azurdia-Meza, “Multimodal deep learning approaches for lung disease detection: A review,” Medicina, vol. 62, Art. no. 1223, 2026. L. Brunese, F. Mercaldo, A. Reginelli, and A. Santone, “Explainable deep learning for pulmonary disease and coronavirus COVID-19 detection from X-rays,” Comput. Methods Programs Biomed., vol. 196, Art. no. 105608, 2020, doi: 10.1016/j.cmpb.2020.105608.