Multimodal Artificial Intelligence for Breast Cancer Detection Using Medical Imaging and Clinical Data

A breast cancer detection and analysis system that is based on the intelligence framework of mammogram images and clinical text information. This system enables the user to upload an image or enter text and get the diagnostic results as output. In this system, the results are displayed as the risk score, confidence level, severity level, and localisation of tumors using heatmaps and bounding boxes to indicate the suspicious regions in the image. Finally, the system also provides an AI chatbot to answer the questions and provide relevant information about breast cancer. The chatbot and results are stored in a database, and the user can view their history of analysis and queries in a history page.  

[1] Acosta J.N., Falcone G.J., Rajpurkar P., et al. (2022) Integrated Multimodal Artificial  Intelligence Framework for Healthcare  Applications. PubMed.   [2]         Baltrušaitis T., Ahuja C., Morency L.P. (2022) Multimodal Learning for Medical Decision  Support Systems. Pattern Recognition Letters  (Elsevier).   [3]         Chen Z., Wang Y., et al. (2023) Vision–Language  Models for Medical Imaging:Survey. ACM Computing Surveys.   [4]         Huang S., Yang J., et al. (2021) Multimodal Deep  Learning for Precision Oncology. IEEE Access.   [5] Khan S., Hussain M., et al. (2022) A Systematic  Review of Multimodal Fusion and Explainable AI  in Breast Cancer Diagnosis. Computer Modeling in  Engineering & Sciences.   [6]         Li X., et al. (2024) Vision–Language Foundation  Models for Medical Imaging: A Review. Elsevier.   [7]         Singhal K., Azizi S., Tu T., et al. (2024) Towards  Generalist Biomedical AI. NEJM AI.   [8]         Tjoa E., Guan C. (2020) Explainable AI for  Cancer Diagnosis Using Deep Learning, Artificial  Intelligence in Medicine (Springer).   [9]         Zhang Y., Jiang M., et al. (2023) Integrating Language into Medical Visual  Recognition and Reasoning: A Survey. PubMed.