A Multimodal Deep Learning Framework for Tuberculosis Antibiotic Resistance Prediction Using Chest X-Rays and Genomic Features

This paper presented TB-OS V2.0, a multimodal research prototype that combines a ResNet-18 chest X-ray model and an XGBoost ge-nomic model through an antibiotic-specific logistic-regression late-fusion layer to estimate resistance of M. tuberculosis to rifampicin, isoniazid, ethambutol, streptomycin and pyrazinamide. Fifteen frozen models were trained on antibiotic-specific cohorts with zero patient overlap across splits and exposed through a Flask API and a React dashboard that returns five independent probabilities. On the held-out test sets, the genomic model was the strongest single modality for every drug (ROC-AUC 0.9245 to 0.9739), the chest X-ray model had limited discrimination (0.6376 to 0.7589), and late fusion gave only small changes: a decrease for rifampicin (0.9481 against 0.9623) and gains of at most 0.0108 ROC-AUC for the other four drugs. The ethambutol case also showed that a fixed 0.50 threshold can hide good ranking performance. The work therefore demonstrates that multimodal integration is practical, while showing that it does not automatically beat the best single modality. The system remains a research prototype, and external validation, calibration and clinical evaluation are required before any practical use can be considered.  

[1]   Y. Xu, Y. Mao, X. Hua, et al., “Machine learning-based prediction of antimicrobial resistance and identification of AMR-related SNPs in My-cobacterium tuberculosis,” BMC Genom. Data, vol. 26, Art. no. 48, 2025, doi: 10.1186/s12863-025-01338-x. [2]   B. G. Akwaronwu, F. Y. Ayankoya, S. O. Kuyoro, I. U. Akwaronwu, and O. J. Adeniyi, “Machine learning for predicting drug resistance in tuberculosis: a systematic review of model performance and clinical ap-plications,” Big Data Min. Anal., vol. 9, no. 1, pp. 314–337, 2026, doi: 10.26599/BDMA.2025.9020063. [3]   M. I. Gröschel, M. Owens, L. Freschi, et al., “GenTB: a user-friendly genome-based predictor for tuberculosis resistance powered by machine learning,” Genome Med., vol. 13, Art. no. 138, 2021, doi: 10.1186/s13073-021-00953-4. [4]   B. Lei, T. Zhang, J. Li, J. Qu, W. Xiong, K. Yu, and L. Yang, “Rapidly antibiotic susceptibility prediction via deep learning from bacterial fluo-rescence microscopy images,” IEEE Trans. Autom. Sci. Eng., vol. 22, pp. 21469–21481, 2025, doi: 10.1109/TASE.2025.3607484. [5]   G. Y. Liu, D. Yu, M. M. Fan, et al., “Antimicrobial resistance crisis: could artificial intelligence be the solution?,” Mil. Med. Res., vol. 11, Art. no. 7, 2024, doi: 10.1186/s40779-024-00510-1. [6]   Y. Wang, Z. Jiang, P. Liang, et al., “TB-DROP: deep learning-based drug resistance prediction of Mycobacterium tuberculosis utilizing whole genome mutations,” BMC Genomics, vol. 25, Art. no. 167, 2024, doi: 10.1186/s12864-024-10066-y. [7]   A. Yurtseven, S. Buyanova, A. Agrawal, et al., “Machine learning and phylo-genetic analysis allow for predicting antibiotic resistance in M. tuberculosis,” BMC Microbiol., vol. 23, Art. no. 404, 2023, doi: 10.1186/s12866-023-03147-7. [8]   L. F. Parra-Sánchez, S. I. Rivera-Manrique, A. González-Vega, and L. C. Padierna, “A novel antibacterial susceptibility test from microscopic images by deep learning,” IEEE Access, vol. 13, pp. 166670–166685, 2025, doi: 10.1109/ACCESS.2025.3612431. [9]   A. Farrar, C. Feehily, P. Turner, et al., “Infection Inspection: using the power of citizen science for image-based prediction of antibiotic resistance in Escherichia coli treated with ciprofloxacin,” Sci. Rep., vol. 14, Art. no. 19543, 2024, doi: 10.1038/s41598-024-69341-3. [10]   S. Kumar, S. Sharma, and K. T. Megra, “Transformer enabled multi-modal medical diagnosis for tuberculosis classification,” J. Big Data, vol. 12, Art. no. 5, 2025, doi: 10.1186/s40537-024-01054-w.