Multimodal Computer Vision for Rapid Disaster Damage Analysis and Victim Detection using Deep Learning

This paper has presented a hybrid computer vi-sion framework for real-time disaster scene analysis that integrates YOLO-based instance segmentation with clas-sical image processing algorithms within a full-stack web application. The system combines YOLOv8n-seg for vic-tim detection, HSV color-space thresholding for fire and smoke identification, Canny edge density analysis for structural damage estimation, and YOLO11x for struc-tural context detection in a late-stage fusion architecture served by a FastAPI backend with MJPEG streaming sup-port. Experimental evaluation on commodity CPU hardware demonstrated streaming inference rates of 13–24 FPS across operational modes, with comprehensive static-frame analysis completing in under 900 millisec-onds. Qualitative assessment across victim, fire, and dam-age scene categories confirmed practical detection capa-bilities within the well-understood limitations of classical visual analysis. The hybrid architecture's core contribu-tion is its demonstration that meaningful multi-category disaster scene analysis can be achieved without GPU ac-celeration or domain-specific model training, using a combination of pretrained general-purpose neural net-works and computationally inexpensive algorithmic methods. The proposed framework is positioned as a prac-tical first-pass triage tool suitable for early deployment in emergency response operations, providing actionable vis-ual intelligence from the first moments of a disaster re-sponse without the infrastructure requirements of more sophisticated deep learning systems. Future work will fo-cus on replacing algorithmic heuristics with quantized do-main-specific neural models and extending the system to support multi-camera and UAV-based inputs..

[1]  J. Redmon et al., “You Only Look Once: Unified, Real-Time Object Detection,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp. 779–788.   [2]   A. Bochkovskiy, C. Y. Wang, and H. Y. M. Liao, “YOLOv4: Optimal Speed and Accuracy of Object Detec-tion,” arXiv preprint arXiv:2004.10934, Apr. 2020.   [3]       Ultralytics, “YOLOv8 Documentation,” 2023. [Online]. Available: https://docs.ultralytics.com [4]   R. Gupta et al., “Creating xBD: A Dataset for As-sessing Building Damage from Satellite Imagery,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Long Beach, CA, USA, Jun. 2019, pp. 10–17.   [5]  Z. Xu, H. Wang, and Y. Zhang, “Building Damage De-tection in Satellite Imagery Using Convolutional Neural Networks,” arXiv preprint arXiv:1910.06444, Oct. 2019.   [6]  T. Çelik and H. Demirel, “Fire Detection in Video Se-quences Using a Generic Color Model,” Fire Safety Jour-nal, vol. 44, no. 2, pp. 147–158, Feb. 2009.   [7]   W. Phillips III, M. Shah, and N. V. Lobo, “Flame Recognition in Video,” Pattern Recognition Letters, vol. 23, no. 1–3, pp. 319–327, Jan. 2002.   [8]      S. Minaee, Y. Boykov, F. Porikli, A. Plaza, N. Ke-htarnavaz, and D. Terzopoulos, “Image Segmentation Us-ing Deep Learning: A Survey,” IEEE Transactions on Pat-tern Analysis and Machine Intelligence, vol. 44, no. 7, pp. 3523–3542, Jul. 2022.