Hybrid Multimodal Generative AI System for Integrated Prescription and Laboratory Report Analysis with Personalized Drug Guidance and Clinical Decision Support
Dr. Farheen Mohammed Assistant Professor, Dept. of CSE(AIML)
Bapatla Engineering College Bapatla 522101, Andhra Pradesh, India
farheen0122@gmail.com
Bodapati Lakshmi Sowjanya Final Year Student, Department of CSE(AIML)
Bapatla Engineering College Bapatla 522101, Andhra Pradesh, India
bodapapisowjanya098@gmail.com
Bonguluri Venkata Lakshmi Final Year Student, Dept. of CSE(AIML)
Bapatla Engineering College Bapatla 522101, Andhra Pradesh, India
venkatalakshmib13@gmail.com
Chagalamarri Ummar Farook Final Year Student, Dept. of CSE(AIML)
Bapatla Engineering College Bapatla 522101, Andhra Pradesh, India
ch.ummarfarook1234@gmail.com
Shaik Althaf Ahmed Final Year Student, Dept. of
CSE(AIML)
Bapatla Engineering College Bapatla 522101, Andhra Pradesh, India
skalthafahmed0987@gmail.com
Corresponding Author Email: venkatalakshmib13@gmail.com
Abstract— The increasing complexity of medical documents, including handwritten prescriptions and structured laboratory diagnostic reports, continues to hinder patient comprehensibility, leading to preventable medication errors in healthcare ecosystems around the world. In this paper, we propose a novel AI-powered multimodal and multilingual web-based system aimed at addressing the communicate gap between medical documents and patient comprehensibility. The proposed system integrates a web- based optional character recognition (OCR) pipeline with the vision-enabled large language model, namely meta-llama/llama-4- scout-17b-16e instruct, powered by Groq inference infrastructure, to enable dual-input analysis of medical documents submitted in image or PDF formats. The proposed system architecture consists of three functional modules: prescription analysis, laboratory report analysis, and contextual drug information. The proposed system also supports three languages: English, Telugu, Hindi, Tamil, and Kannada, which can effectively cater to the linguistic diversity of the Indian subcontinent. The proposed system was comprehensively tested using a four-metric-based performance evaluation framework, which was applied to all three task types. The performance of the proposed system was found to achieve 96% accuracy, 93% ROUGE-L, 0.081 hallucination rate, and 89.02% METEOR. The proposed system is significant contribution to the field of patient-centric AI healthcare.
Keywords— Large Language Model, Multimodal Medical Document Analysis, Optical Character Recognition, Prescription Interpretation, Clinical AI systems.
