Retrieval-Augmented Generation for Semantic Querying of PDF Documents Using Open-Source Language Models

Retrieval-Augmented Generation for Semantic Querying of PDF Documents Using Open-Source Language Models

Ansh Bharghava
Dept. of Computer Science and Engineering
Sir M. Visvesvaraya Institute of Technology
Bengaluru, Karnataka

Arunita Sahu
Dept. of Computer Science and Engineering
Sir M. Visvesvaraya Institute of Technology
Bengaluru, Karnataka

Brinda B V
Dept. of Computer Science and Engineering
Sir M. Visvesvaraya Institute of Technology
Bengaluru, Karnataka

Nandini Kumari
Dept. of Computer Science and Engineering
Sir M. Visvesvaraya Institute of Technology
Bengaluru, Karnataka

Mrs. Itishree Barik
Dept. of Computer Science and Engineering
Sir M. Visvesvaraya Institute of Technology
Bengaluru, Karnataka

Abstract — Due to their dependence on static, pre-trained knowledge, large language models (LLMs) frequently experience hallucination, producing outputs that appear confident but are factually incorrect. This restriction is lessened by retrieval-augmented generation (RAG), which grounds responses in outside context. A document-based RAG system that uses the open-source reasoning model DeepSeek and is locally deployed using Ollama is presented in this paper. A Chroma vector database is used to store the PDF documents that users upload after they have been parsed, segmented, and embedded using LangChain. When a query is received, the system uses DeepSeek to produce grounded responses and retrieve semantically relevant content. The system may optionally call the web-based search API Tavily if no appropriate context is found. Streamlit, which provides an easy-to-use interface for document upload, parameter tuning, and chat interaction, is used to deploy the application. The system enhances reliability and decreases hallucinations in knowledge-intensive tasks by integrating contextual retrieval, local inference, and optional fallback.

Keywords — LLMs, Hallucination, RAG, DeepSeek, Ollama, Local Deployment, Vector Database.