Document Vectorization for Large-Scale Information Retrieval Systems

Document Vectorization for Large-Scale Information Retrieval Systems

T Maheswaran, Associate Professor, Department of Electronics and Communication Engineering,

Sri Shakthi Institute of Engineering and Technology, L&T Bypass, Coimbatore, maheswaran@siet.ac.in

Prames M, Priyadharshini K, Sarika O R, Yogeswari S Department of Electronics and Communication Engineering,

Sri Shakthi Institute of Engineering and Technology, L&T Bypass, Coimbatore

Abstract

The invention presents an entirely new paradigm of document-search framework to make the retrieval and access of specific information from several thousands of documents, especially PDF files, as simple and easy as possible.

The document content and metadata are retrieved for search using a vector-based approach with state-of-the-art vectorization techniques and vector databases such as MilvusDB.

Document content is prepared and vectorized using preprocessing methods such as stop word removal, tokenization, normalization, etc. The vector data are stored in the database and indexed by such techniques that embrace similarity cosines to make fast and accurate retrievals of relevant results.

It is an offline system that works in one's locality, enhancing privacy and dependability in the absence of a network.

The solution is scalable, and flexible and supports every possible document format; thus, it is one stop solution for students, research scholars, and job-workers. Spending less time searching through documents increases productivity and has taken working with an incredibly user-friendly approach enriched with metadata.

Keywords: Document Search, Vector Store, Information Retrieval, PDF Search, MilvusDB, Vectorization, Tokenization, Metadata, Similarity Search, Offline Search, Scalable Solution, AI-powered Search, Research Tool, Productivity Enhancement.