HYBRID WEIGHTED LSA AND EMBEDDING-BASED SEMANTIC TEXT SEGMENTATION FOR ENHANCED DOCUMENT RETRIEVAL

Empirical results on the WikiSection dataset demonstrate that the proposed framework significantly outperforms conventional chunking approaches, achieving Recall@1 = 0.743 and MRR = 0.786 while maintaining semantically coherent retrieval-ready chunks. Notably, the Hybrid + FT (Optimized) framework successfully satisfies all target performance criteria presented in Table I. These findings highlight the effectiveness of combining transformer-based embeddings with weighted latent semantic analysis and structural document signals for retrieval-oriented text segmentation. Future work will investigate larger-scale evaluation, multilingual corpora, and LLM-assisted boundary detection to further improve semantic chunking for Retrieval-Augmented Generation (RAG) systems [8], [13].

[1] N. Reimers and I. Gurevych, "Sentence-BERT: Sentence embeddings using Siamese BERT-networks," in Proc. 2019 Conf. Empir. Methods Nat. Lang. Process. (EMNLP), 2019. [2] M. A. Hearst, "TextTiling: Segmenting text into multi-paragraph subtopic discussions," Comput. Linguist., vol. 23, no. 1, pp. 33–64, 1997. [3] F. Y. Choi, "A latent semantic indexing approach to text segmentation," in Proc. 1st Meet. North Amer. Chapter Assoc. Comput. Linguist. (NAACL), 2000. [4] S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer, and R. Harshman, "Indexing by latent semantic analysis," J. Amer. Soc. Inf. Sci., vol. 41, no. 6, pp. 391–407, 1990. [5] D. M. Blei, A. Y. Ng, and M. I. Jordan, "Latent Dirichlet allocation," J. Mach. Learn. Res., vol. 3, pp. 993–1022, 2003. [6] A. Vaswani et al., "Attention is all you need," in Adv. Neural Inf. Process. Syst. (NeurIPS), 2017. [7] J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding," in Proc. 2019 Conf. North Amer. Chapter Assoc. Comput. Linguist. (NAACL), 2019. [8] P. S. H. Lewis et al., "Retrieval-augmented generation for knowledge-intensive NLP tasks," in Adv. Neural Inf. Process. Syst. (NeurIPS), 2020. [9] J. Gao, J. Kulkarni, and B. Zhang, "Text segmentation for information retrieval: A survey," ACM Comput. Surv., vol. 55, no. 8, pp. 1–35, 2022. [10] V. Karpukhin et al., "Dense passage retrieval for open-domain question answering," in Proc. 2020 Conf. Empir. Methods Nat. Lang. Process. (EMNLP), 2020, pp. 6769–6781.