Semantic Search Engine Using AI Embeddings

This paper presented a hybrid semantic search engine addressing the vocabulary mismatch problem that fundamentally limits keyword-based information retrieval. By combining dense vector retrieval (using Sentence Transformer embeddings), token-overlap scoring, and exact string matching under a unified REST API, the system provides comprehensive coverage across the spectrum of retrieval intents. Empirical evaluation on a purpose-built 1,000-document, 5-domain benchmark demonstrates that semantic mode achieves MRR = 0.89 and P@5 = 0.84 on synonym-dependent queries — a complete and statistically significant improvement over BM25 (MRR = 0.00) on queries with zero lexical overlap. Across all query types, semantic mode outperforms all baselines on all reported metrics. The threshold sweep identifies θ = 0.35 as the optimal cosine similarity cutoff for this model–corpus combination. The system is publicly deployed, fully containerised, and released under MIT License, addressing reproducibility concerns that frequently limit the practical impact of academic IR prototypes.

[1]  G. Salton and M. J. McGill, Introduction to Modern Information Retrieval. McGraw-Hill, 1983. [2] S. E. Robertson and S. Walker, "Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval," in Proc. ACM SIGIR, 1994, pp. 232–241. [3] G. W. Furnas, T. K. Landauer, L. M. Gomez, and S. T. Dumais, "The vocabulary problem in human-system communication," Commun. ACM, vol. 30, no. 11, pp. 964–971, 1987. [4]  Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019, June). Bert Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) (pp. 4171-4186) [5]   Reimers, N., & Gurevych, I. (2019, November). Sentence-bert Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP) (pp. 3982-3992). [6] Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26. [7]  Pennington, J., Socher, R., & Manning, C. D. (2014, October). Glove Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532-1543). [8]  Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., & Zhou, M. (2020). Minilm Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in neural information processing systems, 33, 5776-5788. [9]  Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., ... & Yih, W. T. (2020, November). Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP) (pp. 6769-6781). [10]  Johnson, J., Douze, M., & Jégou, H. (2019). Billion-scale similarity search with GPUs. IEEE transactions on big data, 7(3), 535-547.