AI-Based Prompt Optimizer, Evaluator & Multi-LLM Comparison Platform

This paper presented the design of an AI-based prompt optimizer, evaluator and multi-LLM comparison platform. It combines a deterministic, explainable prompt-quality analyzer, a multi-round  LLM-driven  optimizer  and  concurrent, failure-isolated benchmarking across OpenAI, Google Gemini, Anthropic Claude and Groq. By scoring every response with a configurable-weight composite of reference-based lexical similarity, consistency, efficiency and LLM-as-a-judge results, and by saving every experiment for later comparison, the approach aims to replace ad hoc, unrecorded prompt trial and error with a reproducible and auditable workflow. It also provides a scalable base for future additions such as automated PDF reporting, ablation-study analysis and self-consistency majority voting in prompt-engineering tools.

[1]      Y. Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba, “Large language models are human-level prompt engineers,” in Proc. Int. Conf. Learning Representations (ICLR), 2023. [2]      C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, and X. Chen, “Large language models as optimizers,” in Proc. ICLR, 2024. [3]      L. Zheng, W.-L. Chiang, Y. Sheng, et al., “Judging LLM-as-a-judge with MT-Bench and Chatbot Arena,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 36, 2023. [4]      K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: a method for automatic evaluation of machine translation,” in Proc. 40th Annu. Meeting of the ACL, 2002, pp. 311–318. [5]      C.-Y. Lin, “ROUGE: a package for automatic evaluation of summaries,” in Text Summarization Branches Out, ACL, 2004, pp. 74–81. [6]      B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, et al., “Mathematical discoveries from program search with large language models,” Nature, vol. 625, pp. 468–475, 2024, doi: 10.1038/s41586-023-06924-6. [7]      F. Liu, X. Tong, M. Yuan, X. Lin, et al., “Evolution of heuristics: towards efficient automatic algorithm design using large language model,” in Proc. 41st Int. Conf. Machine Learning (ICML), 2024. [8]      N. van Stein and T. Bäck, “LLaMEA: a large language model evolutionary algorithm for automatically generating metaheuristics,” IEEE Trans. Evol. Comput., 2024, doi: 10.1109/TEVC.2024.3402686. [9]      X. Wu, Y. Zhong, J. Wu, F. Liu, and Q. Zhang, “AS-LLM: when algorithm selection meets large language model,” arXiv:2307.08784, 2023. [10]    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” in Advances in Neural Information Processing Systems, vol. 35, 2022, pp. 24824–24837.