Benchmarking Hallucination Mitigation Techniques for Open-Source License Compatibility Reasoning A Comparative Study of Prompting, RAG, and Fine-Tuning

This paper presented a comparative evaluation of prompting, retrieval-augmented generation, and LoRA fine-tuning for open-source license compatibility reasoning. On the matched 50-case Ground Truth V2 benchmark, Gemini Flash 3.6 with standardized prompting achieved 88.0% accuracy and 0.909 Macro-F1, while the evaluated RAG configuration achieved 82.0% accuracy and 0.862 Macro-F1. The principal error pattern in both Gemini conditions was CONDITIONAL → COMPATIBLE, showing that retrieval did not by itself resolve the distinction between a compatibility mechanism and the conditions required to use it.   The Qwen2.5-1.5B-Instruct feasibility experiment produced a smaller and different result: LoRA increased accuracy from 20.0% to 30.0% on 10 held-out cases, but Macro-F1 remained 0.167 and output behaviour became strongly conditional-biased, including two invalid COMPLIANT labels. These findings argue against evaluating mitigation methods solely by aggregate accuracy. The central contribution of the study is therefore diagnostic. Reliable license reasoning requires a model to reach the correct compatibility decision, interpret the governing clause correctly, and connect that clause to the actual scenario facts. Future work should evaluate these dimensions separately, preserve evidence provenance, and use larger controlled benchmarks. The present results support structured prompting as the strongest tested baseline while identifying conditionality resolution as the clearest target for improved retrieval and fine-tuning.

1. X. Cui, J. Wu, X. Ling, T. Luo, M. Yang, and W. Ou, “Exploring Large Language Models for Analyzing Open Source License Conflicts: How Far Are We?,” Proc. IEEE/ACM 47th International Conference on Software Engineering: Companion Proceedings, 2025, pp. 291–302, doi: 10.1109/ICSE-Companion66252.2025.00083. 2.  A. Kahol, A. C. Tummepalli, and P. R. Anish, “OSS-LCAF: Open-Source Software License Conflict Analysis Framework,” Proc. IEEE/ACM 47th International Conference on Software Engineering: Companion Proceedings, 2025, pp. 303–314, doi: 10.1109/ICSE-Companion66252.2025.00084. 3.  W. Xu, K. Gao, H. He, and M. Zhou, “LiCoEval: Evaluating LLMs on License Compliance in Code Generation,” Proc. IEEE/ACM  47th  International  Conference  on  Software Engineering,           2025,           pp.           1665–1677,           doi: 10.1109/ICSE55347.2025.00052.  4.  I. Ben Khalifa, M. Ben Messaoud, and M. Wiem Mkaouer, “Toward Explainable and Automated Software License Checking via RAG and Knowledge Graphs,” IEEE Software, 2026, doi: 10.1109/MS.2026.3677583. 5.    M. Liang et al., “THaMES: An End-to-End Tool for Hallucination Mitigation and Evaluation in Large Language Models,” Advances in Neural Information Processing Systems, 2024. 6.  Z. P. Lee, A. Lin, and C. Tan, “Finetune-RAG: Fine-Tuning Language Models to Resist Hallucination in Retrieval-Augmented Generation,” arXiv preprint arXiv:2505.10792, 2025.  7.  J. Song, X. Wang, J. Zhu, Y. Wu, X. Cheng, R. Zhong, and C. Niu, “RAG-HAT: A Hallucination-Aware Tuning Pipeline for LLM in Retrieval-Augmented Generation,” Proc. 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, 2024, pp. 1548–1558, doi: 10.18653/v1/2024.emnlp-industry.113. 8.  M. Hong et al., “RAL2M: Retrieval Augmented Learning-To-Match Against Hallucination in Compliance-Guaranteed Service Systems,” arXiv preprint arXiv:2601.02917, 2026.  9.  Software Package Data Exchange (SPDX), “SPDX License List,” SPDX Project, 2026.  10.   Open Source Initiative, “Open Source Licenses,” Open Source Initiative, 2026.