Empirical Evaluation of Automated and Expert-Guided Prompt Engineering for Structured Data Extraction in Large Language Models

This paper presented a controlled comparison of three prompting strategies — a minimal baseline, a human expert prompt, and an automated prompt optimizer — for structured JSON extraction from noisy receipt and invoice text, evaluated on a held-out test set of 10 samples using fully deterministic metrics. The comparative results reveal a clear performance hierarchy. The minimal baseline, with no task-specific guidance, achieved only 12.9% field-level accuracy, confirming that unstructured prompting is insufficient for reliable extraction. The automated optimizer, starting from this same minimal baseline and iteratively refining over five iterations, reached 84.3% field-level accuracy and 30.0% exact match — a 71.4 percentage point improvement over the baseline achieved entirely without human intervention. The human expert prompt achieved the highest performance at 90.4% field-level accuracy and 45.0% exact match, outperforming the automated optimizer by 6.1 percentage points in field accuracy and 15.0 percentage points in exact match. A field-level comparison further highlights the complementary strengths of each approach. The human expert prompt excelled at fields requiring domain-specific reasoning, such as vendor_name (100% vs. 60%) and line_items (97.5% vs. 90.0%), where anticipation of OCR errors and complex structural parsing proved critical. Conversely, the automated optimizer achieved superior performance on fields with standardized formats — currency (90% vs. 80%), date (90% vs. 85%), and total (100% vs. 95%) — demonstrating that automated pattern discovery is effective for well-structured data elements. Both methods achieved identical performance on tax extraction (100%). Despite the observed performance gap, statistical tests (Wilcoxon p = 0.375, McNemar p = 1.000) did not reach significance at α = 0.05, and the bootstrap 95% confidence interval ([-0.17, 0.01]) marginally includes zero, indicating that the difference, while consistent, is not statistically conclusive at this sample size. From a cost perspective, the automated optimizer incurred approximately 7.2× the computational cost of the human expert ($0.0224 vs. $0.0031), though this does not account for the human labor investment required to craft the expert prompt. The parallels between the prompt optimization methodology presented in this paper and the constructive neural network frameworks of Kaur and Gupta [9, 10, 11] suggest a unifying principle: iterative, performance-driven construction from minimal starting points is an effective optimization strategy across different levels of the AI stack, from neural architectures to prompt engineering. Combined with the transparency and auditability afforded by blockchain technologies [12, 13, 14], these approaches point toward a future of more reliable, trustworthy, and automatically optimized intelligent computing systems. Future work should investigate: (a) scaling to larger and more diverse datasets to improve statistical power; (b) extending to multiple extraction domains including medical records, legal documents, and financial statements; (c) hybrid human-optimizer pipelines where the optimizer refines a human-authored starting prompt; (d) multi-model evaluation to test prompt transferability across different LLM architectures; (e) longer optimization horizons with early stopping criteria to balance improvement against overfitting; and (f) integration with blockchain-based audit trails for production deployment of optimized extraction pipelines.

[1]              Brown, T. B., Mann, B., Ryder, N., et al. "Language Models are Few-Shot Learners." Advances in Neural Information Processing Systems 33 (2020): 1877–1901. [2]              Wei, J., Wang, X., Schuurmans, D., et al. "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." Advances in Neural Information Processing Systems 35 (2022): 24824–24837. [3]              White, J., Fu, Q., Hays, S., et al. "A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT." arXiv preprint arXiv:2302.11382 (2023). [4]              Zhou, Y., Muresanu, A. I., Han, Z., et al. "Large Language Models Are Human-Level Prompt Engineers." International Conference on Learning Representations (2023). [5]              Yang, C., Wang, X., Lu, Y., et al. "Large Language Models as Optimizers." International Conference on Learning Representations (2024). [6]              Khattab, O., Singhvi, A., Maheshwari, P., et al. "DSPy: Compiling Declarative Language Model Calls into SelfImproving Pipelines." International Conference on Learning Representations (2024). [7]              Opsahl-Ong, K., Ryan, M. J., Hardy, J., and Khattab, O. "Optimizing Instructions and Demonstrations for MultiStage Language Model Programs." arXiv preprint arXiv:2406.11695 (2024). [8]              Kaur, J., and Gupta, N. "Artificial Neural Network: A Review." International Journal of Technical Research & Science (n.d.): 1–4. [9]              Kaur, J., and Gupta, N. "Constructive Neural Network: A Framework." International Journal of Engineering and Advanced Technology 9(2) (2019). https://doi.org/10.35940/ijeat.b3304.129219 [10]           Kaur, J., and Gupta, N. "Bipolar Sigmoid Algorithm for Designing Constructive Neural Network." International Journal on Emerging Technologies 11(2) (2020): 991–996.