Towards Standardized Evaluation of Large Language Model-Based Agents

The rise of Large Language Model (LLM)-based agents marks a major shift in artificial intelligence, enabling autonomous systems to plan, reason, use external tools, and retain memory while interacting with dynamic environments. This paper presents a comprehensive survey of evaluation methodologies for such agents. We systematically examine benchmarks and frameworks across four key dimensions: (1) fundamental agent capabilities, including planning, tool use, memory, and self-reflection; (2) application-specific benchmarks for domains such as web navigation, software engineering, scientific discovery, and conversational agents; (3) benchmarks for generalist agents; and (4) frameworks for evaluating agent performance. Our study highlights emerging trends, such as the move toward more realistic and continuously updated evaluation settings, while also identifying critical gaps in assessing safety, robustness, cost-efficiency, and scalability. By mapping the rapidly evolving landscape of LLM-agent evaluation, this survey outlines current limitations and proposes promising directions for future research.