Loading article...
Loading article...
Stream/Area:
-
Subject:
-
Paper ID:
IJSREM67510
Volume:
10 |
9 |
Sep 2026
Pages:
1-7
Publication Frequency:
Monthly (12 issues per year)
License:International Journal of Scientific Research in Engineering and Management follows the Creative Commons Attribution 4.0 (CC BY 4.0) License, allowing free use and distribution with proper attribution.
Information Technology
COCHIN UNIVERSITY OF SCIENCE AND TECHNOLOGY
Production AI agents increasingly route work, select tools, construct arguments, maintain state, and initiate consequential actions. Evaluating only the final response, or summarizing many cases into an aggregate score, can therefore conceal failures that make a release unacceptable. This paper formalizes that condition as critical-failure dilution: an aggregate satisfies its acceptance threshold while at least one applicable release-critical behavior fails. To address it, the paper proposes an Agent Behavior Contract, a three-class consequence model, a non-compensatory Critical Constraint Preservation Rule, and a PASS-REVIEW-BLOCK release-decision function. A controlled synthetic retail-support experiment compared baseline and candidate agent configurations using deterministic constraint checks, descriptive model-based response evaluation, operational signals, and execution traces. The candidate improved its aggregate quality result from 0.791 to 0.993 and crossed the aggregate acceptance threshold, while five critical route failures and one critical tool-and-argument failure remained. The consequence-aware policy therefore returned BLOCK despite the aggregate PASS. Mandatory-escalation failures decreased from 65 to zero, showing that improvement in one critical class did not compensate for failures in another. Model-based response scores remained uncalibrated and are reported descriptively with human review required. The observed results satisfy the defined critical-failure dilution condition within the evaluated synthetic scenario and provide an explainable bridge between agent evaluation and release authorization.
KEYWORDS
Production-agent release assurance cannot be reduced to a single average. In the controlled synthetic experiment, the candidate reached an aggregate quality result of 0.993 and passed the aggregate threshold, yet five critical route failures and one critical tool-and-argument failure remained. The consequence-aware rule therefore returned BLOCK. This directly demonstrates critical-failure dilution within the evaluated scenario: aggregate acceptance coexisted with release-critical behavioral failure. The framework contributes an Agent Behavior Contract, a three-class consequence model, a non-compensatory Critical Constraint Preservation Rule, and a PASS-REVIEW-BLOCK decision function. The observed comparison also demonstrates why improvements across aggregate quality, escalation, tool behavior, response quality, and operational signals must not be allowed to compensate for any remaining critical constraint failure. Model-based response results remain descriptive because the evaluator was not calibrated. The findings provide empirical support for the proposed engineering method in the stated synthetic setting, not proof of general production safety or cross-domain validity. Further validation should calibrate model-based evaluators, add uncertainty-only cases that can exercise REVIEW independently, repeat non-deterministic runs, preserve diagnosed failures as regression cases, and test the decision rule across additional agent domains and architectures.
[1] Microsoft, “Evaluate your AI agents,” Microsoft Learn, 2026. [Online]. Available: https://learn.microsoft.com/azure/foundry/observability/how-to/evaluate-agent [2] Microsoft, “Agent evaluators for generative AI,” Microsoft Learn, 2026. [Online]. Available: https://learn.microsoft.com/azure/foundry/concepts/evaluation-evaluators/agent-evaluators [3] Microsoft, “Agent tracing overview,” Microsoft Learn, 2026. [Online].Available: https://learn.microsoft.com/azure/foundry/observability/concepts/t race-agent-concept [4] Microsoft, “Observability in generative AI,” Microsoft Learn, 2026.[Online].Available: https://learn.microsoft.com/azure/foundry/concepts/observability [5] Microsoft, “Custom evaluators,” Microsoft Learn, 2026. [Online]Available: https://learn.microsoft.com/azure/foundry/concepts/evaluation-evaluators/custom-evaluators [6] Microsoft Research, “Agent-Pex: Automated evaluation and testing of AI agents,” 2026. [Online]. Available: https://www.microsoft.com/research/project/agent-pex- automated-evaluation-and-testing-of-ai-agents/ [7] Microsoft, “Observability for generative AI and agentic AI systems,” Microsoft Learn, 2026. [Online]. Available: https://learn.microsoft.com/security/zero-trust/sfi/observability-ai-systems
Publication Date
9/26/2026
Peer Review Type
Double-Blind Peer Review
Review Rounds: 2
Documents
Article Views
0
Total views since publication
PDF Downloads
18
Total downloads since publication
Geographic Reach
5
Digital Preservation: This article is permanently archived on Zenodo and complies with all data preservation standards across major indexing platforms.
Plagiarism Screening
This manuscript has been checked for plagiarism using TurnitinPlagiarismChecker software.
Conflict of Interest
The authors declare no conflict of interest related to this research work.
Funding Information
No funding information available for this research.
Related research in the same subject area
Loading similar articles...