Data Engineering–Driven Hierarchical Planning for Consequence-Aware Agentic Systems.
Brahma Reddy Katam
Technical Lead, Data Engineering and Advanced Computing
Abstract: Recent advances in Large Language Models (LLMs) have enabled the development of agentic systems capable of generating code, automating workflows, and interacting with data platforms through natural language. Despite these improvements, most current agents remain fundamentally reactive. They execute tasks step-by-step at a low operational level without reasoning about higher-level goals or long-horizon strategies. As highlighted by recent research in artificial intelligence, including perspectives from Yann LeCun, the primary limitation of modern AI systems is not low-level action generation but the lack of abstract and hierarchical planning. Humans naturally plan by decomposing complex objectives into layered goals and subgoals, enabling efficient decision-making and robust problem solving. In contrast, today’s AI agents often jump directly to individual actions, resulting in unstable behavior, inefficient resource usage, and suboptimal outcomes in production environments.
This limitation is particularly evident in large-scale data engineering platforms, where operational decisions inherently follow hierarchical structures. Engineers typically reason from high-level objectives, such as reducing cost or improving reliability, down to intermediate strategies and finally to concrete system commands. However, LLM-based agents lack this abstraction capability and therefore struggle to manage complex data systems autonomously.
To address this gap, this paper proposes a Data Engineering–Driven Hierarchical Planning framework for consequence-aware agentic systems. We introduce a multi-layer planning architecture that separates goals, strategies, and executable actions, allowing agents to reason at appropriate levels of abstraction before execution. The framework integrates structured telemetry, learned world models for consequence prediction, and a hierarchical planner that decomposes objectives into safe and optimized operational steps. By combining abstraction with predictive validation, agents can make more reliable, cost-efficient, and stable decisions.
Experimental evaluation on representative lakehouse workloads demonstrates that hierarchical planning reduces unnecessary actions, improves runtime performance, lowers infrastructure cost, and enhances operational stability compared to flat or reactive automation approaches. The results suggest that hierarchical reasoning is a critical foundation for building truly autonomous and trustworthy data platforms.
Keywords
Hierarchical Planning, Agentic AI, Data Engineering, Autonomous Systems, Consequence-Aware Decision Making, World Models, Digital Twin, Intelligent Data Platforms, Lakehouse Architecture, Predictive Optimization
