Designing Ultra-Low Latency Data Pipelines to Power ML Models for Real-Time Autonomous Decision Making.
Brahma Reddy Katam
Technical Lead Data Engineer in Data Engineering & Advanced Computing
Abstract: As real-time decision-making becomes increasingly critical in autonomous systems such as self-driving vehicles, drones, robotic automation, and industrial IoT, the need for ultra-low latency data infrastructure has never been more urgent. This research paper presents a comprehensive and modular architecture for designing ultra-low latency data pipelines capable of delivering high-frequency streaming data to machine learning (ML) models for real-time autonomous decision-making.
The proposed architecture integrates state-of-the-art tools such as Apache Kafka for ingestion, Apache Flink for stream processing, Delta Lake and Redis for high-speed data storage, and NVIDIA Triton Inference Server for GPU-accelerated model serving. A key objective of this work is to ensure end-to-end latency remains under 100 milliseconds — enabling timely and accurate predictions in mission-critical environments.
The paper uses a real-world simulation of autonomous drone telemetry to demonstrate the effectiveness of the pipeline. The pipeline processes telemetry data streams at 100ms intervals, performs real-time transformations such as risk score calculation, and triggers decisions such as flight path adjustments based on model predictions. The system also supports fault tolerance, schema evolution, and model versioning, making it highly adaptable and production-ready.
Through benchmark testing and architectural evaluation, the pipeline consistently achieved end-to-end latency in the range of 80–95 milliseconds across all components, including ingestion, processing, feature lookup, and ML inference. Furthermore, this research highlights ethical considerations, such as data privacy, safe fallback mechanisms, and transparent decision logic.
In conclusion, this paper offers a practical and scalable framework for building ultra-low latency data pipelines tailored for intelligent autonomous systems. It serves as a foundation for further exploration into agentic AI, federated learning, and edge-native ML applications in real-time environments.
Keywords: Real-time machine learning, Ultra-low latency, Data pipelines, Autonomous systems, Kafka, Flink, ML inference, Delta Lake, Edge AI
