AI-Driven Self-Healing ETL Pipeline Using AWS Cloud Services for Automated Data Processing

Modern enterprises ingest vast amounts of data from diverse sources, requiring robust ETL (Extract-Transform-Load) pipelines. Traditional pipelines often break due to data inconsistencies or infrastructure faults. This paper proposes a self-healing ETL pipeline architecture built on AWS serverless services (S3, Glue, Lambda, Step Functions, etc.) augmented with AI/ML for automatic error detection and recovery. In our design, raw data lands in Amazon S3, triggers AWS Glue jobs for transformation, and any failures are detected via anomaly detection (e.g. using AWS Glue Data Quality or a SageMaker-trained model). A Step Functions workflow coordinates the ETL tasks and can retry or invoke corrective routines when errors occur. We include the architecture diagram, workflow, methodology, and cost analysis. Experimental results from related work show that self-healing pipelines significantly improve uptime and reduce manual intervention.