Optimizing Execution Time in Big Data Warehouses

Optimizing Execution Time in Big Data Warehouses

Lakshmi Deepthi Nukala, Gadipelly Varshitha, Kalagani Vivek, Talari Srinidhi

Department of Data Science Institute of Aeronautical Engineering

Hyderabad, India

Abstract—Traditional data warehouses (DWs) have long been fundamental to business intelligence and decision support sys- tems. However, with the exponential growth of data produced by modern applications, traditional data warehousing systems face challenges that require adaptation to new paradigms. In the context of big data, it’s essential to modify current warehouse sys- tems to address these emerging limitations. One major drawback of traditional Extract, Transform, Load (ETL) processes is their inability to efficiently handle massive volumes of data, especially when dealing with unstructured or semi-structured formats. The processing speed is also significantly hindered, leading to high execution times.To overcome these challenges, a new approach is proposed, which introduces a model based on four layers: Extract, Clean, Load, and Transform (ECLT). This model is specifically designed to optimize data processing and improve efficiency when managing large-scale and complex datasets. This aims to reduce execution time in ECLT and compare time between ETL and ECLT and adding analysis phase to ECLT named ECLTA (Extract Transform Load Transform Analysis) for queries and making analysis of the data.

Index Terms—Big Data,ETL (Extract Transform Load) ,ECLT (Extract clean Load Transform),ECLTA (Extract Clean Load Transform Analysis).