DATA PROVENANCE AND LINEAGE TRACKING IN BIG DATA PIPELINES

The Data Provenance & Lineage Tracking in Big Data Pipelines project successfully addresses the critical challenge of opacity in modern data engineering. In shifting from manual static document processing to an automated dynamic metadata management system this project bridges the gap between data processing and data governance. Use of automated collectors together with a graph-based storage engine has been found to be an effective approach for dealing with the non-linear relationships in Big Data processes. This approach puts the power in the hands of the organization in order to gain control of their assets. It ensures the reversal of "Data Swamps" to "Data Lakes" because in the latter it is possible to trace the quality of each piece of data. The benefits of limited to technicalities of convenience. It forms the cornerstone of regulatory   requirements   helping organizations to easily implement tough requirements such as GDPR without much human intervention. It helps in improving the efficiency of operations by greatly minimizing the efforts required in debugging as well as analysing its impact. This particular project clearly proves that though Big Data Processing has many complexities yet the management of Big Data Lifecycle can be very transparent visual and intuitive.

1.                    Simmhan Y. Plale B. & Gannon D. (2005). "A Survey of Data Provenance in e-Science." ACM SIGMOD Record 34(3) 31-36. (This foundational paper establishes the core concepts of provenance tracking). 2.                    Moreau L. (2010). "The Foundations for Provenance on the Web." Foundations and Trends in Web Science 2(2-3) 99-241. (Discusses the Open Provenance Model which influenced the graph design). 3.                    Zaharia M. et al. (2016). "Apache Spark: A Unified Engine for Big Data Processing." Communications of the ACM 59(11) 56-65. (Primary reference for the Spark architecture used in the data collection layer). 4.                    Ikeda R. & Widom J. (2009). "Panda: A System  for  Provenance  and Data." Stanford InfoLab Technical Report. (Provides insights into lineage tracing in warehouse environments).