A Comprehensive Framework for End-to-End Data Lineage, Anomaly Detection, and Quality Validation in Enterprise-Scale Distributed Data Processing Pipelines
DOI:
https://doi.org/10.63282/3117-5481/AIJCST-V7I2P112Keywords:
Data Lineage, Anomaly Dsetection, Data Quality, Distributed Pipelines, Root-Cause Analysis, ReproducibilityAbstract
Enterprise data pipelines face challenges from schema changes, distributional drift, and limited coordination between data lineage, anomaly detection, and data-quality validation. This study reviews 23 provenance resources and proposes a layered reference architecture integrating these capabilities through a shared lineage graph. The framework introduces a formal lineage model, lineage-aware anomaly detection, a six-dimensional quality score, and root-cause and impact analysis. Although the framework has not been implemented or experimentally evaluated, existing literature provides supporting performance results while highlighting limited evidence of integrated evaluation. The proposed architecture addresses this gap by enabling unified monitoring, improved defect identification, and downstream impact assessment.
References
[1] H. Ren, B. Xu, Y. Wang, C. Yi, C. Huang, X. Kou, T. Xing, M. Yang, J. Tong, and Q. Zhang, "Time-series anomaly detection service at Microsoft," in Proc. 25th ACM SIGKDD Conf. Knowledge Discovery & Data Mining (KDD '19), Anchorage, AK, USA, 2019, pp. 3009–3017. doi: 10.1145/3292500.3330680.
[2] D. Tu, Y. He, W. Cui, S. Ge, H. Zhang, H. Shi, D. Zhang, and S. Chaudhuri, "Auto-validate by-history: Auto-program data quality constraints to validate recurring data pipelines," in Proc. 29th ACM SIGKDD Conf. Knowledge Discovery & Data Mining (KDD '23), 2023. doi: 10.1145/3580305.3599776.
[3] E. Widad, E. Saida, and Y. Gahi, "Quality anomaly detection using predictive techniques: An extensive big data quality framework for reliable data analysis," IEEE Access, vol. 11, 2023. doi: 10.1109/ACCESS.2023.3317354.
[4] K. R. Chirumamilla, "Predicting data contract failures using machine learning," The Eastasouth Journal of Information System and Computer Science, vol. 1, no. 1, pp. 144–155, Aug. 2023.
[5] K. Ahmed, A. Altaf, N. S. Mohd Jamail, F. Iqbal, and R. Latif, "ADAL-NN: Anomaly detection and localization using deep relational learning in distributed systems," Applied Sciences, vol. 13, no. 12, art. 7297, 2023. doi: 10.3390/app13127297.
[6] A. Lavin and S. Ahmad, "Evaluating real-time anomaly detection algorithms the Numenta Anomaly Benchmark," in Proc. 14th Int. Conf. Machine Learning and Applications (ICMLA), Miami, FL, USA, 2015. Preprint: arXiv:1510.03336.
[7] N. Polyzotis, S. Roy, S. E. Whang, and M. Zinkevich, "Data management challenges in production machine learning," in Proc. ACM SIGMOD Int. Conf. Management of Data (tutorial), 2017. doi: 10.1145/3035918.3054782.
[8] S. Schelter, D. Lange, P. Schmidt, M. Celikel, F. Biessmann, and A. Grafberger, "Automating large-scale data quality verification," Proc. VLDB Endowment, vol. 11, no. 12, 2018. doi: 10.14778/3229863.3229867.
[9] M. Franklin, A. Halevy, and D. Maier, "From databases to dataspaces: A new abstraction for information management," in Proc. ACM SIGMOD Record / CIDR 2005, 2005.
[10] L. Ehrlinger and W. Wöß, "A survey of data quality measurement and monitoring tools," Frontiers in Big Data, vol. 5, art. 850611, Mar. 2022. doi: 10.3389/fdata.2022.850611.
[11] M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, and I. Stoica, "Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing," in Proc. 9th USENIX Symp. Networked Systems Design and Implementation (NSDI), 2012.
[12] T. Rekatsinas, X. Chu, I. F. Ilyas, and C. Ré, "HoloClean: Holistic data repairs with probabilistic inference," Proc. VLDB Endowment, vol. 10, no. 11, pp. 1190–1201, 2017. doi: 10.14778/3137628.3137631.
[13] K.-K. Muniswamy-Reddy, D. A. Holland, U. Braun, and M. Seltzer, "Provenance-aware storage systems," in Proc. USENIX Annual Technical Conference, 2006.
[14] S. B. Davidson and J. Freire, "Provenance and scientific workflows: Challenges and opportunities," in Proc. ACM SIGMOD Int. Conf. Management of Data, 2008, pp. 1345–1350. doi: 10.1145/1376616.1376772.
[15] Y. L. Simmhan, B. Plale, and D. Gannon, "A survey of data provenance in e-science," ACM SIGMOD Record, vol. 34, no. 3, 2005. doi: 10.1145/1084805.1084812.
[16] R. Bose and J. Frew, "Lineage retrieval for scientific data processing: A survey," ACM Computing Surveys, vol. 37, no. 1, pp. 1–28, Mar. 2005. doi: 10.1145/1057977.1057978.
[17] M. Zaharia, R. S. Xin, P. Wendell, T. Das, M. Armbrust, A. Dave, X. Meng, J. Rosen, S. Venkataraman, M. J. Franklin, A. Ghodsi, J. Gonzalez, S. Shenker, and I. Stoica, "Apache Spark: A unified engine for big data processing," Communications of the ACM, vol. 59, no. 11, 2016. doi: 10.1145/2934664.
[18] M. Armbrust, A. Ghodsi, R. Xin, and M. Zaharia, "Lakehouse: A new generation of open platforms that unify data warehousing and advanced analytics," in Proc. 11th Conf. Innovative Data Systems Research (CIDR), 2021.
[19] P. Carbone, S. Ewen, S. Haridi, A. Katsifodimos, V. Markl, and K. Tzoumas, "Apache Flink: Stream and batch processing in a single engine," IEEE Data Engineering Bulletin, vol. 38, no. 4, pp. 28–38, 2015.
[20] J. Kreps, N. Narkhede, and J. Rao, "Kafka: A distributed messaging system for log processing," in Proc. NetDB Workshop, 2011.
[21] J. Dean and S. Ghemawat, "MapReduce: Simplified data processing on large clusters," in Proc. 6th USENIX Symp. Operating Systems Design and Implementation (OSDI), 2004.
[22] Z. Abedjan, X. Chu, D. Deng, R. Castro Fernandez, I. F. Ilyas, M. Ouzzani, P. Papotti, M. Stonebraker, and N. Tang, "Detecting data errors: Where are we and what needs to be done?," Proc. VLDB Endowment, vol. 9, no. 12, pp. 993–1004, 2016. doi: 10.14778/2994509.2994518.
[23] M. Zipperle, F. Gottwalt, E. Chang, and T. Dillon, "Provenance-based intrusion detection systems: A survey," ACM Computing Surveys, vol. 55, no. 7, 2022/2023. doi: 10.1145/3539605.
[24] U. Allala, “Predictive Incident Management Using Observability Data in Cloud-Native Systems”, IJETMR, vol. 3, no. 1, pp. 01–19, Mar. 2020, doi: 10.67228/30715636/IJETMR-2020PI2V8C.
[25] Gajula, S. (2024). Cybersecurity risk prediction using graph neural networks. Journal of Information Systems Engineering and Management.
