A Scalable Distributed Data Processing Framework for Large-Scale Heterogeneous Data Integration
DOI:
https://doi.org/10.63282/3117-5481/AIJCST-V6I1P114Keywords:
Distributed Data Processing, Heterogeneous Data Integration, Apache Hadoop, Apache Spark, Batch Data Processing, Parallel Computing, Scalability OptimizationAbstract
Because a multitude of heterogeneous data is created across enterprise information systems there is skyrocketing demand for scalable and efficient distributed data processing frameworks. Computational constraints, processing delays and scalability challenges provide major limitations to traditional data integration solutions, especially dealing with large amount of structured and semi-structured data. In this paper, integration using the Apache Hadoop and Apache Spark big data technologies has been proposed which can achieve the functionality of distributing and processing the data in an upward scalable way. Proposed Architecture: An integrated Processing Architecture for the customers to batch ingest data, perform transformations, data partitioning and distributed data parallel processing. It is all about reconciling a ton of different kinds of data, and making data processing more efficient and ensuring data can be transferred reliably from one end of the enterprise data spectrum – to the other. The distributed storage system, workload distribution and resource usage and the optimization of performance for enabling data-intensive operations are explored as well. Some of these would be data partition strategies, parallelism of data and how hadoop and spark can complement each other in processing the data/computational workloads. Scalability is another important factor covered by the framework, such as data expansion, cluster sources and resources management as well as the processing bottlenecks and versatility of the infrastructure. Overall, the solution can be used to provide flexible data processing, efficiency in data processing, distributed data processing, and can be used to establish scalable data platforms for enterprises to combine various types of data.
References
[1] Silva, B., Moreira, J., & Costa, R. L. C. (2023). Logical big data integration and near real-time data analytics. Data & Knowledge Engineering, 146, 102185. https://doi.org/10.1016/j.datak.2023.102185
[2] Costa, R. L. de C., Moreira, J., Pintor, P., dos Santos, V., & Lifschitz, S. (2021). A survey on data-driven performance tuning for big data analytics platforms. Big Data Research, 25, 100206. https://doi.org/10.1016/j.bdr.2021.100206
[3] Habeeb, R. A. A., Nasaruddin, F., Gani, A., Targio Hashem, I. A., Ahmed, E., & Imran, M. (2019). Real-time big data processing for anomaly detection: A survey. International Journal of Information Management, 45, 289–307. https://doi.org/10.1016/j.ijinfomgt.2018.08.006
[4] Hajjaji, Y., Boulila, W., Farah, I. R., Romdhani, I., & Hussain, A. (2021). Big data and IoT-based applications in smart environments: A systematic review. Computer Science Review, 39, 100318. https://doi.org/10.1016/j.cosrev.2020.100318
[5] Küfner, T., Schönig, S., Jasinski, R., & Ermer, A. (2021). Vertical data continuity with lean edge analytics for Industry 4.0 production. Computers in Industry, 125, 103389. https://doi.org/10.1016/j.compind.2020.103389
[6] Tan, R., Chirkova, R., Gadepally, V., & Mattson, T. G. (2017). Enabling query processing across heterogeneous data models: A survey. In 2017 IEEE International Conference on Big Data (Big Data) (pp. 3211–3220). IEEE. https://doi.org/10.1109/BigData.2017.8258302
[7] Kolev, B., Valduriez, P., Bondiombouy, C., Jiménez-Peris, R., Pau, R., & Pereira, J. (2016). CloudMdsQL: Querying heterogeneous cloud data stores with a common language. Distributed and Parallel Databases, 34(4), 463–503. https://doi.org/10.1007/s10619-015-7185-y
[8] Sethi, R., Traverso, M., Sundstrom, D., Phillips, D., Xie, W., Sun, Y., Yegitbasi, N., Jin, H., Hwang, E., Shingte, N., & Berner, C. (2019). Presto: SQL on everything. In 2019 IEEE 35th International Conference on Data Engineering (ICDE) (pp. 1802–1813). IEEE. https://doi.org/10.1109/ICDE.2019.00196
[9] Silva, B., Moreira, J., & Costa, R. L. C. (2021). EasyBDI: Near real-time data analytics over heterogeneous data sources. In Proceedings of the 24th International Conference on Extending Database Technology (EDBT) (pp. 702–705). https://doi.org/10.5441/002/edbt.2021.88
[10] Garg, B., & Kaur, K. (2015). Integration of heterogeneous databases. In 2015 International Conference on Advances in Computer Engineering and Applications (pp. 1033–1038). IEEE. https://doi.org/10.1109/ICACEA.2015.7164859
[11] Rahm, E., & Bernstein, P. A. (2001). A survey of approaches to automatic schema matching. The VLDB Journal, 10, 334–350. https://doi.org/10.1007/s007780100057
[12] Özsu, M. T., & Valduriez, P. (2020). Principles of distributed database systems (4th ed.). Springer. https://doi.org/10.1007/978-3-030-26253-2
[13] Ali, A., Nordin, A., Alzeber, M., & Zaid, A. (2017). A survey of schema matching research using database schemas and instances. International Journal of Advanced Computer Science and Applications, 8(10). https://doi.org/10.14569/IJACSA.2017.081014
[14] Bonifati, A., Mecca, G., Papotti, P., & Velegrakis, Y. (2011). Discovery and correctness of schema mapping transformations. In Schema matching and mapping (pp. 111–147). Springer. https://doi.org/10.1007/978-3-642-16518-4_5
[15] Abadi, D., Ailamaki, A., Andersen, D., Bailis, P., Balazinska, M., Bernstein, P., Boncz, P., Chaudhuri, S., Cheung, A., Doan, A., et al. (2020). The Seattle report on database research. ACM SIGMOD Record, 48(4), 44–53.
