Hybrid Cloud-Oriented Architecture for Big Data Processing with Adaptive Resource Allocation and Energy Optimization
DOI:
https://doi.org/10.63282/3117-5481/AIJCST-V3I1P102Keywords:
Hybrid Cloud, Big Data, Adaptive Resource Allocation, Energy Optimization, Container Virtualization, Predictive Modeling, Machine Learning, Sustainable ComputingAbstract
The current and fast development of big data has created challenges of having never seen before to the conventional computing infrastructures and it has compelled the development of innovative designs that have the capability of handling vast numbers, velocity and scale of data. The hybrid cloud-based architectures offer a future solution based on the integration of resources of both the private and the public cloud resources, and consequently create scalable, flexible and cost effective computing solution. It is the hypothesis of this paper to develop an innovative hybrid cloud-based architecture of big data processing that combines adaptive resources allocation plans and energy optimization methodologies. Our solution is dynamically deployed to balance the computational resources between the private and the public clouds according to the workload requirements and optimization of energy consumption without affecting the performance. Experimental measurements show that there is a large-scale energy consumption with minimal throughput and latency. The containerized framework proposed is able to use the power of container based virtualization, predictive workload modeling, and machine learning algorithms in making intelligent decisions throughout resource provisioning. This study offers a viable solution to the future of the large scale big data system by offering a balance between power consumption and the capacity to compute
References
[1] Manzoor, M. F., et al. (2020). Resource Allocation Techniques in Cloud Computing: A Review and Future Directions. Journal of Cloud Computing: Advances, Systems and Applications, 9(1), 1-20.
[2] A novel energy efficient resource allocation algorithm based on immune clonal optimization for green cloud computing (2014)
[3] Adaptive Provisioning of Heterogeneous Cloud Resources for Big Data Processing (2019)
[4] Energy efficient temporal load aware resource allocation in cloud computing datacenters (2018) — Shahin Vakilinia.
[5] Green Cloud Software Engineering for Big Data Processing (Ganesan et al., 2020)
[6] Energy Efficient Management of Data Center Resources for Cloud Computing: A Vision, Architectural Elements, and Open Challenges – Buyya, Beloglazov & Abawajy (2010)
[7] Integrated Green Cloud Computing Architecture – M.N. Hulkury & M.R. Doomun (2012)
[8] Enabling Big Data Analytics in the Hybrid Cloud Using Iterative MapReduce for UCC 2015 (2015) — Francisco J. Clemente Castello et al.
[9] SLA Oriented Resource Provisioning for Cloud Computing: Challenges, Architecture, and Solutions – R. Buyya, S.K. Garg & R.N. Calheiros (2012)
[10] Cloud Computing and Emerging IT Platforms: Vision, Hype, and Reality for Delivering Computing as the 5th Utility (Buyya, Yeo, Venugopal, Broberg & Brandic, 2009)
[11] Nimrod/G: An Architecture of a Resource Management and Scheduling System in a Global Computational Grid — Rajkumar Buyya, David Abramson & Jon Giddy, 2000.
[12] Optimizing capacity allocation for big data applications in cloud datacenters — Spicuglia, Chen, Binder et al. (2015)
[13] Kanji, R. K. (2020). Federated Learning in Big Data Analytics Privacy and Decentralized Model Training. Journal of Scientific and Engineering Research, 7(3), 343-352.
[14] Optimizing LTE RAN for High-Density Event Environments: A Case Study from Super Bowl Deployments - Varinder Kumar Sharma - IJAIDR Volume 11, Issue 1, January-June 2020. DOI 10.71097/IJAIDR.v11.i1.1542
[15] Thallam, N. S. T. (2020). Comparative Analysis of Data Warehousing Solutions: AWS Redshift vs. Snowflake vs. Google BigQuery. European Journal of Advances in Engineering and Technology, 7(12), 133-141.
