Hybrid NoSQL-SQL Database Architectures for Real-Time Big Data Analytics and Scalable Cloud Applications
DOI:
https://doi.org/10.63282/3117-5481/AIJCST-V6I1P105Keywords:
Hybrid Database, Nosql, SQL, Real-Time Data, Scalable Systems, Stream Processing Frameworks, Data Management, Cloud ComputingAbstract
The increasing data volume, velocity and variety has been a serious challenge to the conventional data management systems. To overcome scalability, heterogeneity and low-latency processing issues, this paper discusses a hybrid SQL-NoSQL architecture, which combines in-memory databases (MMDB), relational databases (RDB) and distributed file systems (DFS) to manage spatio-temporal big data efficiently. The framework that is proposed allows real-time ingestion of data, on-the-fly processing of analytics, and scalable long-term data storage using a single integrated hybrid framework. The paper also analyzes data interoperability, ACID- BASIS consistency trade-offs, streaming systems/frameworks, e.g., Spark, Flink and Storm and event-driven high-velocity analytics architectures. Also, the use of AI/ML methods in predictive scaling and performance optimization of cloud-based systems is pointed out. Literature review is conducted to establish major gaps in the areas of schema transformation, distributed query optimization, and hybrid consistency management. The results reveal that hybrid SQL-NoSQL systems provide a robust platform of real-time analysis of large amounts of data and scalability cloud applications as well as modern decision-making systems. On the overall, the study presents a thorough architectural framework to the creation of scalable, real-time, big data analytics systems.
References
[1] S. Venkatraman, S. Kaspi, Kiran Fahd, and R. Venkatraman, “SQL Versus NoSQL Movement with Big Data Analytics,” Int. J. Inf. Technol. Comput. Sci., vol. 8, no. 12, pp. 59–66, Dec. 2016, doi: 10.5815/ijitcs.2016.12.07.
[2] S. A. Hossain, “NoSQL Database : New Era of Databases for Big data Analytics - Classification , Characteristics and Comparison,” vol. 6, no. 4, pp. 1–14, 2013.
[3] C. Băzăr, “The Transition from RDBMS to NoSQL. A Comparative Analysis of Three Popular Non-Relational Solutions: Cassandra, MongoDB and Couchbase,” Database Syst. Journa, vol. V, no. 2, pp. 49–59, 2014.
[4] V. K. Tambi, “Analysis Of Sql And Nosql Database Management,” vol. 2, no. 3, pp. 99–113, 2015.
[5] B. R. Cherukuri, “Future of cloud computing: Innovations in multi-cloud and hybrid architectures,” World J. Adv. Res. Rev., vol. 1, no. 1, pp. 068–081, 2019.
[6] C. Wu et al., “A NoSQL–SQL Hybrid Organization and Management Approach for Real-Time Geospatial Data: A Case Study of Public Security Video Surveillance,” Int. J. Geo-Information, vol. 6, p. 21, 2017, doi: 10.3390/ijgi6010021.
[7] M. N. Mami, D. Graux, H. Thakkar, S. Scerri, and S. Auer, “The Query Translation Landscape : a Survey,” no. 1, 2019.
[8] N. Fikri, M. Rida, N. Abghour, K. Moussaid, and A. El Omri, “An adaptive and real-time based architecture for financial data integration,” J. Big Data, vol. 6, no. 1, p. 97, 2019, doi: 10.1186/s40537-019-0260-x.
[9] A. Pothuganti, “Big Data Analytics: Hadoop-Map Reduce & NoSQL Databases,” Int. J. Comput. Sci. Inf. Technol., vol. 6, no. 1, 2015.
[10] V. M. L. G. Nerella, “Automated Cross-Platform Database Migration And High Availability Implementation,” Turkish J. Comput. Math. Educ., vol. 9, no. 2, pp. 823–835, Jul. 2018, doi: 10.61841/turcomat.v9i2.15284.
[11] W. Ali, M. U. Shafique, M. A. Majeed, and A. Raza, “Comparison between SQL and NoSQL Databases and Their Relationship with Big Data Analytics,” Asian J. Res. Comput. Sci., pp. 1–10, Oct. 2019, doi: 10.9734/ajrcos/2019/v4i230108.
[12] Y.-L. Choi, W.-S. Jeon, and S.-H. Yoon, “Improving Database System Performance by Applying NoSQL,” J. Inf. Process. Syst., vol. 10, no. 3, pp. 355–364, Sep. 2014, doi: 10.3745/JIPS.04.0006.
[13] A. Kobusińska, C. Leung, C.-H. Hsu, R. S., and V. Chang, “Emerging trends, issues and challenges in Internet of Things, Big Data and cloud computing,” Futur. Gener. Comput. Syst., vol. 87, pp. 416–419, Oct. 2018, doi: 10.1016/j.future.2018.05.021.
[14] A. Kushwaha, P. Pathak, and S. Gupta, “Review of Optimize Load Balancing Algorithms in Cloud.,” Int. J. Distrib. Cloud Comput., vol. 4, no. 2, p. 1, 2016.
[15] B. M. Balachandran and S. Prasad, “Challenges and Benefits of Deploying Big Data Analytics in the Cloud for Business Intelligence,” Procedia Comput. Sci., vol. 112, pp. 1112–1122, 2017, doi: 10.1016/j.procs.2017.08.138.
[16] S. Achouche and U. B. Yalamanch, “Method, apparatus, and computer-readable medium for performing a data exchange on a data exchange framework,” 2019
[17] W. Inoubli, S. Aridhi, H. Mezni, and M. Maddouri, “A Comparative Study on Streaming Frameworks for Big Data,” pp. 17–24, 2018.
[18] Q. Zhou, Y. Simmhan, and V. Prasanna, “Knowledge-infused and consistent Complex Event Processing over real-time and persistent streams,” Futur. Gener. Comput. Syst., vol. 76, pp. 391–406, Nov. 2017, doi: 10.1016/j.future.2016.10.030.
[19] S. Garg, “AI/ML Driven Proactive Performance Monitoring, Resource Allocation and Effective Cost Management in SaaS Operations,” Int. J. Core Eng. Manag., vol. 6, no. 6, pp. 263–273, 2019.
[20] G. A. Schreiner, D. Duarte, and R. dos S. Mello, “When Relational-Based Applications Go to NoSQL Databases: A Survey,” Information, vol. 10, no. 7, 2019, doi: 10.3390/info10070241.
[21] O. Alotaibi and E. Pardede, “Transformation of Schema from Relational Database (RDB) to NoSQL Databases,” Data, vol. 4, no. 4, p. 148, Nov. 2019, doi: 10.3390/data4040148.
[22] K. Ezéchiel, S. Kant, and D. Agarwal, “A systematic review on Distributed Databases Systems and their techniques,” J. Theor. Appl. Inf. Technol., vol. 96, no. 1, 2019.
[23] S. Mazumdar, D. Seybold, K. Kritikos, and Y. Verginadis, “A survey on data storage and placement methodologies for Cloud-Big Data ecosystem,” J. Big Data, vol. 6, no. 1, p. 15, 2019, doi: 10.1186/s40537-019-0178-3.
[24] M. Sarnovsky, P. Bednar, and M. Smatana, “Big Data Processing and Analytics Platform Architecture for Process Industry Factories,” Big Data Cogn. Comput., vol. 2, no. 1, 2018, doi: 10.3390/bdcc2010003.
[25] P. Asghari and H. Zarrabi, “A Survey on NoSQL and its Terminology,” vol. 5, no. 12, pp. 405–410, 2016, doi: 10.17148/IJARCCE.2016.51293.
[26] Mamidala, J. V., Enokkaren, S. J., Attipalli, A., Bitkuri, V., Kendyala, R., & Kurma, J. (2023). Machine Learning Models Powered by Big Data for Health Insurance Expense Forecasting. International Research Journal of Economics and Management Studies IRJEMS, 2(1).
[27] Nadella, V. M. (2023). Zero Trust Architecture for Telecom Operations. International Journal of Emerging Research in Engineering and Technology, 4(3), 115-129.
[28] Bitkuri, V., Kendyala, R., Kurma, J., Enokkaren, S. J., & Mamidala, J. V. (2023). Forecasting Stock Price Movements With Deep Learning Models for time Series Data Analysis. Journal of Artificial Intelligence & Cloud Computing. SRC/JAICC-531. DOI: doi. org/10.47363/JAICC/2023 (2), 489, 2-9.
[29] Nadella, V. M. (2023). Anomaly Detection and Fault Prediction using ML in Telecom Operations. International Journal of Emerging Trends in Computer Science and Information Technology, 4(3), 134-143.
[30] Kosaraju, P., & Nadella, V. M. (2022). Security and Privacy in IoT Ecosystems. Universal Library of Engineering Technology, (Issue).
[31] Singh, A. A. S. S., Mania, V., Kothamaram, R. R., Rajendran, D., Namburi, V. D. N., & Tamilmani, V. (2023). Exploration of Java-Based Big Data Frameworks: Architecture, Challenges, and Opportunities. Journal of Artificial Intelligence & Cloud Computing, 2(4), 1-8.
[32] Routhu, K. K. (2023). AI-driven succession planning in Oracle HCM Cloud: Building resilient leadership pipelines through predictive analytics. International Journal of Science, Engineering and Technology, 11(5).
[33] Tamilmani, V., Namburi, V. D., Singh Singh, A. A., Maniar, V., Kothamaram, R. R., & Rajendran, D. (2023). Real-Time Identification of Phishing Websites Using Advanced Machine Learning Methods. Available at SSRN 5837142.
[34] Routhu, K. K. (2023). AI-driven succession planning in Oracle HCM Cloud: Building resilient leadership pipelines through predictive analytics. International Journal of Science, Engineering and Technology, 11(5). https://doi.org/10.5281/zenodo.17292018
[35] From Fragmentation to Focus: The Benefits of Centralizing Procurement. (2023). International Journal of Research and Applied Innovations, 6(6), 9820-9833. https://doi.org/10.15662/
[36] Routhu, K. K. (2023). Embedding fairness into the digital enterprise, data driven DEI strategies with Oracle HCM Analytics. International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 9(8), 266-274.
[37] Routhu, K. K. (2023). AI-driven skills forecasting in Oracle HCM Cloud: From static competencies to predictive workforce design. International Journal of Science, Engineering and Technology, 11(1).
[38] Padur, S. K. R. (2023). AI-Augmented Enterprise ERP Modernization: Zero-Downtime Strategies for Oracle E-Business Suite R12. 2 and Beyond. Available at SSRN 5605510.
[39] Routhu, K. K. (2022). From Case Management to Conversational HR: Redefining Help Desks with Oracle’s AI and NLP Framework. International Journal of Science, Engineering and Technology, 10(6).
[40] Vattikonda, N., Gupta, A. K., Polu, A. R., Narra, B., Buddula, D. V. K. R., & Patchipulusu, H. H. S. (2022). Blockchain Technology in Supply Chain and Logistics: A Comprehensive Review of Applications, Challenges, and Innovations. International Journal of Emerging Trends in Computer Science and Information Technology, 3(3), 72-80.
[41] Attipalli, A., BITKURI, V., Mamidala, J. V., Kendyala, R., & KURMA, J. (2022). Empowering Cloud Security with Artificial Intelligence: Detecting Threats Using Advanced Machine learning Technologies. Available at SSRN 5741263.
[42] Padur, S. K. R. (2022). Intelligent resource management: AI methods for predictive workload forecasting in cloud data centers. J. Artif. Intell. Mach. Learn. & Data Sci, 1(1), 2936-2941.
[43] Nadella, V. M. (2022). Digital Twins for Predictive Network Management and System Simulation. International Journal of AI, BigData, Computational and Management Studies, 3(3), 100-111.
[44] Routhu, K. K. (2022). From RFID to Geofencing: IoT-Enabled Smart Time Tracking in Oracle HCM Cloud. International Journal of Science, Engineering and Technology, 10(4).
[45] Nadella, V. (2019). Extracting road traffic data through video analysis using automatic camera calibration and deep neural networks.
[46] Polam, R. M., Kamarthapu, B., Kakani, A. B., Nandiraju, S. K. K., Chundru, S. K., & Vangala, S. R. (2022). Data Security in Cloud Computing: Encryption, Zero Trust, and Homomorphic Encryption. International Journal of Emerging Trends in Computer Science and Information Technology, 3(4), 31-41.
[47] Padur, S. K. R. (2022). AI augmented platform engineering, transforming developer experience through intelligent automation and self optimizing internal platforms. International Journal of Science, Engineering and Technology, 10(5), 10-5281.
[48] Kosaraju, P. , & Nadella, V. M. (2021). Quality of Experience (QoE) and Network Performance Modelling for Multimedia Traffic. Journal of Artificial Intelligence and Big Data, 1(1), 1-13. https://doi.org/10.31586/jaibd.2021.1358
