Cloud-Based Data Hubs and SQL Pipelines for Real-Time Financial Analytics
DOI:
https://doi.org/10.63282/3050-9246.IJETCSIT-V5I4P110Keywords:
Real-Time Analytics, Financial Data Integration, SQL Pipelines, Data Hubs, Cloud Architecture, Data Lakehouse, Streaming Analytics, DataOps, ETL/ELT, FinTech InfrastructureAbstract
Businesses need to get accurate, up-to-date information from more and more transactional, market, and consumer data sources in today's fast-paced financial environment. Dynamic financial analytics data solutions need to be able to alter, grow, and work quickly. Some of the old data systems that are on-site can't do these things. Cloud-based data hubs let everyone on a team and all of their tools access the same data at the same time. This makes it easier for people to get to, share, and use information. These cloud-based apps can move and change data almost instantly with SQL pipelines. This helps banks and other financial firms get rid of data silos, make decisions faster, and process information faster. When you use SQL pipelines, it's easy to automate the steps of acquiring data, cleaning it, and adding to it. This makes sure that analytical workloads can handle the greatest traffic without getting too slow or missing out on growth opportunities. This article speaks about a good way to set up a cloud-first data analytics system for apps that handle money. Instead of one big data lakehouse, you should employ smaller SQL pipelines. This case study of a medium-sized fintech company shows that this architecture made it feasible to always keep an eye on compliance indicators, portfolio performance, and risk exposure, with query times of less than a second. What will happen next? Building infrastructure is cheaper, you can report crimes to the authorities more quickly, and you can modify your company model quickly. This paper talks about essential design ideas, operational benefits, and lessons gained from using SQL-driven pipelines and cloud data platforms in the real world for firms who want to improve their financial analytics infrastructure
Downloads
References
[1] Poojara, Shivananda, et al. "Serverless data pipelines for IoT data analytics: A cloud vendors perspective and solutions." Predictive Analytics in Cloud, Fog, and Edge Computing: Perspectives and Practices of Blockchain, IoT, and 5G. Cham: Springer International Publishing, 2022. 107-132.
[2] Dutta, Kamalika, and Manasi Jayapal. "Big data analytics for real time systems." Big Data analytics seminar. 2015.
[3] Mishra, Sarbaree, and Sairamesh Konidala. “Automated Data Mapping and Schema Matching For Improving Data Quality in Master Data Management”. International Journal of Emerging Trends in Computer Science and Information Technology, vol. 4, no. 3, Oct. 2023, pp. 80-90
[4] Guntupalli, Bhavitha. “ETL Architecture Patterns: Hub-and-Spoke, Lambda, and More”. International Journal of AI, BigData, Computational and Management Studies, vol. 4, no. 3, Oct. 2023, pp. 61-71
[5] Yaganti, Dheerendra. "Leveraging .NET for Real-Time Big Data Analytics and Decision Support Systems." European Journal of Advances in Engineering and Technology 8.2 (2021): 155-160.
[6] Shaik, Babulal, Jayaram Immaneni, and K. Allam. "Unified Monitoring for Hybrid EKS and On-Premises Kubernetes Clusters." Journal of Artificial Intelligence Research and Applications 4.1 (2024): 649-669.
[7] Manda, J. K. "Data privacy and GDPR compliance in telecom: ensuring compliance with data privacy regulations like GDPR in telecom data handling and customer information management." MZ Comput J 3.1 (2022).
[8] Koppad, Saraswati, et al. "Cloud computing enabled big multi-omics data analytics." Bioinformatics and biology insights 15 (2021): 11779322211035921.
[9] Lalith Sriram Datla, and Samardh Sai Malay. “Patient-Centric Data Protection in the Cloud: Real-World Strategies for Privacy Enforcement and Secure Access”. European Journal of Quantum Computing and Intelligent Agents, vol. 8, Aug. 2024, pp. 19-43
[10] Immaneni, J., & Salamkar, M. (2020). Cloud migration for fintech: how kubernetes enables multi-cloud success. International Journal of Emerging Trends in Computer Science and Information Technology, 1(3), 17-28.
[11] Mishra, Sarbaree, et al. “Hyperfocused Customer Insights Based On Graph Analytics and Knowledge Graphs”. International Journal of AI, BigData, Computational and Management Studies, vol. 4, no. 4, Dec. 2023, pp. 88-99
[12] Chen, Weisi, et al. "Real-time analytics: Concepts, architectures, and ML/AI considerations." IEEE Access 11 (2023): 71634-71657.
[13] Shaik, Babulal. "Developing Predictive Autoscaling Algorithms for Variable Traffic Patterns." Journal of Bioinformatics and Artificial Intelligence 1.2 (2021): 71-90.
[14] Nookala, G., Gade, K. R., Dulam, N., & Thumburu, S. K. R. (2024). Post-quantum cryptography: Preparing for a new era of data encryption. MZ Computing Journal, 5(2), 012077.
[15] RABHI, FETHI A., and ANDREW BERRY. "Real-Time Analytics: Concepts, Architectures, and ML/AI Considerations."
[16] Allam, Hitesh. "Bridging the Gap: Integrating DevOps Culture into Traditional IT Structures." International Journal of Emerging Trends in Computer Science and Information Technology 3.1 (2022): 75-85.
[17] Guntupalli, Bhavitha, and Surya Vamshi ch. “Designing Microservices That Handle High-Volume Data Loads”. International Journal of AI, BigData, Computational and Management Studies, vol. 4, no. 4, Dec. 2023, pp. 76-87
[18] Lalith Sriram Datla, and Samardh Sai Malay. “Data-Driven Cloud Cost Optimization: Building Dashboards That Actually Influence Engineering Behavior”. Los Angeles Journal of Intelligent Systems and Pattern Recognition, vol. 4, Feb. 2024, pp. 254-76
[19] Patel, Piyushkumar. "The Implementation of Pillar Two: Global Minimum Tax and Its Impact on Multinational Financial Reporting." Australian Journal of Machine Learning Research & Applications 1.2 (2021): 227-46.
[20] Selvarajan, Guru Prasad. "Optimising Machine Learning Workflows in SnowflakeDB: A Comprehensive Framework Scalable Cloud-Based Data Analytics." Technix International Journal for Engineering Research 8.11 (2021).
[21] Mishra, Sarbaree. “Incorporating Automated Machine Learning and Neural Architecture Searches to Build a Better Enterprise Search Engine”. International Journal of Artificial Intelligence, Data Science, and Machine Learning, vol. 4, no. 4, Dec. 2023, pp. 65-75
[22] Balkishan Arugula, and Vasu Nalmala. “Migrating Legacy Ecommerce Systems to the Cloud: A Step-by-Step Guide”. Los Angeles Journal of Intelligent Systems and Pattern Recognition, vol. 3, Dec. 2023, pp. 342-67
[23] Rosandic, Josip. Real-time streaming data management, processing, analysis and visualisation. Diss. PhD thesis, University of Zagreb, 2022.
[24] Mohammad, Abdul Jabbar. “Dynamic Labor Forecasting via Real-Time Timekeeping Stream”. International Journal of AI, BigData, Computational and Management Studies, vol. 4, no. 4, Dec. 2023, pp. 56-65
[25] Mohammad, Abdul Jabbar. “Chrono-Behavioral Fingerprinting for Workforce Optimization”. International Journal of AI, BigData, Computational and Management Studies, vol. 5, no. 3, Oct. 2024, pp. 91-101
[26] Mohna, Hosne Ara, et al. "AI-ready data engineering pipelines: a review of medallion architecture and cloud-based integration models." American Journal of Scholarly Research and Innovation 1.01 (2022): 319-350.
[27] Mishra, Sarbaree. “The Lifelong Learner - Designing AI Models That Continuously Learn and Adapt To New Datasets”. International Journal of Artificial Intelligence, Data Science, and Machine Learning, vol. 5, no. 1, Mar. 2024, pp. 68-78
[28] Jani, Parth. "Predicting Eligibility Gaps in CHIP Using BigQuery ML and Snowflake External Functions." International Journal of Emerging Trends in Computer Science and Information Technology 3.2 (2022): 42-52.
[29] Nookala, G. (2023). Microservices and Data Architecture: Aligning Scalability with Data Flow. International Journal of Digital Innovation, 4(1).
[30] Manda, Jeevan Kumar. "Zero Trust Architecture in Telecom: Implementing Zero Trust Architecture Principles to Enhance Network Security and Mitigate Insider Threats in Telecom Operations." Journal of Innovative Technologies 5.1 (2022).
[31] Thallam, Naga Surya Teja. "Comparative Analysis of Public Cloud Providers for Big Data Analytics: AWS, Azure, and Google Cloud." International Journal of AI, BigData, Computational and Management Studies 4.3 (2023): 18-29.
[32] Arugula, Balkishan. “AI-Powered Code Generation: Accelerating Digital Transformation in Large Enterprises”. International Journal of AI, BigData, Computational and Management Studies, vol. 5, no. 2, June 2024, pp. 48-57
[33] Jani, Parth, and Sarbaree Mishra. "Governing Data Mesh in HIPAA-Compliant Multi-Tenant Architectures." International Journal of Emerging Research in Engineering and Technology 3.1 (2022): 42-50.
[34] Veluru, Sai Prasad. "Threat Modeling in Large-Scale Distributed Systems." International Journal of Emerging Research in Engineering and Technology 1.4 (2020): 28-37.
[35] Allam, Hitesh. "Declarative Operations: GitOps in Large-Scale Production Systems." International Journal of Emerging Trends in Computer Science and Information Technology 4.2 (2023): 68-77.
[36] Mohammad, Abdul Jabbar, and Seshagiri Nageneini. “Temporal Waste Heat Index (TWHI) for Process Efficiency”. International Journal of Emerging Research in Engineering and Technology, vol. 3, no. 1, Mar. 2022, pp. 51-63
[37] Chaganti, Krishna C. "Leveraging Generative AI for Proactive Threat Intelligence: Opportunities and Risks." Authorea Preprints.
[38] Shaik, Babulal. "Automating Compliance in Amazon EKS Clusters With Custom Policies." Journal of Artificial Intelligence Research and Applications 1.1 (2021): 587-10.
[39] Dubuc, Timothée, Frederic Stahl, and Etienne B. Roesch. "Mapping the big data landscape: technologies, platforms and paradigms for real-time analytics of data streams." IEEE Access 9 (2020): 15351-15374.
[40] Lalith Sriram Datla, and Samardh Sai Malay. “From Drift to Discipline: Controlling AWS Sprawl Through Automated Resource Lifecycle Management”. American Journal of Cognitive Computing and AI Systems, vol. 8, June 2024, pp. 20-43
[41] Talakola, Swetha. “Analytics and Reporting With Google Cloud Platform and Microsoft Power BI”. International Journal of Artificial Intelligence, Data Science, and Machine Learning, vol. 3, no. 2, June 2022, pp. 43-52
[42] Nookala, G., Gade, K. R., Dulam, N., & Thumburu, S. K. R. (2024). Building Cross-Organizational Data Governance Models for Collaborative Analytics. MZ Computing Journal, 5(1).
[43] Abdul Jabbar Mohammad. “Timekeeping Accuracy in Remote and Hybrid Work Environments”. American Journal of Cognitive Computing and AI Systems, vol. 6, July 2022, pp. 1-25
[44] Jaiswal, Jitendra Kumar. "Cloud Computing for Big Data Analytics Projects." (2018).
[45] Guntupalli, Bhavitha. “Data Lake Vs. Data Warehouse: Choosing the Right Architecture”. International Journal of Artificial Intelligence, Data Science, and Machine Learning, vol. 4, no. 4, Dec. 2023, pp. 54-64
[46] Chaganti, Krishna. "Adversarial Attacks on AI-driven Cybersecurity Systems: A Taxonomy and Defense Strategies." Authorea Preprints.
[47] Sreekandan Nair , S. (2023). Digital Warfare: Cybersecurity Implications of the Russia-Ukraine Conflict. International Journal of Emerging Trends in Computer Science and Information Technology, 4(4), 31-40. https://doi.org/10.63282/7a3rq622
[48] Manda, J. K. "Cybersecurity Automation in Telecom: Implementing Automation Tools and Technologies to Enhance Cybersecurity Incident Response and Threat Detection in Telecom Operations." Advances in Computer Sciences 4.1 (2021).
[49] Mishra, Sarbaree, and Jeevan Manda. “Improving Real-Time Analytics through the Internet of Things and Data Processing at the Network Edge ”. International Journal of Emerging Research in Engineering and Technology, vol. 5, no. 2, June 2024, pp. 41-51
[50] Abdul Jabbar Mohammad. “Biometric Timekeeping Systems and Their Impact on Workforce Trust and Privacy”. Journal of Artificial Intelligence & Machine Learning Studies, vol. 8, Oct. 2024, pp. 97-123
[51] Ansari, Aftab. "Evaluation of cloud based approaches to data quality management." (2016).
[52] Arugula , Balkishan. “Ethical AI in Financial Services: Balancing Innovation and Compliance”. International Journal of Artificial Intelligence, Data Science, and Machine Learning, vol. 5, no. 3, Oct. 2024, pp. 46-54
[53] Vasanta Kumar Tarra, and Arun Kumar Mittapelly. “AI-Driven Fraud Detection in Salesforce CRM: How ML Algorithms Can Detect Fraudulent Activities in Customer Transactions and Interactions”. American Journal of Data Science and Artificial Intelligence Innovations, vol. 2, Oct. 2022, pp. 264-85
[54] Allam, Hitesh. “Unifying Operations: SRE and DevOps Collaboration for Global Cloud Deployments”. International Journal of Emerging Research in Engineering and Technology, vol. 4, no. 1, Mar. 2023, pp. 89-98
[55] Saleem, Saima, and Monica Mehrotra. "Data analytics and mining: platforms for real-time applications." Data driven decision making using analytics. CRC Press, 2021. 61-80.
[56] Shantharajah, S. P., and E. Maruthavani. "A survey on challenges in transforming No-SQL data to SQL data and storing in cloud storage based on user requirement." International Journal of Performability Engineering 17.8 (2021): 703.
