Microservices Architecture for Scalable Real-Time Data Processing at the Edge
DOI:
https://doi.org/10.63282/3050-9246.IJETCSIT-V5I2P106Keywords:
Microservices, Edge Computing, Real-Time Data Processing, Scalability, IoT, Containerization, Orchestration, Kubernetes, Low-Latency, Fault ToleranceAbstract
Based on the emerging fields of edge computing and IoT that provide increased scalability, low latency, and tolerance to failures in terms of data processing. It cannot meet such demand levels, especially through traditional monolithic architectures, since these are plagued with high latencies arising from processing bottlenecks and centralized architectures that are not scalable. In this paper, architecture for processing raw data in real-time at the edge using microservices architecture is proposed to improve the system’s efficiency and scalability with the help of the containerization technique, orchestration, and event-driven publishing and listening into the system. Splitting the processing, storage, and communication services into individual microservices makes the different services independent and makes adaptability, modularity, and scalability possible. Docker, Kubernetes, gRPC, MQTT, and Apache Kafka are some technologies used for Mesh and easy deployment of edge computing from node to node. Various tests prove that latency is decreased by an empty-nesting 70-90% and resource performance is increased by 40% less cloud reliance in opposition to a regular monolithic framework based in clouds. Some of the critical issues are network instabilities and fluctuations, security threats and concerns regarding their access, and managing scarcity; the future of AI-based orchestrations, federated learning, and 5 G-based edge computing is also explored. The conclusions presented in the paper would indicate that the future of real-time application in industries can be transformed with the help of microservices-based edge architectures in industrial automation, smart cities, health care, and self-sufficient systems. Drawing lessons from this work enhances the knowledge of context-aware scalable and robust edge computing and advances the area of microservices at the edge in practice
Downloads
References
[1] Podduturi, S. M. (2024). Real-time data processing in microservices architectures. International journal of computer engineering and technology (IJCET), 15(6), 760-773.
[2] Edge Computing for Real-Time Data Analytics: Exploring the Use of Edge Computing to Enable Real-Time Data Analytics in IoT Applications, 2024. online. https://thesciencebrigade.com/iotecj/article/view/86
[3] Ortiz, G., Boubeta-Puig, J., Criado, J., Corral-Plaza, D., Garcia-de-Prado, A., Medina-Bulo, I., & Iribarne, L. (2022). A microservice architecture for real-time IoT data processing: A reusable Web of things approach for smart ports. Computer Standards & Interfaces, 81, 103604.
[4] Hossam Abdel Fattah, Edge Computing in IIoT: Enhancing Real-Time Data Processing, 5ghub, online. https://5ghub.us/edge-computing-in-iiot-enhancing-real-time-data-processing/
[5] Berardi, D., Giallorenzo, S., Mauro, J., Melis, A., Montesi, F., & Prandini, M. (2022). Microservice security: a systematic literature review. PeerJ Computer Science, 8, e779.
[6] Xu, R., Jin, W., & Kim, D. (2019). Microservice security agent based on API gateway in edge computing. Sensors, 19(22), 4905.
[7] Scalability and Performance Optimization Strategies in Edge Computing, LinkedIn, online. https://www.linkedin.com/pulse/scalability-performance-optimization-strategies-edge-computing-hooda-qni1c
[8] Monolith vs microservices: Comparing architectures for software delivery, chronosphere, online. https://chronosphere.io/learn/comparing-monolith-and-microservice-architectures-for-software-delivery/
[9] Edge Computing and IoT: Optimizing Data Processing and Analytics, Cyfutuire, online. https://cyfuture.cloud/blog/edge-computing-and-iot-optimizing-data-processing-and-analytics/
[10] Barczak, A., & Barczak, M. (2021). Performance comparison of monolith and microservices based applications. In Proceedings of the 25th World Multi-Conference on Systemics, Cybernetics and Informatics, WMSCI (pp. 120-125).
[11] Microservices Architecture: Transforming Software Development for Scalability and Agility, UBIMINDS, online. https://ubiminds.com/en-us/microservices-architecture/
[12] Joydipta Chakraborthy, Scaling Realtime Big Data Processing Microservices, 2022. online. https://www.linkedin.com/pulse/scaling-realtime-big-data-processing-microservices-chakraborty
[13] Pandiya, D. K., & Charankar, N. (2023). Integration of microservices and AI for real-time data processing. International journal of computer engineering and technology (IJCET), 14(2), 240-254.
[14] Li, D. C., Huang, C. T., Tseng, C. W., & Chou, L. D. (2021). Fuzzy-based microservice resource management platform for edge computing in the Internet of things. Sensors, 21(11), 3800.
[15] Toomwong, N., & Viyanon, W. (2020). Performance Comparison Between Monolith And Microservices Using Docker And Kubernetes.
[16] Edge Computing and Real-Time Data Testing: Enhancing System Reliability, fpgainsights, 2024. online. https://fpgainsights.com/test-measurement/edge-computing-and-real-time-data-testing/
[17] Tusa, F., Clayman, S., Buzachis, A., & Fazio, M. (2024). Microservices and serverless functions lifecycle, performance, and resource utilisation of edge-based real-time IoT analytics. Future Generation Computer Systems, 155, 204-218.
[18] Containerized Data Processing for IoT: Orchestrating Microservices at the Edge, einfochips, online. https://www.einfochips.com/blog/containerized-data-processing-for-iot-orchestrating-microservices-at-the-edge/
[19] Anees, T., Habib, Q., Al-Shamayleh, A. S., Khalil, W., Obaidat, M. A., & Akhunzada, A. (2023). The integration of WoT and edge computing: Issues and challenges. Sustainability, 15(7), 5983.
[20] Porambage, P., Okwuibe, J., Liyanage, M., Ylianttila, M., & Taleb, T. (2018). Survey on multi-access edge computing for the Internet of Things realization. IEEE Communications Surveys & Tutorials, 20(4), 2961-2991.
