Distributed Stream Processing for Real-Time Healthcare-Motivated Analytics in Multi-Cloud: A Semantics-Aligned Benchmark of Kafka-Centric Pipelines with Flink and Spark Structured Streaming
DOI:
https://doi.org/10.63282/3050-9246.IJETCSIT-V7I1P138Keywords:
Apache Kafka, Apache Flink, Spark Structured Streaming, stream processing, multi-cloud, Delta Lake, benchmarking, fault tolerance, exactly-once semantics, reproducibilityAbstract
Healthcare providers increasingly rely on Kafka-centric streaming architectures for sub-second alerting and continuous analytics over heterogeneous clinical and device events, yet practitioners lack a semantics-aligned benchmark to compare Apache Flink and Spark Structured Streaming in multi-cloud, lakehouse-oriented deployments. We present a benchmark of Kafka-based pipelines using Flink 1.18 and Spark Structured Streaming 3.5 on AWS and Azure, with outputs persisted via a neutral Databricks Delta ingestion job that decouples engine performance from lakehouse commit behavior. Our healthcare-motivated workload suite models stateless FHIR-like validation, stateful event-time window analytics with large patient-level state, and enrichment with deduplication under controlled late arrivals, all under a formal semantic-alignment protocol. Across colocated deployments at 50 k events/s, Flink achieves 3.1x lower p99 alert latency (74 +/- 3.1 ms vs. 231 +/- 8.4 ms) under exactly-once guarantees. RocksDB incremental checkpoints reduce checkpoint duration by 2.6x and recovery time by 2.9x versus Spark’s HDFSBackedStateStore; switching Spark to RocksDBStateStoreProvider narrows the checkpoint gap to 1.4x while leaving Spark’s p99 latency unchanged, confirming the micro-batch trigger as the latency bottleneck. Directional cross-cloud experiments reveal 78–112 ms additional p99 latency, with AWS-to-Azure consistently 18–24 ms slower than Azure-to-AWS. Persistence latency to Delta (SLA-B) is dominated by the ingestion job’s commit interval and is engine-invariant, reframing engine selection for lakehouse architectures.
Downloads
References
[1] A. Rajkomar, E. Oren, K. Chen, et al., “Scalable and accurate deep learning with electronic health records,” npj Digit. Med., vol. 1, art. no. 18, 2018.
[2] C. S. Kruse and A. Beane, “Health information technology continues to show positive effect on medical outcomes: Systematic review,” J. Med. Internet Res., vol. 20, no. 2, e41, 2018.
[3] K. Waehner, “Streaming ETL with Apache Kafka in the healthcare industry,” 2022. [Online]. Available: https://www.kai-waehner.de/blog/2022/04/01/streaming-etl-with-apache-kafka-healthcare-pharma-industry/
[4] P. Carbone, A. Katsifodimos, S. Ewen, V. Markl, S. Haridi, and K. Tzoumas, “Apache Flink™: Stream and batch processing in a single engine,” Bull. IEEE Comput. Soc. Tech. Comm. Data Eng., 2015, pp. 28–38.
[5] M. Armbrust, T. Das, J. Torres, B. Yavuz, S. Zhu, R. Xin, A. Ghodsi, I. Stoica, and M. Zaharia, “Structured Streaming: A Declarative API for Real-Time Applications in Apache Spark,” in Proc. ACM SIGMOD Int. Conf. Manage. Data, 2018, pp. 601–613.
[6] J. Kreps, N. Narkhede, and J. Rao, “Kafka: A distributed messaging system for log processing,” in Proc. NetDB, 2011.
[7] HL7 Int., “HL7 FHIR R4 specification,” 2019. [Online]. Available: https://hl7.org/fhir/R4/
[8] P. Carbone, G. Fóra, S. Ewen, S. Haridi, and K. Tzoumas, “Lightweight asynchronous snapshots for distributed dataflows,” arXiv:1506.08603, 2015.
[9] T. Akidau, A. Balikov, K. Bekiroğlu, et al., “The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing,” Proc. VLDB Endow., vol. 8, no. 12, pp. 1792–1803, 2015.
[10] M. Fragkoulis, P. Carbone, V. Kalavri, and A. Katsifodimos, “A survey on the evolution of stream processing systems,” VLDB J., vol. 33, no. 2, pp. 507–541, 2024.
[11] J. Karimov, T. Rabl, A. Katsifodimos, R. Samarev, H. Heiskanen, and V. Markl, “Benchmarking distributed stream data processing systems,” in Proc. IEEE ICDE, 2018, pp. 1507–1518.
[12] S. Chintapalli, D. Dagit, B. Evans, et al., “Benchmarking streaming computation engines: Storm, Flink and Spark Streaming,” in Proc. IEEE IPDPS Workshops, 2016, pp. 1789–1792.
[13] A. Shukla, S. Chaturvedi, and Y. Simmhan, “RIoTBench: An IoT benchmark for distributed stream processing systems,” Concurr. Comput. Pract. Exp., vol. 29, no. 21, e4257, 2017.
[14] S. Henning and W. Hasselbring, “Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud,” J. Syst. Softw., vol. 208, art. no. 111879, 2024.
[15] R. Lu, G. Wu, B. Xie, and J. Hu, “Stream Bench: Towards Benchmarking Modern Distributed Stream Computing Frameworks,” in Proc. IEEE/ACM Int. Conf. Utility Cloud Comput. (UCC), 2014, pp. 69–78.
[16] N. H. Rotman, Y. Ben-Itzhak, A. Bergman, I. Cidon, I. Golikov, A. Markuze, and E. Zohar, “CloudCast: Characterizing public clouds connectivity,” arXiv:2201.06989, 2022.
[17] M. Armbrust, T. Das, A. Ghodsi, et al., “Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores,” Proc. VLDB Endow., vol. 13, no. 12, pp. 3411–3424, 2020.
[18] P. Carbone, S. Ewen, G. Fóra, S. Haridi, S. Richter, and K. Tzoumas, “State management in Apache Flink®: Consistent stateful distributed stream processing,” Proc. VLDB Endow., vol. 10, no. 12, pp. 1718–1729, 2017.
[19] S. Zeuch, B. Del Monte, J. Karimov, C. Lutz, M. Renz, J. Traub, S. Breß, T. Rabl, and V. Markl, “Analyzing efficient stream processing on modern hardware,” Proc. VLDB Endow., vol. 12, no. 5, pp. 516–530, 2019.
