Latency Optimization Techniques for Near Real‑Time Market Data
DOI:
https://doi.org/10.63282/3050-9246.IJETCSIT-V7I2P142Keywords:
Market Data Latency, High-Frequency Trading, Kernel-Bypass Networking, DPDK, FPGA Acceleration, LMAX Disruptor, NUMA-Aware Computing, UDP Multicast, RDMAAbstract
High-frequency trading (HFT) and algorithmic market-making have reduced the latency between sub-micro second and now challenged near real-time market data infrastructure with one of the most challenging engineering problems in distributed-systems engineering in the modern finance sector. The paper includes a detailed scientific study of methods of minimizing latency of an entire market data consumer system, including both the hardware interface (network) and the consumer software applications at the top. A cumulative sum of all the optimization layers resulted in a 57.9 times decrease in median end-to-end message latency, a 45.2 µs median Latency on a standard Linux stack to 0.78 u Latency with an FPGA feed handler, and a 99th-percentile Latency of 1.38 u Latency on a standard Linux stack to that of 1.38 u Latency on an FPGA feed handler. Regulatory considerations and implications of MiFID II and Reg NMS are addressed and nine practical recommendations are given to practitioners at various levels of the latency-optimization investment spectrum. CXL memory pooling, P4-programmable in-network normalization, and quantum key-distribution of low-latency secure feed channels are the future directions.
Downloads
References
[1] S. Vinnakota, “Optimizing Real-Time Bidding (RTB) Latency in Ad Exchanges: A Comprehensive Analysis,” Journal of Computer Science and Technology Studies, vol. 7, no. 7, pp. 640–650, Jul. 2025, doi: https://doi.org/10.32996/jcsts.2025.7.7.72.
[2] A. Irshad, “Latency Optimization in Edge vs. Cloud Computing: A Comparative Study for Real-Time Applications,” International Journal of Business & Computational Science, vol. 1, no. 1, 2024, doi: https://doi.org/10.0786/37rr2j62.
[3] W. Afua, N. A. O. Ajayi-Nifise, G. Bello, S. Tubokirifuruar, N. O. Odeyemi, and N. T. Falaiye, “Algorithmic Trading and AI: A Review of Strategies and Market Impact,” World Journal of Advanced Engineering Technology and Sciences, vol. 11, no. 1, pp. 258–267, Feb. 2024, doi: https://doi.org/10.30574/wjaets.2024.11.1.0054.
[4] H. Le and A. Gregoriou, “How do you capture liquidity? a review of the literature on low‐frequency stock liquidity,” Journal of Economic Surveys, vol. 34, no. 5, pp. 1170–1186, Jul. 2020, doi: https://doi.org/10.1111/joes.12385.
[5] S. V. Cortés-Calderón, M. D. López-Rodríguez, A. Jiménez-Aceituno, A. J. Castro, and M. Mancilla-García, “Contributions of Net-Map to sustainability action research,” Current Opinion in Environmental Sustainability, vol. 75, p. 101542, May 2025, doi: https://doi.org/10.1016/j.cosust.2025.101542.
[6] E. Freitas, Assis, Pedro, D. F. H. Sadok, and J. Kelner, “Takeaways from an experimental evaluation of eXpress Data Path (XDP) and Data Plane Development Kit (DPDK) under a Cloud Computing environment,” Research, Society and Development, vol. 11, no. 12, p. e26111234200-e26111234200, Sep. 2022, doi: https://doi.org/10.33448/rsd-v11i12.34200.
[7] R. Steinert et al., “Service Provider DevOps network capabilities and tools,” 2026. [Online]. Available: https://arxiv.org/abs/1510.02818
[8] X. Wang et al., “A data privacy protection method for infectious disease prediction models with balanced training speed and accuracy,” Scientific Reports, vol. 16, no. 1, Feb. 2026, doi: https://doi.org/10.1038/s41598-026-38906-9.
[9] J. Dokulil, S. Benkner, and J. Yaghob, “The Open Community Runtime on the Intel Knights Landing Architecture,” Lecture Notes in Computer Science, pp. 801–813, 2017, doi: https://doi.org/10.1007/978-3-319-65482-9_65.
[10] M. Friedman, M. Herlihy, V. Marathe, and E. Petrank, “A persistent lock-free queue for non-volatile memory,” ACM SIGPLAN Notices, vol. 53, no. 1, pp. 28–40, Feb. 2018, doi: https://doi.org/10.1145/3200691.3178490.
[11] O. Arcas-Abella et al., “Hardware Acceleration for Query Processing: Leveraging FPGAs, CPUs, and Memory,” Computing in Science & Engineering, vol. 18, no. 1, pp. 80–87, Jan. 2016, doi: https://doi.org/10.1109/mcse.2016.16.
[12] L. Rosa, L. Foschini, and A. Corradi, “Empowering Cloud Computing With Network Acceleration: A Survey,” IEEE Communications surveys and tutorials/IEEE communications surveys and tutorials, pp. 1–1, Jan. 2024, doi: https://doi.org/10.1109/comst.2024.3377531.
[13] S. Popic, D. Pezer, B. Mrazovac, and N. Teslic, “Performance evaluation of using Protocol Buffers in the Internet of Things communication,” Oct. 2016, doi: https://doi.org/10.1109/sst.2016.7765670.
[14] P. Shantharama, A. S. Thyagaturu, and M. Reisslein, “Hardware-Accelerated Platforms and Infrastructures for Network Functions: A Survey of Enabling Technologies and Research Studies,” IEEE Access, vol. 8, pp. 132021–132085, 2020, doi: https://doi.org/10.1109/access.2020.3008250.
[15] K. Yasukata, M. Honda, D. Santry, and L. Eggert, "StackMap: Low-Latency Networking with the OS Stack and Dedicated NICs," Berkeley, Calif.: USENIX Association, 2016.
[16] H. N. Schuh et al., “CC-NIC: a Cache-Coherent Interface to the NIC,” pp. 52–68, Apr. 2024, doi: https://doi.org/10.1145/3617232.3624868.
[17] S. Arslan, S. Ibanez, A. Mallery, C. Kim, and N. McKeown, “NanoTransport,” pp. 13–26, Oct. 2021, doi: https://doi.org/10.1145/3482898.3483365.
[18] M. Lipiński, “The Inclusion of White Rabbit into the Global Industry Standard IEEE 1588,” JACoW, vol. ICALEPCS2021, pp. 126–131, 2022, doi: https://doi.org/10.18429/JACoW-ICALEPCS2021-MOPV011.
[19] O. Kode and T. Oyemade, “Analysis of Synchronization Mechanisms in Operating Systems,” 2024. [Online]. Available: https://arxiv.org/abs/2409.11271
