Beyond Static Dumps: A Low-Latency Stateful Streaming and Graph Architecture for Proactive Market Defense

To address the limitations of latency and stale predictions in streaming financial surveillance and algorithmic trading systems, this paper introduces a low-latency, real-time feature and graph intelligence platform built using Apache Kafka, Bytewax, Feast, and Neo4j. The architecture integrates Apache Kafka for high-throughput ingestion of multi-source order, quote, and news streams, Bytewax for stateful stream processing and sliding-window aggregations, and Feast as a unified feature store with a Parquet-based offline data lake and a lightweight SQLite online store ensuring sub-millisecond feature retrieval. Furthermore, Neo4j is embedded to model multi-dimensional trade topologies (Account-Stock-Exchange relationships) for dynamic structural risk and PageRank analytics.
We evaluate the platform using simulated high-frequency trading datasets, measuring real-time feature generation throughput, end-to-end pipeline latency, and anomaly detection performance under simulated market manipulation scenarios. A Spatio-Temporal Graph Neural Network (ST-GNN) anomaly detector trained on stream-computed features and graph embeddings achieved an F1-score of 0.962, significantly outperforming spatial-only (F1: 0.814) and temporal-only (F1: 0.785) baselines.
Ablation studies isolating spatial graph connectivity and temporal sequence history confirmed that removing structural graph edges or temporal memory leads to sharp declines in predictive accuracy. Additionally, robustness testing under injected multi-account spoofing attacks demonstrated that real-time feature updates and graph correlation metrics effectively mitigate model performance decay. The proposed Kafka–Bytewax–Feast–Neo4j pipeline offers a scalable, production-ready framework for low-latency feature engineering and fraud-resistant market surveillance.

Apache Flink, Feast, algorithmic trading, low-latency feature store, high-frequency market microstructure

1. Akidau T, Bradshaw R, Chambers C, Chernyak S, Fernández-Moctezuma RJ, Lax R, McVeety S, Mills D, Perry F, Schmidt E, Whittle S (2015) The Dataflow model: A practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing. Proceedings of the VLDB Endowment 8(12):1792–1803. https://doi.org/10.14778/2824032.2824076 [Google Scholar] [Crossref]

2. Carbone P, Katsifodimos A, Ewen S, Markl V, Haridi S, Tzoumas K (2015) Apache Flink: Stream and batch processing in a single engine. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering 38(4):28–38. [Google Scholar] [Crossref]

3. Chandola V, Banerjee A, Kumar V (2009) Anomaly detection: A survey. ACM Computing Surveys 41(3) 15. https://doi.org/10.1145/1541880.1541882 [Google Scholar] [Crossref]

4. Cheng D, Zou Y, Xiang S, Jiang C (2025) Graph neural networks for financial fraud detection: A review. Frontiers of Computer Science 19(9):199609. https://doi.org/10.1007/s11704-024-40474-y [Google Scholar] [Crossref]

5. Fragkoulis M, Carbone P, Kalavri V, Katsifodimos A (2024) A survey on the evolution of stream processing systems. The VLDB Journal 33:507–541. https://doi.org/10.1007/s00778-023-00819-8 [Google Scholar] [Crossref]

6. Hamilton WL, Ying R, Leskovec J (2017) Inductive representation learning on large graphs. In: Advances in Neural Information Processing Systems 30. https://arxiv.org/abs/1706.02216 [Google Scholar] [Crossref]

7. Holme P, Saramäki J (2012) Temporal networks. Physics Reports 519(3):97–125. https://doi.org/10.1016/j.physrep.2012.03.001 [Google Scholar] [Crossref]

8. Kipf TN, Welling M (2017) Semi-supervised classification with graph convolutional networks. In: International Conference on Learning Representations. https://arxiv.org/abs/1609.02907 [Google Scholar] [Crossref]

9. Kreps J, Narkhede N, Rao J (2011) Kafka: A distributed messaging system for log processing. In: Proceedings of the NetDB Workshop. https://notes.stephenholiday.com/Kafka.pdf [Google Scholar] [Crossref]

10. Kumar S, Zhang X, Leskovec J (2019) Predicting dynamic embedding trajectory in temporal interaction networks. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1269–1278. https://doi.org/10.1145/3292500.3330895 [Google Scholar] [Crossref]

11. Menkveld AJ (2015) High frequency market microstructure. Journal of Financial Economics 116(2):257–270. https://doi.org/10.1016/j.jfineco.2015.01.003 [Google Scholar] [Crossref]

12. Pareja A, Domeniconi G, Chen J, Ma T, Suzumura T, Kanezashi H, Kaler T, Schardl T, Leiserson C (2020) EvolveGCN: Evolving graph convolutional networks for dynamic graphs. In: Proceedings of the AAAI Conference on Artificial Intelligence 34(4):5363–5370. [Google Scholar] [Crossref]

13. https://doi.org/10.1609/aaai.v34i04.5984 [Google Scholar] [Crossref]

14. Pourhabibi T, Ong KL, Kam BH, Boo YL (2020) Fraud detection: A systematic literature review of graph-based anomaly detection approaches. Decision Support Systems 133:113303. [Google Scholar] [Crossref]

15. https://doi.org/10.1016/j.dss.2020.113303 [Google Scholar] [Crossref]

16. Pang G, Shen C, Cao L, Van Den Hengel A (2021) Deep learning for anomaly detection: A review. ACM Computing Surveys 54(2) 38. https://doi.org/10.1145/3439950 [Google Scholar] [Crossref]

17. Ranshous S, Shen S, Koutra D, Harenberg S, Faloutsos C, Samatova NF (2015) Anomaly detection in dynamic networks: A survey. WIREs Computational Statistics 7(3):223–247. [Google Scholar] [Crossref]

18. https://doi.org/10.1002/wics.1347 [Google Scholar] [Crossref]

19. Rossi E, Chamberlain B, Frasca F, Eynard D, Monti F, Bronstein M (2020) Temporal graph networks for deep learning on dynamic graphs. arXiv preprint arXiv:2006.10637. [Google Scholar] [Crossref]

20. https://doi.org/10.48550/arXiv.2006.10637 [Google Scholar] [Crossref]

21. Veličković P, Cucurull G, Casanova A, Romero A, Liò P, Bengio Y (2018) Graph attention networks. In: International Conference on Learning Representations. https://arxiv.org/abs/1710.10903 [Google Scholar] [Crossref]

22. Wu Z, Pan S, Chen F, Long G, Zhang C, Yu PS (2021) A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems 32(1):4–24. [Google Scholar] [Crossref]

23. https://doi.org/10.1109/TNNLS.2020.2978386 [Google Scholar] [Crossref]

24. Xu B, Shen H, Sun B, An R, Cao Q, Cheng X (2021) Towards consumer loan fraud detection: Graph neural networks with role-constrained conditional random field. Proceedings of the AAAI Conference on Artificial Intelligence 35(5). https://doi.org/10.1609/aaai.v35i5.16582 [Google Scholar] [Crossref]

25. Zhang Z, Cui P, Zhu W (2022) Deep learning on graphs: A survey. IEEE Transactions on Knowledge and Data Engineering 34(1):249–270. https://doi.org/10.1109/TKDE.2020.2981333 [Google Scholar] [Crossref]