Real-time big data analytics: tools and architectural solutions

Abstract

This article outlines the development and testing of a real-time big-data analytics architecture that uses state-of-the-art stream-processing tools, designed for high-load and crisis-prone operational environments typical of current operations in Ukraine. The motivation for this research stems from the rapid growth of streaming data across GovTech, energy systems, telecommunications, distance learning systems, and cybersecurity, which necessitates timely analysis for effective operational decision-making. An applied experimental methodology using both synthetic and publicly available streaming datasets was developed to simulate realistic operational scenarios in Ukraine and evaluate the performance of the proposed system. An integrated Layered Architecture based on Apache Kafka, Apache Flink, Apache Spark Streaming, and ClickHouse was developed to enable continuous ingestion, processing, and analytical storage of data. The performance of the proposed system was evaluated under controlled workload conditions in a containerized environment. Experimental results demonstrated that the proposed solution achieved end-to-end processing latency of 120-310 ms under Medium to High Data Load conditions (up to 150,000 events/sec), while sustaining throughput of over 140,000 events/sec with super-linear scalability. Additionally, the results indicated that the proposed system's average latency decreased by approximately 69% relative to a baseline batch-oriented pipeline, and the average recovery time following a node failure decreased by approximately 47%. Overall, these results confirm that the proposed architecture can provide stable, efficient real-time analytics under high uncertainty and intense workloads. From a practical standpoint, the proposed solution can be directly adopted by Ukrainian public and private organizations to support operational monitoring, incident response, and data-driven decision-making in critical infrastructure.