From Data Chaos to Real-Time Intelligence: Data Engineering Challenges & Enterprise Streaming

Abstract:

As global data volumes surge past hundreds of zettabytes per year, real-time streaming has become imperative for modern enterprises. While streaming engines form the backbone of event-driven architectures, deploying them at enterprise scale exposes critical challenges that go far beyond the engine itself, from schema evolution and data governance to multi-tenancy, integration, and multi-cluster high-availability. Left unaddressed, these gaps translate into silent outages, compliance risks, and platform teams becoming bottlenecks rather than enablers. In this talk, we examine why these problems arise, why handling them is critical, and how they can be solved, using Axual as a concrete example of an enterprise streaming platform, with real-world results from banking and energy, including billion-message-per-day pipelines and sub-second payment latency.

Short bio of speaker:

Pantelis Ypsilantis is a Data Platform Engineer at Axual in Utrecht, Netherlands, where he works on enterprise streaming solutions built around Apache Kafka and Kubernetes. He holds an MSc in Data & Web Science and a BSc, both from the School of Informatics at the Aristotle University of Thessaloniki, with an MSc thesis on a storage fabric for autonomous collaborative edge devices published in IC2E 2024. Before joining Axual, he spent over five years as a Software Engineer at Dataviva, architecting backend systems for large-scale multidimensional data, managing multi-terabyte databases, and operating streaming and observability stacks across multiple client environments, while also contributing to mission-critical applications for the Hellenic Army. His interests include distributed systems, real-time data streaming, and cloud-native architectures.