van Deenen SupportProjects→ KPN Datalab Data Streaming Engine
bb-sim tdssim

KPN Datalab Data Streaming Engine

Design and Development of a system for real-time high volume event stream processing
Role: Architect/Lead developer
Client: KPN Technium from 2016 until 2018
I worked at the KPN 'Datascience Lab' in a 6 person innovation team. We developed new techniques and methodologies for realtime event processing and analytics on streaming data. An initial pilot showed live "Customer Journeys". The team only developed stream processing; batch processing and ETL were not part of its mandate.

We designed the so-called Realtime Analytics Engine ("RAE"), a horizontally scalable and fault-tolerant system where it was easy to apply algorithms to live event data. This system was comprised of state-of-the-art components:

Part of my job was writing an overall architecture document that described and justified the design choices.

During the two years I spent on this project I:

A lot of my time was spent in debugging issues on the still immature Kubernetes cluster.

The key design idea of our engine was Dr. Martin Kleppmann's the database inside-out.

References:

Categorized skills