van Deenen SupportProjects→ Data-Engineer Streaming at
strm bb-sim

Data-Engineer Streaming at

Passing up to 60000 measurements per second through Kafka and Flink into Bigquery
Role: data-engineer
Client: from September 2019 until Februari 2020 is the largest Dutch online retailer. I joined the 'Measurements' team that measures all aspects of performance and behavior on the website and the app, using a modern Kafka + Flink + BigQuery stack with Avro encoded messages.

The challenge in this team/project is the fast growth of the volume of streaming data, while at the same time being able to evolve the system capabilities. Earyl 2020 our stack processed several Terabytes per day.

We also designed and built an implementation for the GDPR "Do not Track" and "Right to be Forgotten" requirements, both on the streaming measurements, and the 'at rest' data in BigQuery.

Categorized skills