Data-Engineer Streaming at Bol.com
Passing up to 60000 measurements per second through Kafka and Flink into Bigquery
from September 2019
until Februari 2020
Bol.com is the largest Dutch online retailer. I joined the 'Measurements' team that measures all aspects of performance and behavior on the website and the app, using a modern Kafka + Flink + BigQuery stack with Avro encoded messages.
The challenge in this team/project is the fast growth of the volume of streaming data, while at the same time being able to evolve the system capabilities. Earyl 2020 our stack processed several Terabytes per day.
We also designed and built an implementation for the GDPR "Do not Track" and "Right to be Forgotten" requirements, both on the streaming measurements, and the 'at rest' data in BigQuery.