Exactly-once held through nine failures. The window counts did not.
Kafka, Flink, ClickHouse and Iceberg stayed exact through nine failures, yet 10-second window counts fell to 59.8% of on-time events. Why, and the fixes.
13 min read
Topic
Kafka & streaming
Ingestion that survives replay, late events, duplicate deliveries and producers you do not control. Kafka, Kinesis, Structured Streaming, CDC.
Streaming systems have to assume duplicates, late events, partial producer upgrades, and replays. I have built and operated ingestion and change-data-capture paths across several technologies, but the transferable lessons live above the brand names: idempotent writes, explicit event-time policy, backfills that do not double-count, schema compatibility, and a repair path that works while new traffic is still arriving.
2 posts
All writingTen seconds per idle shard: why I maintain a Kinesis connector for Spark 4
The awslabs Kinesis connector for Spark 4 waits on every idle shard. Why I ported the synchronous connector to Spark 4.2 and AWS SDK v2, and maintain it in public.
14 min read