Topic
Practical Spark work: shuffle diagnosis, executor sizing, partitioning, skew, and the evidence to collect before touching a configuration.
I have worked with Spark since 2020. The useful diagnosis usually starts with the stage graph, skew, spill, partition size, and one stubborn task—not with a copied configuration checklist. These notes focus on what I measure before changing a knob, how I separate a scheduling problem from a data-layout problem, and how to make a performance claim reproducible on data that can be shared safely.
The awslabs Kinesis connector for Spark 4 waits on every idle shard. Why I ported the synchronous connector to Spark 4.2 and AWS SDK v2, and maintain it in public.
14 min read
A Hive Metastore 4 upgrade rehearsed on a clone: zero constraints, a validator that passed anyway, and why the plan became blue/green.
14 min read
Spark 4.2 on hadoop-aws 3.5 read 9,029 records and wrote zero objects, and no task failed. Why the S3A committer was bypassed, and the tests that now catch it.
17 min read