Topic
Databricks measured on a real workspace: what COPY INTO, Auto Loader and MERGE do with files that change, and how Delta re-runs and restores a day.
Most Databricks material shows a feature working once. These notes run the features against the things that happen to real data: files sent twice, uploads cut short and corrected, files that arrive late, jobs re-run by hand. Every result is a count against data whose correct answer is known, and the documented behaviour that explains it.
Nulling one person's e-mail rewrote all 31.85 million rows of an event table that copied it into every row. Kept in one user table, the same erasure rewrote 60,000.
6 min read
A client's daily build and reports took 38 seconds on Databricks' smallest SQL warehouse. The billing table charged 11 minutes: 95% of it was waiting to stop.
6 min read
A user's rows deleted from a Delta table on Databricks stayed readable by time travel after DELETE, after a purge and after VACUUM, until the retention passed.
6 min read
Ten days of files loaded six ways on Databricks. A truncated upload, fixed under the same name, was skipped by COPY INTO and Auto Loader: 1,638 events lost.
8 min read