Topic
Table formats in practice: schema evolution, slowly changing dimensions, snapshot expiry, compaction cadence, and repairable history.
I have worked with Apache Iceberg since 2022. The table format matters because it shapes how safely a team can evolve schemas, correct history, compact small files, and expire snapshots. These notes focus on the portable decisions: slowly changing dimension patterns, maintenance cadence, metadata growth, and the recovery path you need before a routine correction turns into an expensive rewrite.
Build Trino 483 on Iceberg with Apache Polaris as the REST catalog, then prove the vended S3 credentials cannot read the table next door.
~45 min to a working stack; the credential-vending section is the part that takes the time
Build a medallion pipeline with dbt-trino on Iceberg, orchestrate it with Airflow 3, and find the partition spec dbt accepts and discards without a warning.
~40 min, most of it waiting on two container builds
Measure what compaction buys on a small-files partition, then delete 909,968 rows and get them back — and find where the safety net stops.
~25 min, and the rollback itself takes under a second
Attach StarRocks to the same Polaris REST catalog Trino writes through, measure both on identical data, and check for standing S3 keys.
~20 min, mostly waiting for the StarRocks backend to register
Scrape to bronze, silver, gold on Spark 4 and Delta, embed into Chroma, and serve answers that cite their sources or decline to answer.
~1 h, most of it pulling images and the first Ivy resolution