Iceberg and Nessie Make Data Changes Safer
Git gave application teams a way to try something without breaking main. Data teams waited a long time for the equivalent.
I've been running Iceberg with Nessie as the catalog for a while now. The thing that sold me wasn't time travel or the file format. It was being able to say try it on a branch and actually mean it.
Branches are the whole point
Schema change, backfill, a partition scheme you're not sure about: do it on a branch, point a test job at that branch, look at what comes out. Nobody downstream sees a thing until you merge. The alternative used to be a full copy of the table, or a very careful WHERE clause and a night of sleep you weren't going to get.
-- Spark SQL with the Nessie extensions enabled.
CREATE BRANCH IF NOT EXISTS backfill_orders IN nessie FROM main;
USE REFERENCE backfill_orders IN nessie;
-- Run the backfill here. Nobody reading main sees any of it.
INSERT OVERWRITE nessie.sales.orders
SELECT * FROM nessie.staging.orders_backfill;
-- Compare, check the downstream reads, then promote.
MERGE BRANCH backfill_orders INTO main IN nessie;
-- When someone says the numbers changed:
SELECT snapshot_id, committed_at, operation
FROM nessie.sales.orders.snapshots
ORDER BY committed_at DESC;Snapshots turn arguments into diffs
When someone says the numbers changed, you can compare snapshot 4 against snapshot 5 instead of reconstructing what the pipeline did last Tuesday. Querying db.table.snapshots takes two seconds and has saved me several long conversations.
Promotion should be unremarkable
Merge the branch, check the downstream reads still work, keep the previous snapshot id somewhere you can find it in a hurry. If a release needs a runbook and two people watching a dashboard, the branch didn't do its job.
It pays off where things change often
Worth the setup when contracts are still moving, tables keep evolving, or new consumers show up every month. On a table that has had the same six columns for three years it's mostly overhead. Not everything needs a versioned catalog, and pretending otherwise is how platforms get heavy.