Bronze to Silver Is Where Trust Starts

Bronze is evidence

Keep it exactly as it arrived, encoding problems and all. The first time someone asks whether a number was wrong in March or only wrong in the report, you'll want the original file, not your cleaned-up version of it. Storage is cheap. Regret isn't.

Silver is where the arguments end

This is where cust_id, customer_id and CustomerID become one column with one type. Where you decide, once, that a null ship date means not shipped rather than unknown. Skip that decision and it gets made again in every downstream query, differently each time.

Most of the duplicated logic I've found in warehouses traces back to a silver layer that left something ambiguous. Someone needed an answer on a Friday, made a local call, and shipped it.

Put the checks where the change happens

I'd rather a run fail on a row count than have a dashboard sit quietly four percent off for a week. Assert at the transform, not after a model has already consumed the output. And keep the table plain enough that another engineer can open it and work out what belongs there without booking time with you.

# Inside the bronze -> silver job, before anything is written.
bronze_rows = bronze_df.count()
silver_rows = silver_df.count()
dropped = bronze_rows - silver_rows

if dropped > bronze_rows * 0.01:
    raise ValueError(
        f"orders_silver would drop {dropped} of {bronze_rows} rows (limit is 1%). "
        "Look at the dedup keys before rerunning."
    )

silver_df.write.mode("overwrite").saveAsTable("silver.orders")