A 3-day bug hunt on a 3-person team costs up to β¬7,200 in lost engineering time. This workshop teaches you to prevent that β unit tests, data tests, and integration tests for PySpark and Databricks Lakeflow, including Spark Declarative Pipelines.
My backlog of data engineering design patterns candidates starts to fill in and when it happens, it's *the* moment to make some space for new ideas. That's why today you won't read about Apache Spark or Databricks but about a data engineer design pattern.
In the Data Engineering Design Patterns book I'm talking about Data Overwrite as one of the idempotency design patterns. With time, I realized I should have split it into two fine-grained patterns: Static Overwriter and Dynamic Data Overwriter. The difference between them? The capacity to replace only a subset of data thanks to the data layout properties, such as partitions or clustering columns.
