A 3-day bug hunt on a 3-person team costs up to β¬7,200 in lost engineering time. This workshop teaches you to prevent that β unit tests, data tests, and integration tests for PySpark and Databricks Lakeflow, including Spark Declarative Pipelines.
In one of the homework of my Become a Data Engineer course I'm asking students to normalize a dataset. In the dataset a date time field has different supported formats. When I was analyzing the possible solutions, I found a class I've never met before, the DateTimeFormatterBuilder. And it will be the topic of this post.
So far only Google Guava allowed us to easily process collections. But the arrival of Java 8 brought a serious alternative to this library - streams.
One of big changes in Java 8 regarding to Java 7 was the replacement of permanent generation by metaspace. Even if this change dates from more than year ago, it's worth to dedicate some lines about this new element in memory.
