A 3-day bug hunt on a 3-person team costs up to β¬7,200 in lost engineering time. This workshop teaches you to prevent that β unit tests, data tests, and integration tests for PySpark and Databricks Lakeflow, including Spark Declarative Pipelines.
Maybe you didn't like my clickbait title but wanted to test my creativity with it ;) And more seriously, in this post I will cover streams but the ones that you're using in your Scala/Java code rather than the distributed ones provided by Apache Kafka. And I decided to write that because by analyzing a lot of Spark I/O method, I meet streams everywhere and I wanted to shed some light on them.
That's the next post I wrote after my unsuccessful analysis of Apache Kafka source. When I was reading the part responsible for sending requests to the broker, I found that it was partially managed by a Java package that I've never seen before. And that will be the topic of this post.
Java introduces more and more event-driven features. One of them is WatchService which allows to handle files-related events.
