Waiting for code

on waitingforcode.com

Closures in Scala

If you're reading this blog, you've certainly noticed its big interest for Apache Spark. One of first problems we encounter with this data processing framework is a "Task not serializable" error that is caused by a not serializable closure. In this post, outside of Spark's context, we'll focus on these specific functions. Continue Reading →

Stream-to-stream state management

Last weeks we've discovered 2 stream-to-stream join types in Apache Spark Structured Streaming. As told in these posts, state management logic may be sometimes omitted (for inner joins) but generally it's advised to reduce the memory pressure. Apache Spark proposes 3 different state management strategies that will be detailed in the following sections. Continue Reading →

Lazy operator in Scala

Scala's lazy instances generation can be helpful in a lot of places. It simplifies writing since we can declare an instance at right and common place and delay its physical creation up to its first use. In Java we've this possibility too, though, it's much more verbose than in Scala. Continue Reading →

Scala and for loop

For loop was maybe the most used iterative structure in Java before the introduction of lambda expressions. With this loop we're able to write everything - starting with a simple mapping and terminating with a more complex "find-first element in the collection" feature. In Scala we can made these operations with monads and despite that, for loop is a feature offering a lot of possibilities. Continue Reading →

Scala and pattern matching

At first glance Scala's pattern matching looks similar to Java's switch statement. But it's only the first impression because after analyzing the differences we end up with some smarter idea. Continue Reading →