Apache Spark 2.4.0 features - array and higher-order functions
The series about the features introduced in Apache Spark 2.4.0 continues. Today's post will cover higher-order functions that you may know from elsewhere. Continue Reading →
Java, SAM and Scala
During the years Java was much more verbose than its JVM-based colleagues (Clojure, Groovy, Scala, ...). But it changed with Java 8 and its Lambda expressions. However the years of the verbosity brought some patterns into Java-based applications. One of them is known as SAM and fortunately it can be easily used with Scala. Continue Reading →
Loading and saving graphs in Apache Spark GraphX
Until now we've been working only with in-memory graphs. However, Apache Spark GraphX provides a much more convenient and prod-ready methods to load and save them. And this post will try to show them. Continue Reading →
Apache Spark 2.4.0 features - watermark configuration
The series about Apache Spark 2.4.0 features continues. After last week's discovery of bucket pruning, it's time to switch to Structured Streaming module and see its major evolution. Continue Reading →
Enums in Scala
Some people coming from Java are often confused about enumerations. In Java, they're often used not only to enumerate things as in dictionaries but also to create singletons. In Scala, the latter use case is much more reserved to objects and apparently, only the former one remains. We'll see through this post whether it's true or not. Continue Reading →
Edge partitioning strategies
Previously we've learned about the vertices and edges representations in Apache Spark GraphX. At this moment to not introduce too many new concepts at once, we deliberately omitted the discovery of edges partitioning. Luckily, a new week comes and it lets us discuss that. Continue Reading →
Apache Spark 2.4.0 features - bucket pruning
This post begins a new series dedicated to Apache Spark 2.4.0 features. The first covered topic will be bucket pruning. Continue Reading →
Regular expressions in Scala
Even though the regular expressions look similarly in a lot of languages, each of them brings some own constructs. Scala is not an exception for this rule and we'll try to see it in this post. Continue Reading →
Defining schemas in Apache Spark SQL with builder design pattern
Schemas are one of the key parts of Apache Spark SQL and its distinction point with old RDD-based API. When we deal with data coming from a structured data source as a relational database or schema-based file formats, we can let the framework to resolve the schema for us. But the things complicate when we're working with semi-structured data as JSON and we must define the schema by hand. Continue Reading →
Edge representation in Apache Spark GraphX
After last week's discovery of VertexRDD we have still one graph-composing item to explain - EdgeRDD. After all, the graph is about the relationships this RDD guarantees the links between vertices. Continue Reading →
Collections conversion in Scala
Nowadays a lot of SDKs and libraries are written with Java. Fortunately dealing with Java code in Scala is possible and even in case of collections. Continue Reading →
The who, when, how and what of Apache Spark SQL code generation
The code generated by Apache Spark for all the queries defined with higher-level concepts as SQL queries is the key to understand the processing logic performance. This post, started after a discussion on my Github, tries to explain some of the basics of code generation workflow. Continue Reading →
Vertex representation in Apache Spark GraphX
After last week's global overview of graph representation in GraphX module, it's time to go a little bit deeper and analyze the 2 main components of graphs: vertices and edges. We'll begin here with the former ones. Continue Reading →
String interpolators in Scala
I've always found concatenating Strings with "+" a laborious task. Hence when I've discovered Scala and its text construction methods I was immediately convinced. And as you can imagine, this topic merits its own short post in the Scala category. Continue Reading →
DataFrame and file bigger than available memory
Some weeks ago I've written a post about big files impact on RDD-based processing. The post revealed the difficulty to process files bigger than available memory. But as you'll see below, it's not completely true. Continue Reading →