Scala Stream offers something we have not a habit to see in other languages - lazy computation of the values alongside the memoization. However it's sometimes misleading and some people think about Streams as about iterators, i.e. a data structure computing and forgetting about the results. Such thinking can often lead to memory problems, especially with infinite streams. Continue Reading →
One of important characteristics of distributed graph processing which makes it different from classical Map/Reduce approach is the iterative nature of many algorithms. Pregel is one of the computation models that supports such kind of processing very well, while Apache Spark GraphX comes with its own Pregel implementation. Continue Reading →
The series about the features introduced in Apache Spark 2.4.0 continues. Today's post will cover higher-order functions that you may know from elsewhere. Continue Reading →
During the years Java was much more verbose than its JVM-based colleagues (Clojure, Groovy, Scala, ...). But it changed with Java 8 and its Lambda expressions. However the years of the verbosity brought some patterns into Java-based applications. One of them is known as SAM and fortunately it can be easily used with Scala. Continue Reading →
Until now we've been working only with in-memory graphs. However, Apache Spark GraphX provides a much more convenient and prod-ready methods to load and save them. And this post will try to show them. Continue Reading →
The series about Apache Spark 2.4.0 features continues. After last week's discovery of bucket pruning, it's time to switch to Structured Streaming module and see its major evolution. Continue Reading →
Some people coming from Java are often confused about enumerations. In Java, they're often used not only to enumerate things as in dictionaries but also to create singletons. In Scala, the latter use case is much more reserved to objects and apparently, only the former one remains. We'll see through this post whether it's true or not. Continue Reading →
Previously we've learned about the vertices and edges representations in Apache Spark GraphX. At this moment to not introduce too many new concepts at once, we deliberately omitted the discovery of edges partitioning. Luckily, a new week comes and it lets us discuss that. Continue Reading →
This post begins a new series dedicated to Apache Spark 2.4.0 features. The first covered topic will be bucket pruning. Continue Reading →
Even though the regular expressions look similarly in a lot of languages, each of them brings some own constructs. Scala is not an exception for this rule and we'll try to see it in this post. Continue Reading →
Schemas are one of the key parts of Apache Spark SQL and its distinction point with old RDD-based API. When we deal with data coming from a structured data source as a relational database or schema-based file formats, we can let the framework to resolve the schema for us. But the things complicate when we're working with semi-structured data as JSON and we must define the schema by hand. Continue Reading →
After last week's discovery of VertexRDD we have still one graph-composing item to explain - EdgeRDD. After all, the graph is about the relationships this RDD guarantees the links between vertices. Continue Reading →
Nowadays a lot of SDKs and libraries are written with Java. Fortunately dealing with Java code in Scala is possible and even in case of collections. Continue Reading →
The code generated by Apache Spark for all the queries defined with higher-level concepts as SQL queries is the key to understand the processing logic performance. This post, started after a discussion on my Github, tries to explain some of the basics of code generation workflow. Continue Reading →
After last week's global overview of graph representation in GraphX module, it's time to go a little bit deeper and analyze the 2 main components of graphs: vertices and edges. We'll begin here with the former ones. Continue Reading →