Some weeks ago I've written a post about big files impact on RDD-based processing. The post revealed the difficulty to process files bigger than available memory. But as you'll see below, it's not completely true. Continue Reading →
Bad things happen in distributed data processing and if we're prepared for them, it's better. To prevent against such issues Apache Spark is able to recompute failed partition but also to store the computation snapshot as a checkpoint. Both properties apply to GraphX module's fault-tolerance mechanism. Continue Reading →
Each programming language has its own specific standards. Scala is not an exception and also comes with its own coding style specificity. Continue Reading →
With data-intensive applications as the streaming ones, bad memory management can add long pauses for GC. Luckily, we can reduce this impact by writing memory-optimized code and using the storage outside the heap called off-heap. Continue Reading →
Apache Spark uses a common data abstraction for all its higher level data structures. This implementation rule isn't different for GraphX represented by the sets of specialized versions of RDDs. Continue Reading →
Have you ever wondered why in Scala we can directly reverse a String and in Java we must use a StringBuilder especially for it? If yes, this post provides a little bit more explanation by focusing on Scala's data types equivalents to Java's primitives (+ String) called rich wrappers. Continue Reading →
Some months ago bithw1 posted an interesting question on my Github about multiple SparkSessions sharing the same SparkContext. If you have similar interrogations, feel free to ask - maybe it will give a birth to more detailed post adding some more value to the community. This post, at least, tries to do so by answering the question. Continue Reading →
Every time when we learn a new topic, it's important to start from the basics. We couldn't learn a new language without knowing the order of subject and verbs in a sentence. The same rule applies to Apache Spark's GraphX module that will be covered in this category. But before going into details, we'll focus on its basics. Continue Reading →
With increasing number of computation power, the parallel computing gained the popularity during the last years. Java's concurrent package is one of the proofs for that. But Scala, even though it's able to work with Java's concurrent features, comes also with its own mechanisms. Futures are one of them. Continue Reading →
The series about graph processing continues. Today it's the moment to analyze some major graph processing frameworks and choose the framework that I'll present more in details in incoming posts. Continue Reading →
Big Data enforces denormalized storage. Joins are costly and it's often much more efficient to store all related information in a single row. Such rows with a lot of columns are called wide rows and they'll be explained in the sections below. Continue Reading →
The series about Scala types system continues. After last week's article about path-dependent types, it's time to discover another type-related feature - self-types. Continue Reading →
Some time ago on my Github bithw1 pointed out an interesting behavior of Hive integration on Apache Spark SQL. To not delve too much into details now, I can tell that the behavior was about not respected DataFrame schema. Our quick exchange ended up with an explanation but it also encouraged me to go much more into details to understand the hows and whys. Continue Reading →
Because of its connected nature, graph structure has its own branch in data mining. Thanks to this branch we can get insight into relationships and dependencies between vertices. Continue Reading →
Before I went to Scala I had never imagined that we could do such many things nothing but with a types system. Aside of higher-kinded types or type boundaries that we can easily find in other languages, Scala offers more advanced type features as path-dependent types covered below. Continue Reading →