Apache Spark SQL articles

Catalyst Optimizer in Spark SQL

The use of Dataset abstraction is not a single difference between structured and unstructured data processing in Spark. Apart of that, Spark SQL uses a technique helping to get results faster.

Continue Reading →

Dataset in Spark SQL

Spark 2.0 brought some changes at API level. One of them was the merge of DataFrame with Dataset. Thanks to that the 3rd data abstraction, present yet in 1.6, was finally removed.

Continue Reading →