Adaptive Query Execution in Apache Spark articles

What's new in Apache Spark 3.0 - shuffle partitions coalesce

In my previous blog post you could learn about the Adaptive Query Execution improvement added to Apache Spark 3.0. At that moment, you learned only about the general execution flow for the adaptive queries. Today it's time to see one of possible optimizations that can happen at this moment, the shuffle partition coalesce.

Continue Reading →

What's new in Apache Spark 3.0 - Adaptive Query Execution

A query adapting to the data characteristics discovered one-by-one at runtime? Yes, in Apache Spark 3.0 it's possible thanks to the Adaptive Query Execution!

Continue Reading →