Apache Spark Structured Streaming joins articles

on waitingforcode.com
Articles tagged with Apache Spark Structured Streaming joins. There are 4 article(s) corresponding to the tag Apache Spark Structured Streaming joins. If you don't find what you're looking for, please check related tags: Apache Spark 2.4.0 features, Apache Spark data sources, AWS EC2, Big Data patterns implemented, Cerberus + PySpark, Change Data Capture, completable future, data locality, data patterns, errors in Scala.

Check out my new course on Data Engineering!

Are you a data scientist who wants to extend his data engineering skills? Or a software engineer who wants to work with Big Data? If not, maybe a BI developer who wants to evolve to engineering position? My course will help you to achieve your goal! Join the class →

Stream-to-stream state management

Last weeks we've discovered 2 stream-to-stream join types in Apache Spark Structured Streaming. As told in these posts, state management logic may be sometimes omitted (for inner joins) but generally it's advised to reduce the memory pressure. Apache Spark proposes 3 different state management strategies that will be detailed in the following sections. Continue Reading →