Spark SQL schema articles

on waitingforcode.com
Articles tagged with Spark SQL schema. There are 4 article(s) corresponding to the tag Spark SQL schema. If you don't find what you're looking for, please check related tags: access pattern, Ad-hoc polymorphism, Akka Distributed Data, Akka examples, algorithm analysis, algorithm complexity, Apache Beam configuration, Apache Beam internals, Apache Beam partitioning, Apache Beam PCollection.

Defining schemas in Apache Spark SQL with builder design pattern

Schemas are one of the key parts of Apache Spark SQL and its distinction point with old RDD-based API. When we deal with data coming from a structured data source as a relational database or schema-based file formats, we can let the framework to resolve the schema for us. But the things complicate when we're working with semi-structured data as JSON and we must define the schema by hand. Continue Reading →

Schema projection

Even if it's always better to explicit things, in programming we have often the possibility to let the computer to guess. Spark SQL also has this level of intelligence, for example during schema resolving. Continue Reading →

Schemas

Spark SQL - even if the SQL suffix makes automatically think about RDBMS - works well with other data sources, as even plain CSVs or JSON files. This relation would be difficult to achieve without the concept of schema. Continue Reading →