The Medallion architecture is a well known data organization pattern on Databricks. Some people use the recommended Bronze/Silver/Gold names, others rely on more classical raw/cleansed/mart nomenclature, but all organize the datasets the same way, around three storage layers. In this organization the raw data is ingested without any transformation before it's cleansed and exposed as a valuable asset from the last layer. But despite this simplicity, there is a place where you - as a consultant - can use your favorite sentence, "it depends".
What would it take for you to trust your Databricks pipelines in production?
A 3-day bug hunt on a 3-person team costs up to €7,200 in lost engineering time. This workshop teaches you to prevent that — unit tests, data tests, and integration tests for PySpark and Databricks Lakeflow, including Spark Declarative Pipelines.
Konieczny
To better understand this "it depends" answer, I prepared a real-world scenario that you might have already experienced. If not, it's certainly waiting for you, in some not yet explored dataset.
File as table
Picture a user or a process who first extracts some data to a file. The file lands to a Databricks Volume and triggers a Lakeflow Job automatically. The job processes the input and overwrites a table on Unity Catalog. The high-level schema below summarizes this workflow:
Before you can transform this schema to to the low-level design, you will probably ask yourself this question: Since my end table always represents the uploaded file, do I need to follow the classical Bronze/Silver/Gold decomposition?. If you answered yes, the final implementation will have some additional layers, as shows the next diagram:
In this solution, the file first lands to a volume. It's later read and ingested to a bronze table where all its columns are correctly typed. The Bronze-to-Silver job takes care of reading the most recent rows (e.g. from the most recent partition) and typing the columns. At this layer the job is also enriching the rows with the data from other tables. The Silver-to-Gold job later ingests the created table as-is to the Gold layer. It's the Medallion architecture implementation by the book. But is it the right approach?
The Bronze/Silver/Gold layer introduces a lot of complexity when it comes to the code base and mental model for our imaginary scenario when the input, after slight transformations, creates the output:
- First, the output table is always a result of the input file. Therefore we are introducing an additional complexity with a way to retrieve the correct data from Bronze to Silver.
- Second, the dataset in the Gold layer is the exact copy of the Silver layer. Not an augmented version of it, not an aggregated vision, just a copy.
- Third, the backfilling. If your data provider changes the schema of the input file, you have three places to change. With AI-assisted coding it's certainly something easy but what if it can be minimized to one place, and probably less tokens burnt?
What if instead of the classical 3-layered architecture, you used here a 2-layered one where your landing zone (Databricks volume) acts as Bronze layer, and the output table as the Gold layer? Your end users will still explore the same output data and the underlying technical details are now simplified. Your data needs to be reprocessed? No problem, you either re-uploads the file, or retriggers the job by explicitly passing the name of the already uploaded files as the file to process in the execution.
Do not consider it as a rule of thumb, though. Switching from 3 to 2 layers should be fine if you are doing full loads and your input dataset is well curated. If not, the classical Medallion organization would probably be a better fit.
The best fits scenarios
And when having a classical Bronze/Silver/Gold division makes sense?
- Incremental loads. It's easier to reason about them with an intermediary cleansed layer (Silver) and a raw landing zone (Bronze) where the rows are written as-is. In this configuration you will consider the Silver as an improved copy of the Bronze, with data typing and data quality issues fixed. Can it be done without Silver? Probably yes but the advantage of the additional layers is flexibility. Changing the rule and reprocessing Silver data should be enough in that case most of the time.
- Short raw data retention. By loading everything as simple as possible to the Bronze layer you inherently extend the retention period. A typical example here will be a streaming data source that would store records for less time than a regular Delta Lake table.
- Complex data fetching logic. If your data retrieval logic is complex or inefficient due to the input limitations, having the Bronze layer will be a handy long-term strategy. Reprocessings and ad-hoc data exploration should be then easier.
- Audit requirements. Although it's technically possible to query files or even an Apache Kafka topic, it is undoubtedly easier to run a SQL query against a Bronze, Silver, or Gold table. If your use case involves some strong audit requirements, such as the ability to track all rows that have transited through the tables in the past 6 months, the three-layered architecture can be a better fit here.
- Frequent business changes in the Gold layer. If the lower layers remain pretty stable but your Gold layer is changing quite often, having the three layers should simplify reprocessings. Since the curated data doesn't change but only its aggregated part does, three layers avoid performing some unnecessary calculations over and over again.
- Slowly Changing Dimensions. If you need to store it as part of the Silver layer and expose some denormalized version in the Gold layer, you have no other choice than keeping the three-layered architecture.
Most of the time the three-layered Medallion is always the best choice. However, in some simpler cases, like the file-as-table presented in the blog post, reducing the number of layers and simplifying the overall architecture, can be beneficial - even if it breaks the Bronze/Silver/Gold paradigm.
Data Engineering Design Patterns
Looking for a book that defines and solves most common data engineering problems? I wrote
one on that topic! You can read it online
on the O'Reilly platform,
or get a print copy on Amazon.
I also help solve your data engineering problems contact@waitingforcode.com đź“©