A 3-day bug hunt on a 3-person team costs up to β¬7,200 in lost engineering time. This workshop teaches you to prevent that β unit tests, data tests, and integration tests for PySpark and Databricks Lakeflow, including Spark Declarative Pipelines.
You have set up your Databricks platform and are about adding a new epic to your backlog, "Implement a data orchestration layer". Before you hit the add button, let's stop for a while and see if you really need this dedicated layer.
The Medallion architecture is a well known data organization pattern on Databricks. Some people use the recommended Bronze/Silver/Gold names, others rely on more classical raw/cleansed/mart nomenclature, but all organize the datasets the same way, around three storage layers. In this organization the raw data is ingested without any transformation before it's cleansed and exposed as a valuable asset from the last layer. But despite this simplicity, there is a place where you - as a consultant - can use your favorite sentence, "it depends".
Probably two of most frequently used assets in Declarative Automation Bundles on Databricks are wheel-based jobs and notebooks. But do you know the Lakeflow Jobs component can also run regular Python scripts that don't have to be packaged as a wheel.
Surprisingly it may sound, keeping code formatted correctly is not only the matter of Python. If you are more of a SQL user, you can also implement additional linting controls with a library called SQLFluff.
Controlling the code style has never been more important. Your coding agent may generate the code but you'll be the reviewer and the owner. Without consistent and clever coding rules your understanding will be difficult. One of ways to help you in that task are linters and code formatters, or simply Ruff that does both things!
We all know more or less how to manage code for data processing jobs, either in Python, Scala, or Java. But oftentimes those jobs operate on other resources such as Delta Lake tables, or row files, that may not follow the same management rules.
Query optimizers are impressive. They profile your data, estimate costs, and pick execution strategies faster than any human could. But they don't know everything and when they guess wrong, the consequences show up in your bill and your SLAs. Hints are the escape hatch. A small instruction you embed directly in SQL that tells the engine: I know something you don't
In March Databricks engineers have shared some tasty details about incremental refreshes on their paper about Enzyme. Since I have been actively exploring Lakeflow Spark Declarative Pipelines recently, I couldn't miss this opportunity to better understand what's behind!
It's the 3rd blog post of the series dedicated to Lakeflow Spark Declarative Pipelines. Today we're going to see a topic that amazed me when I first came to the data engineering and data warehousing world, Slowly Changing Dimensions (SCD).
Welcome to the second blog post on Lakeflow Spark Declarative Pipelines. Today we are going beyond the environment to see how to declare the processing jobs.
Even though I've wrapped up my exploration of Spark Declarative Pipelines, there is still one topic on my mind. How does "vanilla" SDP relate to the Databricks version, known as Lakeflow Spark Declarative Pipelines? I'll try to answer that today and, hopefully, share some interesting insights with you.
Enforcing team conventions is never an easy task. In my experience, I've moved from clearly defined collaboration conventions on a wiki page to code review checklists, but neither has made my life easier. The only tools that have truly helped are the Git hooks I've been using for over a decade.
Unlike a standard cron schedule or a file-arrival trigger, the continuous trigger transforms your Lakeflow job into an "always-on" service. It ensures that as soon as one run finishes (or fails), the next one kicks off immediately. It's the sweet spot for data engineers who need the reliability of a job but the low latency of a stream. In this post, we'll dive into how continuous triggers work, why they are the secret weapon for Structured Streaming stability, and how to set them up in your next Lakeflow project.
Some time ago I had an unpleasant surprise for a MERGE query that despite the small table to merge, and the liquid clustering enabled on the target table, was taking ages. The solution came from a Photon's feature called the Dynamic File Pruning.
If you are running data processing jobs on Lakeflow Jobs, you certainly noticed these two options that might look the same but in fact have two different purposes. If not, it's even better because I'm now sure the blog post will be useful for you!
Variables are an essential part of any deployment process. You don't want to write a dedicated YAML or Python script for every environment, do you? Databricks Asset Bundles (DAB) is no exception, as its variable handling is designed to significantly simplify your workflow.
Make and Makefiles have been around for a while to facilitate tasks definitions, even in Python. But Python has an alternative that we are going to discover in this blog post.
Since the early days of data lakes, datasets persisted in object stores have not had primary and foreign key constraints enforced. Databricks is no exception; however, the platform supports unenforced PRIMARY KEY and FOREIGN KEY constraints, which the query optimizer uses to improve performance.
Even though you mostly find ANSI-supported SQL features on Databricks, there are some useful Databricks-specific functions. One of them is the INSERT...REPLACE statement that you can use to overwrite datasets matching given conditions.
You need to write a Lakeflow job that is going to start upon a file upload. Sounds easy, isn't it? But what if the same job also had to support the CRON trigger? Unfortunately, you cannot set multiple triggers on the job, so you will have to engineer the workflow differently.
