The Real Cost of
Untested Data Pipelines.

With the foundations in place, we turn to the first line of defense: unit tests. They remain essential for catching logic errors early and fast. We cover what makes a unit test genuinely useful versus one that gives false confidence, the code patterns and libraries that make writing them less painful, and how to handle the unique challenges of testing PySpark and Databricks workloads. We close by embedding unit tests into the development lifecycle and clearing up the misconceptions that lead teams to either over-rely on them or abandon them too soon

• Why do they still matter as the first guard?
• Golden rules of useful and reliable unit tests
• Useful code constructs for efficient unit testing
• Life-saving libraries
• Unit tests for PySpark and Databricks
• Automation in development lifecycle
• Misconceptions and gotchas
• Hands-on lab adding tests to an existing PySpark code base
• Q&A live discussion on what we learned

Moving beyond unit tests, we explore how data tests operate at a different level — validating that the pieces work together against real data and real systems. We build a data quality layer that turns passive observations into active test controls, and rethink assertions in this context. The section closes by wiring everything into the CI/CD process so data tests become a natural checkpoint in every deployment.

• Difference with Unit Tests and Integration Tests
• Building data quality layer for tests
• Transforming data quality observations into actionable test controls
• Assertions differently
• Integrating with the CI/CD process

Here we tackle integration tests in practice, focusing on the specific challenges that come with Databricks environments. We look at how to keep maintenance overhead from becoming a burden as the test suite grows, and how to automate execution so integration tests run reliably without manual intervention.

• Testing Databricks-specific features
• Mitigating Integration Test maintenance overhead
• Automating the execution

We apply the testing strategies to the Medallion architecture, following data as it moves through the Bronze, Silver, and Gold layers. Each layer introduces its own failure modes, so we map out where problems are most likely to originate and how to trace them back to their source. We then address the most common issues that surface at each layer, showing how unit tests, data tests, and integration tests each play a role in keeping the pipeline healthy end to end.

• The Bronze → Silver → Gold principle
• Identifying where the problems can came from
• Addressing most common issues with Unit Tests, Data Tests, and Integration Tests

Spark Declarative Pipelines come with a common misconception: no SparkSession must mean difficult to test. We unpack how the main SDP script works like a Python __main__ function — it declares the workflow while the real logic lives elsewhere, and that separation is actually your testing advantage. Any business logic extracted from the pipeline can be covered with the unit tests from the previous section. We walk through concrete test examples and finish by integrating SDP tests into the CI/CD pipeline.

• Integrating tests best practices to the declarative world
• Learning how to approach SDP pipelines to make them testable
• Testing incremental logic across Bronze, Silver, and Gold layers
• Keeping LSDP capabilities for testable workloads

Before diving into the capstone project, we take a step back to look at one of the most exciting shifts happening in the data engineering space right now: using LLMs as an active partner in building and maintaining your testing layer. We explore how LLMs can generate unit test cases from your pipeline code, produce realistic synthetic datasets, translate business requirements into data quality rules, and help you spot logic paths your current test suite is missing.

We close the workshop with a capstone project that brings everything together. Starting from a realistic Databricks data pipeline, participants will apply the full testing stack hands-on: writing unit tests to guard business logic, data tests to enforce quality at each layer of the Medallion architecture, and integration tests to validate the pipeline end to end. The project is designed to reflect the real challenges data engineers face — non-determinism, external state, PySpark specifics and challenges participants to make deliberate choices about what to test, at which level, and how to wire it all into a CI/CD pipeline. By the end, the testing strategies covered throughout the workshop stop being abstract concepts and become a working, cohesive test suite

We close the day with an open retrospective and Q&A. This is a space to surface lingering doubts, challenge the approaches presented, and share lessons from the capstone project. No slides, no structure — just an honest conversation about what works, what doesn't.