Delta Lake tips

How to check data regression with Delta Lake and SQL?

You just changed the value of one column in your processing logic and want to check whether there wasn't other impact. You can do that by launching the query like this: SELECT x.id, COUNT(*) FROM (...

Continue Reading โ†’

How to create a data validation query?

The goal is to write a query that for each table will compare the content and return any rows that are missing in one of the compared tables. The query combines FULL OUTER JOIN, time travel, and ro...

Continue Reading โ†’