Databricks tips

How to get columns of one or multiple tables?

To get the columns for one or multiple tables, you can combine SHOW TABLES and DESCRIBE EXTENDED commands. The SHOW TABLEs command lists all tables within a schema while the DESCRIBE EXTENDED command ...

Continue Reading β†’

How to define permissions for the jobs in Databricks Asset Bundles?

You can define the permissions with the...permissions block, as follows: # source: https://docs.databricks.com/aws/en/dev-tools/bundles/permissions#define-specific-resource-permissions # ... ...

Continue Reading β†’

How to define a single-node cluster in Databricks Asset Bundles?

The definition for a single-node cluster can look like that: new_cluster: spark_version: 15.4.x-scala2.12 node_type_id: i4i.large autotermination_minutes: 5 runtime_engine: STANDARD...

Continue Reading β†’

How to disable triggers on dev environment with Databricks Asset Bundles?

If you use Databricks Asset Bundles (DAB) in development mode (mode: development), the deployed workflows will be prefixed by the environment name and your users, like in the next picture: Howeve...

Continue Reading β†’

Databricks Asset Bundles and lack of files synchronization after workspace files removal

You may encounter a files synchronization issue after manually deleting the files from your Databricks Asset Bundle's workspace (e.g. /Workspace/Users/your_user/.bundle/your_project/dev/files). The is...

Continue Reading β†’

You encounter an 'Invalid cross-device link' error on Databricks?

The error happens when you use Path.rename to move a file between incompatible storage layers, for example from a local file system to a Databricks volume. In that case, Databricks will fail with an e...

Continue Reading β†’

How to debug queries on Databricks for your PowerBI dashboard created with the import mode?

The import mode loads all the data to PowerBI which makes debugging more challenging. To facilitate the debug, you can ask for termporarly switching from the import mode to the direct query mode and m...

Continue Reading β†’