Databricks tips

How to get columns of one or multiple tables?

To get the columns for one or multiple tables, you can combine SHOW TABLES and DESCRIBE EXTENDED commands. The SHOW TABLEs command lists all tables within a schema while the DESCRIBE EXTENDED command ...

Continue Reading β†’

How to define permissions for the jobs in Databricks Asset Bundles?

You can define the permissions with the...permissions block, as follows: # source: https://docs.databricks.com/aws/en/dev-tools/bundles/permissions#define-specific-resource-permissions # ... ...

Continue Reading β†’

How to define a single-node cluster in Databricks Asset Bundles?

The definition for a single-node cluster can look like that: new_cluster: spark_version: 15.4.x-scala2.12 node_type_id: i4i.large autotermination_minutes: 5 runtime_engine: STANDARD...

Continue Reading β†’

How to disable triggers on dev environment with Databricks Asset Bundles?

If you use Databricks Asset Bundles (DAB) in development mode (mode: development), the deployed workflows will be prefixed by the environment name and your users, like in the next picture: Howeve...

Continue Reading β†’

Databricks Asset Bundles and lack of files synchronization after workspace files removal

You may encounter a files synchronization issue after manually deleting the files from your Databricks Asset Bundle's workspace (e.g. /Workspace/Users/your_user/.bundle/your_project/dev/files). The is...

Continue Reading β†’