You have a PySpark project you want to run locally, e.g. for unit tests, and on Databricks, e.g. for remote tests from your local machine on real data.
How to define both required Python dependencies, so databricks-connect and pyspark?
One solution would be using optional-dependencies where both libraries are put into different groups. In my snippet I called them local and remote:
[project.optional-dependencies]
remote = [
"databricks-connect==15.4.12",
]
local = [
"pyarrow>=12.0.1",
"delta-spark>=2.0.0",
"pyspark==3.5.0",
"pandas==2.1.4",
]
With uv you can then use the uv sync --extra command to use one or another set of dependencies:
uv sync --extra local uv sync --extra remote
Running one command will uninstall all dependencies specified for another target. There is no need to keep two separate virtual environments.