Member-only story
Databricks-SDK for Python
I needed to run Databricks CLI in a workspace via a notebook. But this is what I got:
It seems like that Databricks does not recognize the new version of the CLI. After researching, I found out that the new version of Databricks CLI is implemented on GO language, and is not distributed using pip. I had two options to:
- Using Databricks asset bundles to capture the CLI results, or
- Using Python Databricks-SDK that has built in authentication.
I went with Python Databricks-SDK. This post shows how to do it.
Databricks-SDK for Python is at https://pypi.org/project/databricks-sdk/ at PyPI. To use it in a workspace in a notebook, you can pip install it in the notebook, but it has limitations. The best is to install it on a cluster.
To install it on a cluster, the best approach is to download the package and store in Unity Catalog Volumes (or in DBFS if your workspace is not enabled for UC). Go to Downloads page https://pypi.org/project/databricks-sdk/#files and download the Build Distribution (wheels) in your local machine. Upload it to Unity Catalog Volumes in your Databricks workspace.
