databricks/databricks-cli

Stars
384
Rank 111,726 (Top 3 %)
Language
Python
License
Other
Created over 7 years ago
Updated about 1 year ago

databricks/databricks-cli

databricks

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

(Legacy) Command Line Interface for Databricks

databricks-cli

The Databricks Command Line Interface (CLI) is an open source tool which provides an easy to use interface to the Databricks platform. The CLI is built on top of the Databricks REST APIs.

Note: This CLI is under active development and is released as an experimental client. This means that interfaces are still subject to change.

If you're interested in contributing to the project please reach out. In addition, please leave bug reports as issues on our GitHub project.

Requirements

Python Version > 2.7.9 or > 3.6

Installation

To install simply run pip install --upgrade databricks-cli

Then set up authentication using username/password or authentication token. Credentials are stored at ~/.databrickscfg.

databricks configure (enter hostname/username/password at prompt)
databricks configure --token (enter hostname/auth-token at prompt)

Multiple connection profiles are also supported with databricks configure --profile <profile> [--token]. The connection profile can be used as such: databricks workspace ls --profile <profile>.

To test that your authentication information is working, try a quick test like databricks workspace ls.

Known Issues

AttributeError: 'module' object has no attribute 'PROTOCOL_TLSv1_2'

The Databricks web service requires clients speak TLSV1.2. The built in version of Python for MacOS does not have this version of TLS built in.

To use the Databricks CLI you must install a version of Python that has ssl.PROTOCOL_TLSv1_2. For MacOS, the easiest way may be to install Python with Homebrew.

Using Docker

# build image
docker build -t databricks-cli .

# run container
docker run -it databricks-cli

# run command in docker
docker run -it databricks-cli fs --help

Documentation

For the latest CLI documentation, see

learning-spark

Example code from Learning Spark book

koalas

Koalas: pandas API on Apache Spark

Spark-The-Definitive-Guide

Spark: The Definitive Guide's Code Repository

scala-style-guide

Databricks Scala Coding Style Guide

spark-deep-learning

Deep Learning Pipelines for Apache Spark

click

The "Command Line Interactive Controller for Kubernetes"

LearningSparkV2

This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]

megablocks

spark-sklearn

(Deprecated) Scikit-learn integration package for Apache Spark

spark-csv

CSV Data Source for Apache Spark 1.x

tensorframes

[DEPRECATED] Tensorflow wrapper for DataFrames on Apache Spark

devrel

This repository contains the notebooks and presentations we use for our Databricks Tech Talks

reference-apps

Spark reference applications

spark-redshift

Redshift data source for Apache Spark

spark-sql-perf

spark-avro

Avro Data Source for Apache Spark

spark-xml

XML data source for Spark SQL and DataFrames

spark-corenlp

Stanford CoreNLP wrapper for Apache Spark

mlops-stacks

This repo provides a customizable stack for starting new ML projects on Databricks that follow production best-practices out of the box.

spark-training

Apache Spark training material

spark-perf

Performance tests for Apache Spark

delta-live-tables-notebooks

terraform-provider-databricks

Databricks Terraform Provider

spark-knowledgebase

Spark Knowledge Base

databricks-ml-examples

sjsonnet

jsonnet-style-guide

Databricks Jsonnet Coding Style Guide

dbt-databricks

A dbt adapter for Databricks.

databricks-sdk-py

Databricks SDK for Python (Beta)

containers

Sample base images for Databricks Container Services

databricks-sql-python

Databricks SQL Connector for Python

sbt-spark-package

Sbt plugin for Spark packages

notebook-best-practices

An example showing how to apply software engineering best practices to Databricks notebooks.

databricks-vscode

VS Code extension for Databricks

benchmarks

A place in which we publish scripts for reproducible benchmarks.

terraform-databricks-examples

Examples of using Terraform to deploy Databricks resources

spark-tfocs

A Spark port of TFOCS: Templates for First-Order Conic Solvers (cvxr.com/tfocs)

intellij-jsonnet

Intellij Jsonnet Plugin

sbt-databricks

An sbt plugin for deploying code to Databricks Cloud

terraform-databricks-lakehouse-blueprints

Set of Terraform automation templates and quickstart demos to jumpstart the design of a Lakehouse on Databricks. This project has incorporated best practices across the industries we work with to deliver composable modules to build a workspace to comply with the highest platform security and governance standards.

spark-integration-tests

Integration tests for Spark

genai-cookbook

spark-pr-dashboard

Dashboard to aid in Spark pull request reviews

run-notebook

ide-best-practices

Best practices for working with Databricks from an IDE

unity-catalog-setup

Notebooks, terraform, tools to enable setting up Unity Catalog

simr

Spark In MapReduce (SIMR) - launching Spark applications on existing Hadoop MapReduce infrastructure

devbox

databricks-sql-go

Golang database/sql driver for Databricks SQL.

diviner

Grouped time series forecasting engine

cli

tmm

security-bucket-brigade

databricks-sdk-go

Databricks SDK for Go

pig-on-spark

proof-of-concept implementation of Pig-on-Spark integrated at the logical node level

databricks-sql-cli

CLI for querying Databricks SQL

automl

databricks-sql-nodejs

Databricks SQL Connector for Node.js

tpch-dbgen

Patched version of dbgen

als-benchmark-scripts

Scripts to benchmark distributed Alternative Least Squares (ALS)

spark-package-cmd-tool

A command line tool for Spark packages

congruity

The goal of this library is to provide a compatibility layer that makes it easier to adopt Spark Connect. The library is designed to be simply imported in your application and will then monkey-patch the existing API to provide the legacy functionality.

python-interview

Databricks Python interview setup instructions

xgb-regressor

MLflow XGBoost Regressor

databricks-accelerators

Accelerate the use of Databricks for customers [public repo]

tableau-connector

files_in_repos

upload-dbfs-temp

spark-sklearn-docs

sqltools-databricks-driver

SQLTools driver for Databricks SQL

genomics-pipelines

secondary analysis pipelines parallelized with apache spark

workflows-examples

databricks-sdk-java

Databricks SDK for Java

dais-cow-bff

Code for the "Path to Production" DAIS 2024 and 2023 talks

Jupyter Notebook

xgboost-linux64

Databricks Private xgboost Linux64 fork

mlflow-example-sklearn-elasticnet-wine

Jupyter Notebook

databricks-ttyd

setup-cli

Sets up the Databricks CLI in your GitHub Actions workflow.

terraform-databricks-mlops-aws-project

This module creates and configures service principals with appropriate permissions and entitlements to run CI/CD for a project, and creates a workspace directory as a container for project-specific resources for the Databricks AWS staging and prod workspaces.

jenkins-job-builder

Fork of https://docs.openstack.org/infra/jenkins-job-builder/ to include unmerged patches

terraform-databricks-mlops-azure-project-with-sp-creation

This module creates and configures service principals with appropriate permissions and entitlements to run CI/CD for a project, and creates a workspace directory as a container for project-specific resources for the Azure Databricks staging and prod workspaces. It also creates the relevant Azure Active Directory (AAD) applications for the service principals.

terraform-databricks-sra

The Security Reference Architecture (SRA) implements typical security features as Terraform Templates that are deployed by most high-security organizations, and enforces controls for the largest risks that customers ask about most often.

databricks-empty-ide-project

Empty IDE project used by the VSCode extension for Databricks

databricks-repos-proxy

databricks-asset-bundles-dais2023

pex

Fork of pantsbuild/pex with a few Databricks-specific changes

SnpEff

Databricks snpeff fork

notebook_gallery

Jupyter Notebook

terraform-databricks-mlops-aws-infrastructure

This module sets up multi-workspace model registry between a Databricks AWS development (dev) workspace, staging workspace, and production (prod) workspace, allowing READ access from dev/staging workspaces to staging & prod model registries.

expectations

homebrew-tap

Homebrew Tap for the Databricks CLI

terraform-databricks-mlops-azure-infrastructure-with-sp-creation

This module sets up multi-workspace model registry between an Azure Databricks development (dev) workspace, staging workspace, and production (prod) workspace, allowing READ access from dev/staging workspaces to staging & prod model registries. It also creates the relevant Azure Active Directory (AAD) applications for the service principals.

mfg_dlt_workshop

DLT Manufacturing Workshop

databricks-dbutils-scala

The Scala SDK for Databricks.

kdd24-forecasting-anomaly-detection

terraform-databricks-mlops-azure-project-with-sp-linking

This module creates and configures service principals with appropriate permissions and entitlements to run CI/CD for a project, and creates a workspace directory as a container for project-specific resources for the Azure Databricks staging and prod workspaces. It also links pre-existing Azure Active Directory (AAD) applications to the service principals.

terraform-databricks-mlops-azure-infrastructure-with-sp-linking

This module sets up multi-workspace model registry between an Azure Databricks development (dev) workspace, staging workspace, and production (prod) workspace, allowing READ access from dev/staging workspaces to staging & prod model registries. It also links pre-existing Azure Active Directory (AAD) applications to the service principals.