• Stars
    star
    437
  • Rank 98,982 (Top 2 %)
  • Language
    Python
  • License
    Other
  • Created over 4 years ago
  • Updated 3 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

🧱 Databricks CLI eXtensions - aka dbx is a CLI tool for development and advanced Databricks workflows management.

dbx by Databricks Labs

logo

🧱Databricks CLI eXtensions - aka dbx is a CLI tool for development and advanced Databricks workflows management.


Documentation Status Latest Python Release codecov downloads We use black for formatting


Concept

dbx simplifies Databricks workflows development, deployment and launch across multiple environments. It also helps to package your project and deliver it to your Databricks environment in a versioned fashion. Designed in a CLI-first manner, it is built to be actively used both inside CI/CD pipelines and as a part of local tooling for rapid prototyping.

Requirements

  • Python Version > 3.8
  • pip or conda

Installation

  • with pip:
pip install dbx

Documentation

Please refer to the docs page.

Interface versioning

For CLI interfaces, we support SemVer approach. However, for API components we don't use SemVer as of now. This may lead to instability when using dbx API methods directly.

Legal Information

This software is provided as-is and is not officially supported by Databricks through customer technical support channels. Support, questions, and feature requests can be communicated through the Issues page of this repo. Please see the legal agreement and understand that issues with the use of this code will not be answered or investigated by Databricks Support.

Feedback

Issues with dbx? Found a bug? Have a great idea for an addition? Feel free to file an issue.

Contributing

Please find more details about contributing to dbx in the contributing doc.

More Repositories

1

dolly

Databricks’ Dolly, a large language model trained on the Databricks Machine Learning Platform
Python
10,796
star
2

pyspark-ai

English SDK for Apache Spark
Python
739
star
3

tempo

API for manipulating time series on top of Apache Spark: lagged time values, rolling statistics (mean, avg, sum, count, etc), AS OF joins, downsampling, and interpolation
Jupyter Notebook
303
star
4

dbldatagen

Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines
Python
281
star
5

mosaic

An extension to the Apache Spark framework that allows easy and fast processing of very large geospatial datasets.
Jupyter Notebook
262
star
6

overwatch

Capture deep metrics on one or all assets within a Databricks workspace
Scala
221
star
7

cicd-templates

Manage your Databricks deployments and CI with code.
Python
200
star
8

ucx

Your best companion for upgrading to Unity Catalog. UCX will guide you, the Databricks customer, through the process of upgrading your account, groups, workspaces, jobs etc. to Unity Catalog.
Python
193
star
9

automl-toolkit

Toolkit for Apache Spark ML for Feature clean-up, feature Importance calculation suite, Information Gain selection, Distributed SMOTE, Model selection and training, Hyper parameter optimization and selection, Model interprability.
HTML
190
star
10

migrate

Old scripts for one-off ST-to-E2 migrations. Use "terraform exporter" linked in the readme.
Python
177
star
11

dataframe-rules-engine

Extensible Rules Engine for custom Dataframe / Dataset validation
Scala
134
star
12

dlt-meta

This is metadata driven DLT based framework for bronze/silver pipelines
Python
125
star
13

discoverx

A Swiss-Army-knife for your Data Intelligence platform administration.
Python
99
star
14

geoscan

Geospatial clustering at massive scale
Scala
92
star
15

jupyterlab-integration

DEPRECATED: Integrating Jupyter with Databricks via SSH
HTML
71
star
16

smolder

HL7 Apache Spark Datasource
Scala
57
star
17

feature-factory

Accelerator to rapidly deploy customized features for your business
Python
55
star
18

databricks-sync

An experimental tool to synchronize source Databricks deployment with a target Databricks deployment.
Python
46
star
19

doc-qa

Python
42
star
20

transpiler

SIEM-to-Spark Transpiler
Scala
41
star
21

delta-oms

DeltaOMS is a solution that help build a centralized repository of Delta Transaction logs and associated operational metrics/statistics for your Delta Lakehouse. Unity Catalog supported in the v0.7.0-rc1 release.Documentation here - https://databrickslabs.github.io/delta-oms/v0.7.0-rc1/
Scala
37
star
22

brickster

R Toolkit for Databricks
R
36
star
23

splunk-integration

Databricks Add-on for Splunk
Python
26
star
24

dbignite

Python
22
star
25

arcuate

Delta Sharing + MLflow for ML model & experiment exchange (arcuate delta - a fan shaped river delta)
Python
21
star
26

databricks-sdk-r

Databricks SDK for R (Experimental)
R
20
star
27

remorph

Cross-compiler and Data Reconciler into Databricks Lakehouse
Python
18
star
28

tika-ocr

Rich Text Format
17
star
29

sandbox

Experimental or low-maturity things
Go
16
star
30

blueprint

Baseline for Databricks Labs projects written in Python
Python
13
star
31

delta-sharing-java-connector

A Java connector for delta.io/sharing/ that allows you to easily ingest data on any JVM.
Java
12
star
32

partner-connect-api

Scala
12
star
33

waterbear

Automated provisioning of an industry Lakehouse with enterprise data model
Python
8
star
34

pylint-plugin

Databricks Plugin for PyLint
Python
8
star
35

lsql

Lightweight SQL execution wrapper only on top of Databricks SDK
Python
6
star