Discover databricks/diviner Open Source project

Koalas: pandas API on Apache Spark

3,864

koalas

Spark: The Definitive Guide's Code Repository

3,330

Spark-The-Definitive-Guide

Databricks Scala Coding Style Guide

2,678

scala-style-guide

2,673

spark-deep-learning

Deep Learning Pipelines for Apache Spark

The "Command Line Interactive Controller for Kubernetes"

1,993

click

Rust

1,416

LearningSparkV2

This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]

megablocks

spark-sklearn

(Deprecated) Scikit-learn integration package for Apache Spark

CSV Data Source for Apache Spark 1.x

1,080

spark-csv

[DEPRECATED] Tensorflow wrapper for DataFrames on Apache Spark

1,051

tensorframes

This repository contains the notebooks and presentations we use for our Databricks Tech Talks

749

devrel

HTML

687

reference-apps

Spark reference applications

Redshift data source for Apache Spark

648

spark-redshift

spark-sql-perf

spark-avro

Avro Data Source for Apache Spark

XML data source for Spark SQL and DataFrames

538

spark-xml

Stanford CoreNLP wrapper for Apache Spark

501

spark-corenlp

This repo provides a customizable stack for starting new ML projects on Databricks that follow production best-practices out of the box.

424

mlops-stacks

Apache Spark training material

415

spark-training

(Legacy) Command Line Interface for Databricks

396

databricks-cli

Performance tests for Apache Spark

384

spark-perf

372

delta-live-tables-notebooks

Databricks Terraform Provider

334

terraform-provider-databricks

333

spark-knowledgebase

Spark Knowledge Base

328

databricks-ml-examples

sjsonnet

jsonnet-style-guide

Databricks Jsonnet Coding Style Guide

205

dbt-databricks

A dbt adapter for Databricks.

Databricks SDK for Python (Beta)

199

databricks-sdk-py

Sample base images for Databricks Container Services

185

containers

Dockerfile

163

databricks-sql-python

Databricks SQL Connector for Python

Sbt plugin for Spark packages

158

sbt-spark-package

An example showing how to apply software engineering best practices to Databricks notebooks.

150

notebook-best-practices

VS Code extension for Databricks

116

databricks-vscode

A place in which we publish scripts for reproducible benchmarks.

114

benchmarks

Examples of using Terraform to deploy Databricks resources

106

terraform-databricks-examples

A Spark port of TFOCS: Templates for First-Order Conic Solvers (cvxr.com/tfocs)

103

spark-tfocs

intellij-jsonnet

Intellij Jsonnet Plugin

An sbt plugin for deploying code to Databricks Cloud

sbt-databricks

Set of Terraform automation templates and quickstart demos to jumpstart the design of a Lakehouse on Databricks. This project has incorporated best practices across the industries we work with to deliver composable modules to build a workspace to comply with the highest platform security and governance standards.

terraform-databricks-lakehouse-blueprints

Integration tests for Spark

spark-integration-tests

genai-cookbook

Dashboard to aid in Spark pull request reviews

spark-pr-dashboard

JavaScript

run-notebook

Best practices for working with Databricks from an IDE

ide-best-practices

Notebooks, terraform, tools to enable setting up Unity Catalog

unity-catalog-setup

simr

Spark In MapReduce (SIMR) - launching Spark applications on existing Hadoop MapReduce infrastructure

devbox

Golang database/sql driver for Databricks SQL.

databricks-sql-go

cli

Databricks CLI

tmm

proof-of-concept implementation of Pig-on-Spark integrated at the logical node level

security-bucket-brigade

JavaScript

databricks-sdk-go

Databricks SDK for Go

pig-on-spark

CLI for querying Databricks SQL

databricks-sql-cli

automl

Databricks SQL Connector for Node.js

databricks-sql-nodejs

Scripts to benchmark distributed Alternative Least Squares (ALS)

tpch-dbgen

Patched version of dbgen

als-benchmark-scripts

A command line tool for Spark packages

spark-package-cmd-tool

The goal of this library is to provide a compatibility layer that makes it easier to adopt Spark Connect. The library is designed to be simply imported in your application and will then monkey-patch the existing API to provide the legacy functionality.

congruity

Databricks Python interview setup instructions

python-interview

xgb-regressor

MLflow XGBoost Regressor

Accelerate the use of Databricks for customers [public repo]

databricks-accelerators

tableau-connector

files_in_repos

upload-dbfs-temp

spark-sklearn-docs

sqltools-databricks-driver

SQLTools driver for Databricks SQL

secondary analysis pipelines parallelized with apache spark

genomics-pipelines

workflows-examples

databricks-sdk-java

Databricks SDK for Java

Code for the "Path to Production" DAIS 2024 and 2023 talks

dais-cow-bff

Jupyter Notebook

xgboost-linux64

Databricks Private xgboost Linux64 fork

C++

mlflow-example-sklearn-elasticnet-wine

Jupyter Notebook

databricks-ttyd

setup-cli

Sets up the Databricks CLI in your GitHub Actions workflow.

Shell

terraform-databricks-mlops-aws-project

This module creates and configures service principals with appropriate permissions and entitlements to run CI/CD for a project, and creates a workspace directory as a container for project-specific resources for the Databricks AWS staging and prod workspaces.

Fork of https://docs.openstack.org/infra/jenkins-job-builder/ to include unmerged patches

jenkins-job-builder

terraform-databricks-mlops-azure-project-with-sp-creation

The Security Reference Architecture (SRA) implements typical security features as Terraform Templates that are deployed by most high-security organizations, and enforces controls for the largest risks that customers ask about most often.

terraform-databricks-sra

Empty IDE project used by the VSCode extension for Databricks

databricks-empty-ide-project

databricks-repos-proxy

databricks-asset-bundles-dais2023

Fork of pantsbuild/pex with a few Databricks-specific changes

pex

SnpEff

Databricks snpeff fork

This module sets up multi-workspace model registry between a Databricks AWS development (dev) workspace, staging workspace, and production (prod) workspace, allowing READ access from dev/staging workspaces to staging & prod model registries.

notebook_gallery

Jupyter Notebook

terraform-databricks-mlops-aws-infrastructure

expectations

Homebrew Tap for the Databricks CLI

homebrew-tap

Ruby

terraform-databricks-mlops-azure-infrastructure-with-sp-creation

This module sets up multi-workspace model registry between an Azure Databricks development (dev) workspace, staging workspace, and production (prod) workspace, allowing READ access from dev/staging workspaces to staging & prod model registries. It also creates the relevant Azure Active Directory (AAD) applications for the service principals.

DLT Manufacturing Workshop

mfg_dlt_workshop

The Scala SDK for Databricks.

databricks-dbutils-scala

kdd24-forecasting-anomaly-detection

terraform-databricks-mlops-azure-project-with-sp-linking

This module sets up multi-workspace model registry between an Azure Databricks development (dev) workspace, staging workspace, and production (prod) workspace, allowing READ access from dev/staging workspaces to staging & prod model registries. It also links pre-existing Azure Active Directory (AAD) applications to the service principals.

terraform-databricks-mlops-azure-infrastructure-with-sp-linking