databricks/LearningSparkV2

Stars
1,158
Rank 40,293 (Top 0.8 %)
Language
Scala
License
Apache License 2.0
Created almost 6 years ago
Updated 7 months ago

databricks/LearningSparkV2

databricks

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]

Learning Spark 2nd Edition

Welcome to the GitHub repo for Learning Spark 2nd Edition.

Chapters 2, 3, 6, and 7 contain stand-alone Spark applications. You can build all the JAR files for each chapter by running the Python script: python build_jars.py. Or you can cd to the chapter directory and build jars as specified in each README. Also, include $SPARK_HOME/bin in $PATH so that you don't have to prefix SPARK_HOME/bin/spark-submit for these standalone applications.

For all the other chapters, we have provided notebooks in the notebooks folder. We have also included notebook equivalents for a few of the stand-alone Spark applications in the aforementioned chapters.

Have Fun, Cheers!

learning-spark

Example code from Learning Spark book

koalas

Koalas: pandas API on Apache Spark

Spark-The-Definitive-Guide

Spark: The Definitive Guide's Code Repository

scala-style-guide

Databricks Scala Coding Style Guide

spark-deep-learning

Deep Learning Pipelines for Apache Spark

click

The "Command Line Interactive Controller for Kubernetes"

megablocks

spark-sklearn

(Deprecated) Scikit-learn integration package for Apache Spark

spark-csv

CSV Data Source for Apache Spark 1.x

tensorframes

[DEPRECATED] Tensorflow wrapper for DataFrames on Apache Spark

devrel

This repository contains the notebooks and presentations we use for our Databricks Tech Talks

reference-apps

Spark reference applications

spark-redshift

Redshift data source for Apache Spark

spark-sql-perf

spark-avro

Avro Data Source for Apache Spark

spark-xml

XML data source for Spark SQL and DataFrames

spark-corenlp

Stanford CoreNLP wrapper for Apache Spark

mlops-stacks

This repo provides a customizable stack for starting new ML projects on Databricks that follow production best-practices out of the box.

spark-training

Apache Spark training material

databricks-cli

(Legacy) Command Line Interface for Databricks

spark-perf

Performance tests for Apache Spark

delta-live-tables-notebooks

terraform-provider-databricks

Databricks Terraform Provider

spark-knowledgebase

Spark Knowledge Base

databricks-ml-examples

sjsonnet

jsonnet-style-guide

Databricks Jsonnet Coding Style Guide

dbt-databricks

A dbt adapter for Databricks.

databricks-sdk-py

Databricks SDK for Python (Beta)

containers

Sample base images for Databricks Container Services

databricks-sql-python

Databricks SQL Connector for Python

sbt-spark-package

Sbt plugin for Spark packages

notebook-best-practices

An example showing how to apply software engineering best practices to Databricks notebooks.

databricks-vscode

VS Code extension for Databricks

benchmarks

A place in which we publish scripts for reproducible benchmarks.

terraform-databricks-examples

Examples of using Terraform to deploy Databricks resources

spark-tfocs

A Spark port of TFOCS: Templates for First-Order Conic Solvers (cvxr.com/tfocs)

intellij-jsonnet

Intellij Jsonnet Plugin

sbt-databricks

An sbt plugin for deploying code to Databricks Cloud

terraform-databricks-lakehouse-blueprints

Set of Terraform automation templates and quickstart demos to jumpstart the design of a Lakehouse on Databricks. This project has incorporated best practices across the industries we work with to deliver composable modules to build a workspace to comply with the highest platform security and governance standards.

spark-integration-tests

Integration tests for Spark

genai-cookbook

spark-pr-dashboard

Dashboard to aid in Spark pull request reviews

run-notebook

ide-best-practices

Best practices for working with Databricks from an IDE

unity-catalog-setup

Notebooks, terraform, tools to enable setting up Unity Catalog

simr

Spark In MapReduce (SIMR) - launching Spark applications on existing Hadoop MapReduce infrastructure

devbox

databricks-sql-go

Golang database/sql driver for Databricks SQL.

diviner

Grouped time series forecasting engine

cli

tmm

security-bucket-brigade

databricks-sdk-go

Databricks SDK for Go

pig-on-spark

proof-of-concept implementation of Pig-on-Spark integrated at the logical node level

databricks-sql-cli

CLI for querying Databricks SQL

automl

databricks-sql-nodejs

Databricks SQL Connector for Node.js

tpch-dbgen

Patched version of dbgen

als-benchmark-scripts

Scripts to benchmark distributed Alternative Least Squares (ALS)

spark-package-cmd-tool

A command line tool for Spark packages

congruity

The goal of this library is to provide a compatibility layer that makes it easier to adopt Spark Connect. The library is designed to be simply imported in your application and will then monkey-patch the existing API to provide the legacy functionality.

python-interview

Databricks Python interview setup instructions

xgb-regressor

MLflow XGBoost Regressor

databricks-accelerators

Accelerate the use of Databricks for customers [public repo]

tableau-connector

files_in_repos

upload-dbfs-temp

spark-sklearn-docs

sqltools-databricks-driver

SQLTools driver for Databricks SQL

genomics-pipelines

secondary analysis pipelines parallelized with apache spark

workflows-examples

databricks-sdk-java

Databricks SDK for Java

dais-cow-bff

Code for the "Path to Production" DAIS 2024 and 2023 talks

Jupyter Notebook

xgboost-linux64

Databricks Private xgboost Linux64 fork

mlflow-example-sklearn-elasticnet-wine

Jupyter Notebook

databricks-ttyd

setup-cli

Sets up the Databricks CLI in your GitHub Actions workflow.

terraform-databricks-mlops-aws-project

This module creates and configures service principals with appropriate permissions and entitlements to run CI/CD for a project, and creates a workspace directory as a container for project-specific resources for the Databricks AWS staging and prod workspaces.

jenkins-job-builder

Fork of https://docs.openstack.org/infra/jenkins-job-builder/ to include unmerged patches

terraform-databricks-mlops-azure-project-with-sp-creation

This module creates and configures service principals with appropriate permissions and entitlements to run CI/CD for a project, and creates a workspace directory as a container for project-specific resources for the Azure Databricks staging and prod workspaces. It also creates the relevant Azure Active Directory (AAD) applications for the service principals.

terraform-databricks-sra

The Security Reference Architecture (SRA) implements typical security features as Terraform Templates that are deployed by most high-security organizations, and enforces controls for the largest risks that customers ask about most often.

databricks-empty-ide-project

Empty IDE project used by the VSCode extension for Databricks

databricks-repos-proxy

databricks-asset-bundles-dais2023

pex

Fork of pantsbuild/pex with a few Databricks-specific changes

SnpEff

Databricks snpeff fork

notebook_gallery

Jupyter Notebook

terraform-databricks-mlops-aws-infrastructure

This module sets up multi-workspace model registry between a Databricks AWS development (dev) workspace, staging workspace, and production (prod) workspace, allowing READ access from dev/staging workspaces to staging & prod model registries.

expectations

homebrew-tap

Homebrew Tap for the Databricks CLI

terraform-databricks-mlops-azure-infrastructure-with-sp-creation

This module sets up multi-workspace model registry between an Azure Databricks development (dev) workspace, staging workspace, and production (prod) workspace, allowing READ access from dev/staging workspaces to staging & prod model registries. It also creates the relevant Azure Active Directory (AAD) applications for the service principals.

mfg_dlt_workshop

DLT Manufacturing Workshop

databricks-dbutils-scala

The Scala SDK for Databricks.

kdd24-forecasting-anomaly-detection

terraform-databricks-mlops-azure-project-with-sp-linking

This module creates and configures service principals with appropriate permissions and entitlements to run CI/CD for a project, and creates a workspace directory as a container for project-specific resources for the Azure Databricks staging and prod workspaces. It also links pre-existing Azure Active Directory (AAD) applications to the service principals.

terraform-databricks-mlops-azure-infrastructure-with-sp-linking

This module sets up multi-workspace model registry between an Azure Databricks development (dev) workspace, staging workspace, and production (prod) workspace, allowing READ access from dev/staging workspaces to staging & prod model registries. It also links pre-existing Azure Active Directory (AAD) applications to the service principals.