databricks/spark-avro

This repository has been archived on 20/Dec/2018
Stars
538
Rank 82,538 (Top 2 %)
Language
Scala
License
Apache License 2.0
Created about 10 years ago
Updated almost 6 years ago

databricks/spark-avro

databricks

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Avro Data Source for Apache Spark

Avro Data Source for Apache Spark

Databricks has donated this library to the Apache Spark project, as of Spark 2.4.0. Databricks customers can also use this library directly on the Databricks Unified Analytics Platform without any additional dependency configurations.

Please refer to the original README file for older Spark versions (< 2.4.0).

learning-spark

Example code from Learning Spark book

koalas

Koalas: pandas API on Apache Spark

Spark-The-Definitive-Guide

Spark: The Definitive Guide's Code Repository

scala-style-guide

Databricks Scala Coding Style Guide

spark-deep-learning

Deep Learning Pipelines for Apache Spark

click

The "Command Line Interactive Controller for Kubernetes"

LearningSparkV2

This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]

megablocks

spark-sklearn

(Deprecated) Scikit-learn integration package for Apache Spark

spark-csv

CSV Data Source for Apache Spark 1.x

tensorframes

[DEPRECATED] Tensorflow wrapper for DataFrames on Apache Spark

devrel

This repository contains the notebooks and presentations we use for our Databricks Tech Talks

reference-apps

Spark reference applications

spark-redshift

Redshift data source for Apache Spark

spark-sql-perf

spark-xml

XML data source for Spark SQL and DataFrames

spark-corenlp

Stanford CoreNLP wrapper for Apache Spark

mlops-stacks

This repo provides a customizable stack for starting new ML projects on Databricks that follow production best-practices out of the box.

spark-training

Apache Spark training material

databricks-cli

(Legacy) Command Line Interface for Databricks

spark-perf

Performance tests for Apache Spark

delta-live-tables-notebooks

terraform-provider-databricks

Databricks Terraform Provider

spark-knowledgebase

Spark Knowledge Base

databricks-ml-examples

sjsonnet

jsonnet-style-guide

Databricks Jsonnet Coding Style Guide

dbt-databricks

A dbt adapter for Databricks.

databricks-sdk-py

Databricks SDK for Python (Beta)

containers

Sample base images for Databricks Container Services

databricks-sql-python

Databricks SQL Connector for Python

sbt-spark-package

Sbt plugin for Spark packages

notebook-best-practices

An example showing how to apply software engineering best practices to Databricks notebooks.

databricks-vscode

VS Code extension for Databricks

benchmarks

A place in which we publish scripts for reproducible benchmarks.

terraform-databricks-examples

Examples of using Terraform to deploy Databricks resources

spark-tfocs

A Spark port of TFOCS: Templates for First-Order Conic Solvers (cvxr.com/tfocs)

intellij-jsonnet

Intellij Jsonnet Plugin

sbt-databricks

An sbt plugin for deploying code to Databricks Cloud

terraform-databricks-lakehouse-blueprints

Set of Terraform automation templates and quickstart demos to jumpstart the design of a Lakehouse on Databricks. This project has incorporated best practices across the industries we work with to deliver composable modules to build a workspace to comply with the highest platform security and governance standards.

spark-integration-tests

Integration tests for Spark

genai-cookbook

spark-pr-dashboard

Dashboard to aid in Spark pull request reviews

run-notebook

ide-best-practices

Best practices for working with Databricks from an IDE

unity-catalog-setup

Notebooks, terraform, tools to enable setting up Unity Catalog

simr

Spark In MapReduce (SIMR) - launching Spark applications on existing Hadoop MapReduce infrastructure

devbox

databricks-sql-go

Golang database/sql driver for Databricks SQL.

diviner

Grouped time series forecasting engine

cli

tmm

security-bucket-brigade

databricks-sdk-go

Databricks SDK for Go

pig-on-spark

proof-of-concept implementation of Pig-on-Spark integrated at the logical node level

databricks-sql-cli

CLI for querying Databricks SQL

automl

databricks-sql-nodejs

Databricks SQL Connector for Node.js

tpch-dbgen

Patched version of dbgen

als-benchmark-scripts

Scripts to benchmark distributed Alternative Least Squares (ALS)

spark-package-cmd-tool

A command line tool for Spark packages

congruity

The goal of this library is to provide a compatibility layer that makes it easier to adopt Spark Connect. The library is designed to be simply imported in your application and will then monkey-patch the existing API to provide the legacy functionality.

python-interview

Databricks Python interview setup instructions

xgb-regressor

MLflow XGBoost Regressor

databricks-accelerators

Accelerate the use of Databricks for customers [public repo]

tableau-connector

files_in_repos

upload-dbfs-temp

spark-sklearn-docs

sqltools-databricks-driver

SQLTools driver for Databricks SQL

genomics-pipelines

secondary analysis pipelines parallelized with apache spark

workflows-examples

databricks-sdk-java

Databricks SDK for Java

dais-cow-bff

Code for the "Path to Production" DAIS 2024 and 2023 talks

Jupyter Notebook

xgboost-linux64

Databricks Private xgboost Linux64 fork

mlflow-example-sklearn-elasticnet-wine

Jupyter Notebook

databricks-ttyd

setup-cli

Sets up the Databricks CLI in your GitHub Actions workflow.

terraform-databricks-mlops-aws-project

This module creates and configures service principals with appropriate permissions and entitlements to run CI/CD for a project, and creates a workspace directory as a container for project-specific resources for the Databricks AWS staging and prod workspaces.

jenkins-job-builder

Fork of https://docs.openstack.org/infra/jenkins-job-builder/ to include unmerged patches

terraform-databricks-mlops-azure-project-with-sp-creation

This module creates and configures service principals with appropriate permissions and entitlements to run CI/CD for a project, and creates a workspace directory as a container for project-specific resources for the Azure Databricks staging and prod workspaces. It also creates the relevant Azure Active Directory (AAD) applications for the service principals.

terraform-databricks-sra

The Security Reference Architecture (SRA) implements typical security features as Terraform Templates that are deployed by most high-security organizations, and enforces controls for the largest risks that customers ask about most often.

databricks-empty-ide-project

Empty IDE project used by the VSCode extension for Databricks

databricks-repos-proxy

databricks-asset-bundles-dais2023

pex

Fork of pantsbuild/pex with a few Databricks-specific changes

SnpEff

Databricks snpeff fork

notebook_gallery

Jupyter Notebook

terraform-databricks-mlops-aws-infrastructure

This module sets up multi-workspace model registry between a Databricks AWS development (dev) workspace, staging workspace, and production (prod) workspace, allowing READ access from dev/staging workspaces to staging & prod model registries.

expectations

homebrew-tap

Homebrew Tap for the Databricks CLI

terraform-databricks-mlops-azure-infrastructure-with-sp-creation

This module sets up multi-workspace model registry between an Azure Databricks development (dev) workspace, staging workspace, and production (prod) workspace, allowing READ access from dev/staging workspaces to staging & prod model registries. It also creates the relevant Azure Active Directory (AAD) applications for the service principals.

mfg_dlt_workshop

DLT Manufacturing Workshop

databricks-dbutils-scala

The Scala SDK for Databricks.

kdd24-forecasting-anomaly-detection

terraform-databricks-mlops-azure-project-with-sp-linking

This module creates and configures service principals with appropriate permissions and entitlements to run CI/CD for a project, and creates a workspace directory as a container for project-specific resources for the Azure Databricks staging and prod workspaces. It also links pre-existing Azure Active Directory (AAD) applications to the service principals.

terraform-databricks-mlops-azure-infrastructure-with-sp-linking

This module sets up multi-workspace model registry between an Azure Databricks development (dev) workspace, staging workspace, and production (prod) workspace, allowing READ access from dev/staging workspaces to staging & prod model registries. It also links pre-existing Azure Active Directory (AAD) applications to the service principals.