• Stars
    star
    751
  • Rank 57,983 (Top 2 %)
  • Language
    Scala
  • License
    Apache License 2.0
  • Created about 8 years ago
  • Updated about 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

[DEPRECATED] Tensorflow wrapper for DataFrames on Apache Spark

build

TensorFrames (Deprecated)

Note: TensorFrames is deprecated. You can use pandas UDF instead.

Experimental TensorFlow binding for Scala and Apache Spark.

TensorFrames (TensorFlow on Spark DataFrames) lets you manipulate Apache Spark's DataFrames with TensorFlow programs.

This package is experimental and is provided as a technical preview only. While the interfaces are all implemented and working, there are still some areas of low performance.

Supported platforms:

This package only officially supports linux 64bit platforms as a target. Contributions are welcome for other platforms.

See the file project/Dependencies.scala for adding your own platform.

Officially TensorFrames supports Spark 2.4+ and Scala 2.11.

See the user guide for extensive information about the API.

For questions, see the TensorFrames mailing list.

TensorFrames is available as a Spark package.

Requirements

  • A working version of Apache Spark (2.4 or greater)

  • Java 8+

  • (Optional) python 2.7+/3.6+ if you want to use the python interface.

  • (Optional) the python TensorFlow package if you want to use the python interface. See the official instructions on how to get the latest release of TensorFlow.

  • (Optional) pandas >= 0.19.1 if you want to use the python interface

Additionally, for developement, you need the following dependencies:

  • protoc 3.x

  • nose >= 1.3

How to run in python

Assuming that SPARK_HOME is set, you can use PySpark like any other Spark package.

$SPARK_HOME/bin/pyspark --packages databricks:tensorframes:0.6.0-s_2.11

Here is a small program that uses TensorFlow to add 3 to an existing column.

import tensorflow as tf
import tensorframes as tfs
from pyspark.sql import Row

data = [Row(x=float(x)) for x in range(10)]
df = sqlContext.createDataFrame(data)
with tf.Graph().as_default() as g:
    # The TensorFlow placeholder that corresponds to column 'x'.
    # The shape of the placeholder is automatically inferred from the DataFrame.
    x = tfs.block(df, "x")
    # The output that adds 3 to x
    z = tf.add(x, 3, name='z')
    # The resulting dataframe
    df2 = tfs.map_blocks(z, df)

# The transform is lazy as for most DataFrame operations. This will trigger it:
df2.collect()

# Notice that z is an extra column next to x

# [Row(z=3.0, x=0.0),
#  Row(z=4.0, x=1.0),
#  Row(z=5.0, x=2.0),
#  Row(z=6.0, x=3.0),
#  Row(z=7.0, x=4.0),
#  Row(z=8.0, x=5.0),
#  Row(z=9.0, x=6.0),
#  Row(z=10.0, x=7.0),
#  Row(z=11.0, x=8.0),
#  Row(z=12.0, x=9.0)]

The second example shows the block-wise reducing operations: we compute the sum of a field containing vectors of integers, working with blocks of rows for more efficient processing.

# Build a DataFrame of vectors
data = [Row(y=[float(y), float(-y)]) for y in range(10)]
df = sqlContext.createDataFrame(data)
# Because the dataframe contains vectors, we need to analyze it first to find the
# dimensions of the vectors.
df2 = tfs.analyze(df)

# The information gathered by TF can be printed to check the content:
tfs.print_schema(df2)
# root
#  |-- y: array (nullable = false) double[?,2]

# Let's use the analyzed dataframe to compute the sum and the elementwise minimum 
# of all the vectors:
# First, let's make a copy of the 'y' column. This will be very cheap in Spark 2.0+
df3 = df2.select(df2.y, df2.y.alias("z"))
with tf.Graph().as_default() as g:
    # The placeholders. Note the special name that end with '_input':
    y_input = tfs.block(df3, 'y', tf_name="y_input")
    z_input = tfs.block(df3, 'z', tf_name="z_input")
    y = tf.reduce_sum(y_input, [0], name='y')
    z = tf.reduce_min(z_input, [0], name='z')
    # The resulting dataframe
    (data_sum, data_min) = tfs.reduce_blocks([y, z], df3)

# The final results are numpy arrays:
print(data_sum)
# [45., -45.]
print(data_min)
# [0., -9.]

Notes

Note the scoping of the graphs above. This is important because TensorFrames finds which DataFrame column to feed to TensorFrames based on the placeholders of the graph. Also, it is good practice to keep small graphs when sending them to Spark.

For small tensors (scalars and vectors), TensorFrames usually infers the shapes of the tensors without requiring a preliminary analysis. If it cannot do it, an error message will indicate that you need to run the DataFrame through tfs.analyze() first.

Look at the python documentation of the TensorFrames package to see what methods are available.

How to run in Scala

The scala support is a bit more limited than python. In scala, operations can be loaded from an existing graph defined in the ProtocolBuffers format, or using a simple scala DSL. The Scala DSL only features a subset of TensorFlow transforms. It is very easy to extend though, so other transforms will be added without much effort in the future.

You simply use the published package:

$SPARK_HOME/bin/spark-shell --packages databricks:tensorframes:0.6.0-s_2.11

Here is the same program as before:

import org.tensorframes.{dsl => tf}
import org.tensorframes.dsl.Implicits._

val df = spark.createDataFrame(Seq(1.0->1.1, 2.0->2.2)).toDF("a", "b")

// As in Python, scoping is recommended to prevent name collisions.
val df2 = tf.withGraph {
    val a = df.block("a")
    // Unlike python, the scala syntax is more flexible:
    val out = a + 3.0 named "out"
    // The 'mapBlocks' method is added using implicits to dataframes.
    df.mapBlocks(out).select("a", "out")
}

// The transform is all lazy at this point, let's execute it with collect:
df2.collect()
// res0: Array[org.apache.spark.sql.Row] = Array([1.0,4.0], [2.0,5.0])   

How to compile and install for developers

It is recommended you use Conda Environment to guarantee that the build environment can be reproduced. Once you have installed Conda, you can set the environment from the root of project:

conda create -q -n tensorframes-environment python=$PYTHON_VERSION

This will create an environment for your project. We recommend using Python version 3.7 or 2.7.13. After the environemnt is created, you can activate it and install all dependencies as follows:

conda activate tensorframes-environment
pip install --user -r python/requirements.txt

You also need to compile the scala code. The recommended procedure is to use the assembly:

build/sbt tfs_testing/assembly
# Builds the spark package:
build/sbt distribution/spDist

Assuming that SPARK_HOME is set and that you are in the root directory of the project:

$SPARK_HOME/bin/spark-shell --jars $PWD/target/testing/scala-2.11/tensorframes-assembly-0.6.1-SNAPSHOT.jar

If you want to run the python version:

PYTHONPATH=$PWD/target/testing/scala-2.11/tensorframes-assembly-0.6.1-SNAPSHOT.jar \
$SPARK_HOME/bin/pyspark --jars $PWD/target/testing/scala-2.11/tensorframes-assembly-0.6.1-SNAPSHOT.jar

Acknowledgements

Before TensorFlow released its Java API, this project was built on the great javacpp project, that implements the low-level bindings between TensorFlow and the Java virtual machine.

Many thanks to Google for the release of TensorFlow.

More Repositories

1

learning-spark

Example code from Learning Spark book
Java
3,864
star
2

koalas

Koalas: pandas API on Apache Spark
Python
3,317
star
3

Spark-The-Definitive-Guide

Spark: The Definitive Guide's Code Repository
Scala
2,678
star
4

scala-style-guide

Databricks Scala Coding Style Guide
2,673
star
5

spark-deep-learning

Deep Learning Pipelines for Apache Spark
Python
1,984
star
6

click

The "Command Line Interactive Controller for Kubernetes"
Rust
1,416
star
7

LearningSparkV2

This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]
Scala
1,077
star
8

spark-sklearn

(Deprecated) Scikit-learn integration package for Apache Spark
Python
1,076
star
9

spark-csv

CSV Data Source for Apache Spark 1.x
Scala
1,051
star
10

devrel

This repository contains the notebooks and presentations we use for our Databricks Tech Talks
HTML
672
star
11

reference-apps

Spark reference applications
Scala
648
star
12

spark-redshift

Redshift data source for Apache Spark
Scala
598
star
13

spark-sql-perf

Scala
543
star
14

spark-avro

Avro Data Source for Apache Spark
Scala
538
star
15

spark-xml

XML data source for Spark SQL and DataFrames
Scala
481
star
16

spark-corenlp

Stanford CoreNLP wrapper for Apache Spark
Scala
424
star
17

spark-training

Apache Spark training material
Scala
396
star
18

databricks-cli

(Legacy) Command Line Interface for Databricks
Python
376
star
19

spark-perf

Performance tests for Apache Spark
Scala
372
star
20

terraform-provider-databricks

Databricks Terraform Provider
Go
333
star
21

spark-knowledgebase

Spark Knowledge Base
328
star
22

delta-live-tables-notebooks

Python
296
star
23

databricks-ml-examples

Python
284
star
24

sjsonnet

Scala
252
star
25

mlops-stacks

This repo provides a customizable stack for starting new ML projects on Databricks that follow production best-practices out of the box.
Python
243
star
26

jsonnet-style-guide

Databricks Jsonnet Coding Style Guide
205
star
27

databricks-sdk-py

Databricks SDK for Python (Beta)
Python
185
star
28

dbt-databricks

A dbt adapter for Databricks.
Python
179
star
29

containers

Sample base images for Databricks Container Services
Dockerfile
157
star
30

sbt-spark-package

Sbt plugin for Spark packages
Scala
150
star
31

databricks-sql-python

Databricks SQL Connector for Python
Python
125
star
32

benchmarks

A place in which we publish scripts for reproducible benchmarks.
Python
106
star
33

databricks-vscode

VS Code extension for Databricks
TypeScript
104
star
34

terraform-databricks-examples

Examples of using Terraform to deploy Databricks resources
HCL
103
star
35

notebook-best-practices

An example showing how to apply software engineering best practices to Databricks notebooks.
Python
102
star
36

spark-tfocs

A Spark port of TFOCS: Templates for First-Order Conic Solvers (cvxr.com/tfocs)
Scala
88
star
37

intellij-jsonnet

Intellij Jsonnet Plugin
Java
82
star
38

sbt-databricks

An sbt plugin for deploying code to Databricks Cloud
Scala
71
star
39

spark-integration-tests

Integration tests for Spark
Scala
68
star
40

terraform-databricks-lakehouse-blueprints

Set of Terraform automation templates and quickstart demos to jumpstart the design of a Lakehouse on Databricks. This project has incorporated best practices across the industries we work with to deliver composable modules to build a workspace to comply with the highest platform security and governance standards.
Python
61
star
41

spark-pr-dashboard

Dashboard to aid in Spark pull request reviews
JavaScript
54
star
42

run-notebook

TypeScript
44
star
43

simr

Spark In MapReduce (SIMR) - launching Spark applications on existing Hadoop MapReduce infrastructure
Java
44
star
44

ide-best-practices

Best practices for working with Databricks from an IDE
Python
40
star
45

devbox

Scala
37
star
46

unity-catalog-setup

Notebooks, terraform, tools to enable setting up Unity Catalog
37
star
47

diviner

Grouped time series forecasting engine
Python
33
star
48

cli

Databricks CLI
Go
32
star
49

security-bucket-brigade

JavaScript
30
star
50

databricks-sdk-go

Databricks SDK for Go
Go
29
star
51

pig-on-spark

proof-of-concept implementation of Pig-on-Spark integrated at the logical node level
Scala
28
star
52

databricks-sql-cli

CLI for querying Databricks SQL
Python
27
star
53

automl

Python
26
star
54

databricks-sql-go

Golang database/sql driver for Databricks SQL.
Go
24
star
55

tpch-dbgen

Patched version of dbgen
C
22
star
56

als-benchmark-scripts

Scripts to benchmark distributed Alternative Least Squares (ALS)
Scala
22
star
57

databricks-sql-nodejs

Databricks SQL Connector for Node.js
JavaScript
21
star
58

spark-package-cmd-tool

A command line tool for Spark packages
Python
18
star
59

python-interview

Databricks Python interview setup instructions
15
star
60

xgb-regressor

MLflow XGBoost Regressor
Python
15
star
61

databricks-accelerators

Accelerate the use of Databricks for customers [public repo]
Python
15
star
62

tableau-connector

Scala
12
star
63

files_in_repos

Python
12
star
64

upload-dbfs-temp

TypeScript
12
star
65

spark-sklearn-docs

HTML
11
star
66

genomics-pipelines

secondary analysis pipelines parallelized with apache spark
Scala
10
star
67

workflows-examples

10
star
68

databricks-sdk-java

Databricks SDK for Java
Java
10
star
69

sqltools-databricks-driver

SQLTools driver for Databricks SQL
TypeScript
9
star
70

xgboost-linux64

Databricks Private xgboost Linux64 fork
C++
8
star
71

tmm

Python
7
star
72

mlflow-example-sklearn-elasticnet-wine

Jupyter Notebook
7
star
73

databricks-ttyd

C
6
star
74

dais-cow-bff

Code for the "Bridging the Production Gap" DAIS 2023 talk
Jupyter Notebook
4
star
75

setup-cli

Sets up the Databricks CLI in your GitHub Actions workflow.
Shell
4
star
76

terraform-databricks-mlops-aws-project

This module creates and configures service principals with appropriate permissions and entitlements to run CI/CD for a project, and creates a workspace directory as a container for project-specific resources for the Databricks AWS staging and prod workspaces.
HCL
4
star
77

jenkins-job-builder

Fork of https://docs.openstack.org/infra/jenkins-job-builder/ to include unmerged patches
Python
4
star
78

terraform-databricks-mlops-azure-project-with-sp-creation

This module creates and configures service principals with appropriate permissions and entitlements to run CI/CD for a project, and creates a workspace directory as a container for project-specific resources for the Azure Databricks staging and prod workspaces. It also creates the relevant Azure Active Directory (AAD) applications for the service principals.
HCL
4
star
79

terraform-databricks-sra

The Security Reference Architecture (SRA) implements typical security features as Terraform Templates that are deployed by most high-security organizations, and enforces controls for the largest risks that customers ask about most often.
HCL
4
star
80

databricks-empty-ide-project

Empty IDE project used by the VSCode extension for Databricks
3
star
81

databricks-repos-proxy

Python
2
star
82

databricks-asset-bundles-dais2023

Python
2
star
83

pex

Fork of pantsbuild/pex with a few Databricks-specific changes
Python
2
star
84

SnpEff

Databricks snpeff fork
Java
2
star
85

databricks-dbutils-scala

The Scala SDK for Databricks.
Scala
2
star
86

notebook_gallery

Jupyter Notebook
2
star
87

terraform-databricks-mlops-aws-infrastructure

This module sets up multi-workspace model registry between a Databricks AWS development (dev) workspace, staging workspace, and production (prod) workspace, allowing READ access from dev/staging workspaces to staging & prod model registries.
HCL
2
star
88

homebrew-tap

Homebrew Tap for the Databricks CLI
Ruby
1
star
89

terraform-databricks-mlops-azure-infrastructure-with-sp-creation

This module sets up multi-workspace model registry between an Azure Databricks development (dev) workspace, staging workspace, and production (prod) workspace, allowing READ access from dev/staging workspaces to staging & prod model registries. It also creates the relevant Azure Active Directory (AAD) applications for the service principals.
HCL
1
star
90

mfg_dlt_workshop

DLT Manufacturing Workshop
Python
1
star
91

terraform-databricks-mlops-azure-project-with-sp-linking

This module creates and configures service principals with appropriate permissions and entitlements to run CI/CD for a project, and creates a workspace directory as a container for project-specific resources for the Azure Databricks staging and prod workspaces. It also links pre-existing Azure Active Directory (AAD) applications to the service principals.
HCL
1
star
92

terraform-databricks-mlops-azure-infrastructure-with-sp-linking

This module sets up multi-workspace model registry between an Azure Databricks development (dev) workspace, staging workspace, and production (prod) workspace, allowing READ access from dev/staging workspaces to staging & prod model registries. It also links pre-existing Azure Active Directory (AAD) applications to the service principals.
HCL
1
star