• Stars
    star
    958
  • Rank 47,718 (Top 1.0 %)
  • Language
    Scala
  • License
    Apache License 2.0
  • Created about 10 years ago
  • Updated 4 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Sparkling Water provides H2O functionality inside Spark cluster

sparkling-water-logo

mvn-badge apache-2-0-license Powered by H2O.ai

Sparkling Water

Sparkling Water integrates H2O-3, a fast scalable machine learning engine with Apache Spark. It provides:

  • Utilities to publish Spark data structures (RDDs, DataFrames, Datasets) as H2O-3's frames and vice versa.
  • DSL to use Spark data structures as input for H2O's algorithms.
  • Basic building blocks to create ML applications utilizing Spark and H2O APIs.
  • Python interface enabling use of Sparkling Water directly from PySpark.

Getting Started

User Documentation

Read the documentation for Spark 3.3 (or 3.2 , 3.1, 3.0, 2.4, 2.3)

Download Binaries

Download the latest version for Spark 3.3 (or 3.2, 3.1, 3.0, 2.4, 2.3)

Each Sparkling Water release is also published into the Maven Central (more details below).


Try Sparkling Water!

Sparkling Water is distributed as a Spark application library which can be used by any Spark application. Furthermore, we provide also zip distribution which bundles the library and shell scripts.

There are several ways of using Sparkling Water:

  • Sparkling Shell (Spark Shell with Sparkling Water included)
  • Sparkling Water driver (Spark Submit with Sparkling Water included)
  • Spark Shell and include Sparkling Water library via --jars or --packages option
  • Spark Submit and include Sparkling Water library via --jars or --packages option
  • PySpark with PySparkling

Run Sparkling shell

The Sparkling shell encapsulates a regular Spark shell and append Sparkling Water library on the classpath via --jars option. The Sparkling Shell supports creation of an H2O-3 cloud and execution of H2O-3 algorithms.

  1. Either download or build Sparkling Water

  2. Configure the location of Spark cluster:

    export SPARK_HOME="/path/to/spark/installation"
    export MASTER="local[*]"

    In this case, local[*] points to an embedded single node cluster.

  3. Run Sparkling Shell:

    bin/sparkling-shell

    Sparkling Shell accepts common Spark Shell arguments. For example, to increase memory allocated by each executor, use the spark.executor.memory parameter: bin/sparkling-shell --conf "spark.executor.memory=4g"

  4. Initialize H2OContext

    import ai.h2o.sparkling._
    val hc = H2OContext.getOrCreate()

    H2OContext starts H2O services on top of Spark cluster and provides primitives for transformations between H2O-3 and Spark data structures.

Use Sparkling Water with PySpark

Sparkling Water can be also used directly from PySpark and the integration is called PySparkling.

See PySparkling README to learn about PySparkling.

Use Sparkling Water via Spark Packages

To see how Sparkling Water can be used as Spark package, please see Use as Spark Package.

Use Sparkling Water in Windows environments

See Windows Tutorial to learn how to use Sparkling Water in Windows environments.

Sparkling Water examples

To see how to run examples for Sparkling Water, please see Running Examples.

Maven packages

Each Sparkling Water release is published into Maven central with following coordinates:

  • ai.h2o:sparkling-water-core_{{scala_version}}:{{version}} - Includes core of Sparkling Water

  • ai.h2o:sparkling-water-examples_{{scala_version}}:{{version}} - Includes example applications

  • ai.h2o:sparkling-water-repl_{{scala_version}}:{{version}} - Spark REPL integration into H2O Flow UI

  • ai.h2o:sparkling-water-ml_{{scala_version}}:{{version}} - Extends Spark ML package by H2O-based transformations

  • ai.h2o:sparkling-water-scoring_{{scala_version}}:{{version}} - A library containing scoring logic and definition of Sparkling Water MOJO models.

  • ai.h2o:sparkling-water-scoring-package_{{scala_version}}:{{version}} - Lightweight Sparkling Water package including all dependencies required just for scoring with H2O-3 and DAI MOJO models.

  • ai.h2o:sparkling-water-package_{{scala_version}}:{{version}} - Sparkling Water package containing all dependencies required for model training and scoring. This is designed to use as Spark package via --packages option.

    Note: The {{version}} references to a release version of Sparkling Water, the {{scala_version}} references to Scala base version.

The full list of published packages is available here.


Sparkling Water Backends

Sparkling water supports two backend/deployment modes - internal and external. Sparkling Water applications are independent on the selected backend. The backend can be specified before creation of the H2OContext.

For more details regarding the internal or external backend, please see Backends.


FAQ

List of all Frequently Asked Questions is available at FAQ.


Development

Complete development documentation is available at Development Documentation.

Build Sparkling Water

To see how to build Sparkling Water, please see Build Sparkling Water.

Develop applications with Sparkling Water

An application using Sparkling Water is regular Spark application which bundling Sparkling Water library. See Sparkling Water Droplet providing an example application here.

Contributing

Look at our list of JIRA tasks or send your idea to [email protected].

Filing Bug Reports and Feature Requests

You can file a bug report of feature request directly in the Sparkling Water JIRA page at http://jira.h2o.ai/.

  1. Log in to the Sparkling Water JIRA tracking system. (Create an account if necessary.)

  2. Once inside the home page, click the Create button.

    center
  3. A form will display allowing you to enter information about the bug or feature request.

    center

    Enter the following on the form:

    • Select the Project that you want to file the issue under. For example, if this is an open source public bug, you should file it under SW (SW).
    • Specify the Issue Type. For example, if you believe you've found a bug, then select Bug, or if you want to request a new feature, then select New Feature.
    • Provide a short but concise summary about the issue. The summary will be shown when engineers organize, filter, and search for Jira tickets.
    • Specify the urgency of the issue using the Priority dropdown menu.
    • If there is a due date specify it with the Due Date.
    • The Components drop down refers to the API or language that the issue relates to. (See the drop down menu for available options.)
    • You can leave Affects Version/s, Fix Versions, and Assignee fields blank. Our engineering team will fill this in.
    • Add a detailed description of your bug in the Description section. Best practice for descriptions include:
    • A summary of what the issue is
    • What you think is causing the issue
    • Reproducible code that can be run end to end without requiring an engineer to edit your code. Use {code} {code} around your code to make it appear in code format.
    • Any scripts or necessary documents. Add by dragging and dropping your files into the create issue dialogue box.

    You can be able to leave the rest of the ticket blank.

  4. When you are done with your ticket, simply click on the Create button at the bottom of the page.

    center

After you click Create, a pop up will appear on the right side of your screen with a link to your Jira ticket. It will have the form https://0xdata.atlassian.net/browse/SW-####. You can use this link to later edit your ticket.

Please note that your Jira ticket number along with its summary will appear in one of the Jira ticket slack channels, and anytime you update the ticket anyone associated with that ticket, whether as the assignee or a watcher will receive an email with your changes.

Have Questions?

We also respond to questions tagged with sparkling-water and h2o tags on the Stack Overflow.

Change Logs

Change logs are available at Change Logs.


More Repositories

1

h2ogpt

Private chat with local GPT with document, images, video, etc. 100% private, Apache 2.0. Supports oLLaMa, Mixtral, llama.cpp, and more. Demo: https://gpt.h2o.ai/ https://gpt-docs.h2o.ai/
Python
11,235
star
2

h2o-3

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
Jupyter Notebook
6,862
star
3

h2o-llmstudio

H2O LLM Studio - a framework and no-code GUI for fine-tuning LLMs. Documentation: https://docs.h2o.ai/h2o-llmstudio/
Python
3,991
star
4

wave

Realtime Web Apps and Dashboards for Python and R
Python
3,966
star
5

h2o-2

Please visit https://github.com/h2oai/h2o-3 for latest H2O
Java
2,222
star
6

datatable

A Python package for manipulating 2-dimensional tabular data structures
C++
1,807
star
7

h2o-tutorials

Tutorials and training material for the H2O Machine Learning Platform
Jupyter Notebook
1,457
star
8

mli-resources

H2O.ai Machine Learning Interpretability Resources
Jupyter Notebook
479
star
9

h2o4gpu

H2Oai GPU Edition
C++
455
star
10

h2o-meetups

Presentations from H2O meetups & conferences by the H2O.ai team
Jupyter Notebook
412
star
11

awesome-h2o

A curated list of research, applications and projects built using the H2O Machine Learning platform
353
star
12

db-benchmark

reproducible benchmark of database-like ops
R
299
star
13

pystacknet

Jupyter Notebook
286
star
14

deepwater

Deep Learning in H2O using Native GPU Backends
C++
285
star
15

h2o-wizardlm

Open-Source Implementation of WizardLM to turn documents into Q:A pairs for LLM fine-tuning
Python
242
star
16

driverlessai-recipes

Recipes for Driverless AI
Python
224
star
17

nitro

Create apps 10x quicker, without Javascript/HTML/CSS.
TypeScript
200
star
18

wave-apps

Sample AI Apps built with H2O Wave.
Python
144
star
19

h2o-flow

Web based interactive computing environment for H2O
CoffeeScript
131
star
20

tutorials

This is a repo for all the tutorials put out by H2O.ai. This includes learning paths for Driverless AI, H2O-3, Sparkling Water and more...
Jupyter Notebook
129
star
21

enterprise-h2ogpte

Client Code Examples, Use Cases and Benchmarks for Enterprise h2oGPTe RAG-Based GenAI Platform
Python
79
star
22

rsparkling

RSparkling: Use H2O Sparkling Water from R (Spark + R + Machine Learning)
R
64
star
23

steam

DEPRECATED Build, manage and deploy H2O's high-speed machine learning models.
Java
61
star
24

h2o-world-2014-training

training material
Java
47
star
25

h2o-sparkling

DEPRECATED! Use https://github.com/h2oai/sparkling-water repository! H2O and Spark interoperability based on Tachyon.
Scala
43
star
26

app-consumer-loan

HTML
41
star
27

h2o-kubeflow

Jsonnet
37
star
28

h2o-droplets

Templates for projects based on top of H2O.
Java
37
star
29

driverlessai-tutorials

H2OAI Driverless AI Code Samples and Tutorials
Jupyter Notebook
37
star
30

app-malicious-domains

Domain name classifier looking for good vs. possibly malicious providers
HTML
33
star
31

data-science-examples

A collection of data science examples implemented across a variety of languages and libraries.
CSS
33
star
32

xgboost-predictor

Java
32
star
33

wave-ml

Automatic Machine Learning (AutoML) for Wave Apps
Python
32
star
34

AITD

Jupyter Notebook
31
star
35

h2o-LLM-eval

Large-language Model Evaluation framework with Elo Leaderboard and A-B testing
Jupyter Notebook
28
star
36

Deep-Learning-with-h2o-in-R

Deep neural networks on over 50 classification problems from the UC Irvine Machine Learning Repository
R
23
star
37

sql-sidekick

Experiment on QnA tabular data using LLMs and SQL
Python
22
star
38

h2o.js

Node.js bindings to H2O, the open-source prediction engine for big data science.
CoffeeScript
21
star
39

perf

Performance Benchmarks
Jupyter Notebook
21
star
40

typesentry

Python 2.7 & 3.5+ runtime type-checker
Python
20
star
41

covid19-datasets

20
star
42

h2o-kubernetes

H2O Open Source Kubernetes operator and a command-line tool to ease deployment (and undeployment) of H2O open-source machine learning platform H2O-3 to Kubernetes.
Rust
20
star
43

mlops-dai-runtimes

Production ready templates for deploying Driverless AI (DAI) scorers. https://h2oai.github.io/dai-deployment-templates/
Java
17
star
44

genai-app-store-apps

GenAI apps from H2O made Wave
Python
16
star
45

qcon2015

Repository for SF QConf 2015 Workshop
Java
16
star
46

h2o3-sagemaker

Integrating H2O-3 AutoML with Amazon Sagemaker
Python
13
star
47

wave-image-styling-playground

A interactive playground to style and edit images, generate art and have fun.
Python
13
star
48

article-information-2019

Article for Special Edition of Information: Machine Learning with Python
Jupyter Notebook
13
star
49

social_ml

Python
12
star
50

challenge-wildfires

Starter kit for H2O.ai competition Challenge Wildfires.
Jupyter Notebook
11
star
51

h2o-jenkins-pipeline-lib

Library of different Jenkins pipeline building blocks.
Groovy
11
star
52

haic-tutorials

Jupyter Notebook
10
star
53

wave-h2o-automl

Wave App for H2O AutoML
Python
9
star
54

cvpr-multiearth-deforestation-segmentation

Jupyter Notebook
8
star
55

ht-catalog

Diverse collection of 100 Hydrogen Torch Use-Cases by different industries, data-types, and problem types
HTML
8
star
56

app-ask-craig

Ask Craig application
Scala
7
star
57

dai-deployment-examples

Examples for deploying Driverless AI (DAI) scorers.
Java
7
star
58

ml-security-audits

TeX
7
star
59

wave-big-data-visualizer

Python
6
star
60

xai_guidelines

Guidelines for the responsible use of explainable AI and machine learning
Jupyter Notebook
5
star
61

authn-py

Universal Token Provider
Python
5
star
62

h2o-scoring-service

Scoring service backend by model POJOs.
Java
5
star
63

app-news-classification

Scala
5
star
64

jdupes

H2O.ai fork of https://codeberg.org/jbruchon/jdupes
C
5
star
65

covid19-backtesting-publication

Jupyter Notebook
5
star
66

fluid

Rapid application development for a more... civilized age.
CoffeeScript
5
star
67

app-mojo-servlet

Example of putting a mojo zip file as a resource into a java servlet.
Java
5
star
68

cloud-discovery-py

H2O Cloud Discovery Client.
Python
4
star
69

jacocoHighlight

Java
4
star
70

h2o-automl-paper

H2O AutoML paper
R
4
star
71

docai-recipes

Jupyter Notebook
4
star
72

deepwater-nae

Python
3
star
73

h2oai-power-nae

Shell
3
star
74

nitro-matplotlib

Matplotlib plugin for H2O Nitro
Python
3
star
75

h2o-cloud

H2O Cloud code.
Jupyter Notebook
3
star
76

nitro-plotly

Plotly plugin for H2O Nitro
Python
3
star
77

h2o-rf1-bench

Python
3
star
78

residuals-vis

JavaScript
3
star
79

wave-r-data-table

This wave application is a R data.table tutorial and interactive learning environment developed using the wave library for R.
R
3
star
80

python-chat-ui

3
star
81

h2o_genai_training

Repository for H2O.ai's Generative AI Training
Jupyter Notebook
3
star
82

roc-chart

JavaScript
2
star
83

driverlessai-alt-containers

Shell
2
star
84

camelot

Modified version of https://github.com/camelot-dev/camelot
Python
2
star
85

nitro-bokeh

Bokeh plugin for H2O Nitro
Python
2
star
86

wave-amlb

Wave Dashboard for the OpenML AutoML Benchmark
Python
2
star
87

h2o-evals

Bring Your Own Evals
Python
2
star
88

pydart

Dart/Flutter <-> Python transpiler
Python
2
star
89

app-titanic

HTML
2
star
90

h2o3-xgboost-nae

Shell
2
star
91

residuals-vis-example-project

JavaScript
2
star
92

py-repo

Python package repository
HTML
2
star
93

dai-centos7-x86_64-nae

Dockerfile
1
star
94

correlation-graph

JavaScript
1
star
95

lightning

High performance, interactive statistical graphics engine for the web.
CoffeeScript
1
star
96

residuals-vis-data

JavaScript
1
star
97

2017-06-21-hackathon

Meetup Hackathon 06/21/2017
HTML
1
star
98

h2o-health

An initiate of H2O.ai to build AI apps to solve complex healthcare and life science problems
Makefile
1
star
99

h2o-google-bigquery

Python
1
star
100

dallas-tutorials

Temporary repository for fast git cloning during the h2o dallas event.
Jupyter Notebook
1
star