• Stars
    star
    299
  • Rank 134,506 (Top 3 %)
  • Language
    R
  • License
    Mozilla Public Li...
  • Created almost 8 years ago
  • Updated 11 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

reproducible benchmark of database-like ops

Repository for reproducible benchmarking of database-like operations in single-node environment.
Benchmark report is available at h2oai.github.io/db-benchmark.
We focused mainly on portability and reproducibility. Benchmark is routinely re-run to present up-to-date timings. Most of solutions used are automatically upgraded to their stable or development versions.
This benchmark is meant to compare scalability both in data volume and data complexity.
Contribution and feedback are very welcome!

Tasks

  • groupby
  • join
  • groupby2014

Solutions

More solutions has been proposed. Status of those can be tracked in issues tracker of our project repository by using new solution label.

Reproduce

Batch benchmark run

  • edit path.env and set julia and java paths
  • if solution uses python create new virtualenv as $solution/py-$solution, example for pandas use virtualenv pandas/py-pandas --python=/usr/bin/python3.6
  • install every solution, follow $solution/setup-$solution.sh scripts
  • edit run.conf to define solutions and tasks to benchmark
  • generate data, for groupby use Rscript _data/groupby-datagen.R 1e7 1e2 0 0 to create G1_1e7_1e2_0_0.csv, re-save to binary format where needed (see below), create data directory and keep all data files there
  • edit _control/data.csv to define data sizes to benchmark using active flag
  • ensure SWAP is disabled and ClickHouse server is not yet running
  • start benchmark with ./run.sh

Single solution benchmark

  • install solution software
    • for python we recommend to use virtualenv for better isolation
    • for R ensure that library is installed in a solution subdirectory, so that library("dplyr", lib.loc="./dplyr/r-dplyr") or library("data.table", lib.loc="./datatable/r-datatable") works
    • note that some solutions may require another to be installed to speed-up csv data load, for example, dplyr requires data.table and similarly pandas requires (py)datatable
  • generate data using _data/*-datagen.R scripts, for example, Rscript _data/groupby-datagen.R 1e7 1e2 0 0 creates G1_1e7_1e2_0_0.csv, put data files in data directory
  • run benchmark for a single solution using ./_launcher/solution.R --solution=data.table --task=groupby --nrow=1e7
  • run other data cases by passing extra parameters --k=1e2 --na=0 --sort=0
  • use --quiet=true to suppress script's output and print timings only, using --print=question,run,time_sec specify columns to be printed to console, to print all use --print=*
  • use --out=time.csv to write timings to a file rather than console

Running script interactively

  • install software in expected location, details above
  • ensure data name to be used in env var below is present in ./data dir
  • source python virtual environment if needed
  • call SRC_DATANAME=G1_1e7_1e2_0_0 R, if desired replace R with python or julia
  • proceed pasting code from benchmark script

Extra care needed

  • cudf uses conda instead of virtualenv

Example environment

Acknowledgment

Timings for some solutions might be missing for particular data sizes or questions. Some functions are not yet implemented in all solutions so we were unable to answer all questions in all solutions. Some solutions might also run out of memory when running benchmark script which results the process to be killed by OS. Lastly we also added timeout for single benchmark script to run, once timeout value is reached script is terminated. Please check exceptions label in our repository for a list of issues/defects in solutions, that makes us unable to provide all timings. There is also no documentation label that lists issues that are blocked by missing documentation in solutions we are benchmarking.

More Repositories

1

h2ogpt

Private chat with local GPT with document, images, video, etc. 100% private, Apache 2.0. Supports oLLaMa, Mixtral, llama.cpp, and more. Demo: https://gpt.h2o.ai/ https://codellama.h2o.ai/
Python
10,513
star
2

h2o-3

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
Jupyter Notebook
6,658
star
3

wave

Realtime Web Apps and Dashboards for Python and R
Python
3,820
star
4

h2o-llmstudio

H2O LLM Studio - a framework and no-code GUI for fine-tuning LLMs. Documentation: https://h2oai.github.io/h2o-llmstudio/
Python
3,608
star
5

h2o-2

Please visit https://github.com/h2oai/h2o-3 for latest H2O
Java
2,222
star
6

datatable

A Python package for manipulating 2-dimensional tabular data structures
C++
1,790
star
7

h2o-tutorials

Tutorials and training material for the H2O Machine Learning Platform
Jupyter Notebook
1,457
star
8

sparkling-water

Sparkling Water provides H2O functionality inside Spark cluster
Scala
954
star
9

mli-resources

H2O.ai Machine Learning Interpretability Resources
Jupyter Notebook
478
star
10

h2o4gpu

H2Oai GPU Edition
C++
453
star
11

h2o-meetups

Presentations from H2O meetups & conferences by the H2O.ai team
Jupyter Notebook
412
star
12

awesome-h2o

A curated list of research, applications and projects built using the H2O Machine Learning platform
353
star
13

deepwater

Deep Learning in H2O using Native GPU Backends
C++
285
star
14

pystacknet

Jupyter Notebook
284
star
15

h2o-wizardlm

Open-Source Implementation of WizardLM to turn documents into Q:A pairs for LLM fine-tuning
Python
242
star
16

driverlessai-recipes

Recipes for Driverless AI
Python
224
star
17

nitro

Create apps 10x quicker, without Javascript/HTML/CSS.
TypeScript
198
star
18

wave-apps

Sample AI Apps built with H2O Wave.
Python
139
star
19

h2o-flow

Web based interactive computing environment for H2O
CoffeeScript
131
star
20

tutorials

This is a repo for all the tutorials put out by H2O.ai. This includes learning paths for Driverless AI, H2O-3, Sparkling Water and more...
Jupyter Notebook
127
star
21

rsparkling

RSparkling: Use H2O Sparkling Water from R (Spark + R + Machine Learning)
R
64
star
22

steam

DEPRECATED Build, manage and deploy H2O's high-speed machine learning models.
Java
60
star
23

h2o-world-2014-training

training material
Java
47
star
24

h2o-sparkling

DEPRECATED! Use https://github.com/h2oai/sparkling-water repository! H2O and Spark interoperability based on Tachyon.
Scala
43
star
25

app-consumer-loan

HTML
41
star
26

h2o-kubeflow

Jsonnet
37
star
27

h2o-droplets

Templates for projects based on top of H2O.
Java
37
star
28

driverlessai-tutorials

H2OAI Driverless AI Code Samples and Tutorials
Jupyter Notebook
37
star
29

app-malicious-domains

Domain name classifier looking for good vs. possibly malicious providers
HTML
33
star
30

data-science-examples

A collection of data science examples implemented across a variety of languages and libraries.
CSS
33
star
31

xgboost-predictor

Java
32
star
32

wave-ml

Automatic Machine Learning (AutoML) for Wave Apps
Python
32
star
33

h2o-LLM-eval

Large-language Model Evaluation framework with Elo Leaderboard and A-B testing
Jupyter Notebook
28
star
34

Deep-Learning-with-h2o-in-R

Deep neural networks on over 50 classification problems from the UC Irvine Machine Learning Repository
R
23
star
35

h2o.js

Node.js bindings to H2O, the open-source prediction engine for big data science.
CoffeeScript
21
star
36

perf

Performance Benchmarks
Jupyter Notebook
21
star
37

typesentry

Python 2.7 & 3.5+ runtime type-checker
Python
20
star
38

covid19-datasets

20
star
39

h2o-kubernetes

H2O Open Source Kubernetes operator and a command-line tool to ease deployment (and undeployment) of H2O open-source machine learning platform H2O-3 to Kubernetes.
Rust
20
star
40

sql-sidekick

Experiment on QnA tabular data using LLMs and SQL
Python
18
star
41

AITD

Jupyter Notebook
17
star
42

dai-deployment-templates

Production ready templates for deploying Driverless AI (DAI) scorers. https://h2oai.github.io/dai-deployment-templates/
Java
17
star
43

qcon2015

Repository for SF QConf 2015 Workshop
Java
16
star
44

h2o3-sagemaker

Integrating H2O-3 AutoML with Amazon Sagemaker
Python
13
star
45

wave-image-styling-playground

A interactive playground to style and edit images, generate art and have fun.
Python
13
star
46

article-information-2019

Article for Special Edition of Information: Machine Learning with Python
Jupyter Notebook
13
star
47

genai-app-store-apps

GenAI apps from H2O made Wave
Python
12
star
48

social_ml

Python
12
star
49

challenge-wildfires

Starter kit for H2O.ai competition Challenge Wildfires.
Jupyter Notebook
11
star
50

h2o-jenkins-pipeline-lib

Library of different Jenkins pipeline building blocks.
Groovy
11
star
51

haic-tutorials

Jupyter Notebook
10
star
52

wave-h2o-automl

Wave App for H2O AutoML
Python
9
star
53

cvpr-multiearth-deforestation-segmentation

Jupyter Notebook
8
star
54

app-ask-craig

Ask Craig application
Scala
7
star
55

dai-deployment-examples

Examples for deploying Driverless AI (DAI) scorers.
Java
7
star
56

ml-security-audits

TeX
7
star
57

ht-catalog

Diverse collection of 100 Hydrogen Torch Use-Cases by different industries, data-types, and problem types
HTML
7
star
58

wave-big-data-visualizer

Python
6
star
59

xai_guidelines

Guidelines for the responsible use of explainable AI and machine learning
Jupyter Notebook
5
star
60

authn-py

Universal Token Provider
Python
5
star
61

fluid

Rapid application development for a more... civilized age.
CoffeeScript
5
star
62

h2o-scoring-service

Scoring service backend by model POJOs.
Java
5
star
63

app-news-classification

Scala
5
star
64

covid19-backtesting-publication

Jupyter Notebook
5
star
65

app-mojo-servlet

Example of putting a mojo zip file as a resource into a java servlet.
Java
5
star
66

cloud-discovery-py

H2O Cloud Discovery Client.
Python
4
star
67

jacocoHighlight

Java
4
star
68

h2o-automl-paper

H2O AutoML paper
R
4
star
69

docai-recipes

Jupyter Notebook
4
star
70

deepwater-nae

Python
3
star
71

h2oai-power-nae

Shell
3
star
72

nitro-matplotlib

Matplotlib plugin for H2O Nitro
Python
3
star
73

h2o-cloud

H2O Cloud code.
Jupyter Notebook
3
star
74

h2o-rf1-bench

Python
3
star
75

nitro-plotly

Plotly plugin for H2O Nitro
Python
3
star
76

residuals-vis

JavaScript
3
star
77

python-chat-ui

3
star
78

driverlessai-alt-containers

Shell
2
star
79

camelot

Modified version of https://github.com/camelot-dev/camelot
Python
2
star
80

nitro-bokeh

Bokeh plugin for H2O Nitro
Python
2
star
81

wave-amlb

Wave Dashboard for the OpenML AutoML Benchmark
Python
2
star
82

app-titanic

HTML
2
star
83

py-repo

Python package repository
HTML
2
star
84

roc-chart

JavaScript
2
star
85

h2o3-xgboost-nae

Shell
2
star
86

residuals-vis-example-project

JavaScript
2
star
87

wave-r-data-table

This wave application is a R data.table tutorial and interactive learning environment developed using the wave library for R.
R
2
star
88

h2o_genai_training

Repository for H2O.ai's Generative AI Training
Jupyter Notebook
2
star
89

dai-centos7-x86_64-nae

Dockerfile
1
star
90

correlation-graph

JavaScript
1
star
91

residuals-vis-data

JavaScript
1
star
92

pydart

Dart/Flutter <-> Python transpiler
Python
1
star
93

2017-06-21-hackathon

Meetup Hackathon 06/21/2017
HTML
1
star
94

h2o-health

An initiate of H2O.ai to build AI apps to solve complex healthcare and life science problems
Makefile
1
star
95

lightning

High performance, interactive statistical graphics engine for the web.
CoffeeScript
1
star
96

h2o-google-bigquery

Python
1
star
97

fiction

Yet another markdown-to-documentation generator
CoffeeScript
1
star
98

dallas-tutorials

Temporary repository for fast git cloning during the h2o dallas event.
Jupyter Notebook
1
star
99

pydata2016-h2o-loganalysis

Log Analysis Use Case for PyData2016
Java
1
star
100

aggregator-zoom

JavaScript
1
star