• Stars
    star
    163
  • Rank 231,141 (Top 5 %)
  • Language
    R
  • License
    GNU General Publi...
  • Created over 10 years ago
  • Updated almost 9 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

DistributedR

Distributed R is a scalable high-performance platform for the R language. It enables and accelerates large scale machine learning, statistical analysis, and graph processing.

The Distributed R platform exposes data structures, such as distributed arrays, to store data across a cluster. Arrays act as a single abstraction to efficiently express both machine learning algorithms, which primarily use matrix operations, and graph algorithms, which manipulate the graphā€™s adjacency matrix. In addition to distributed arrays, the platform also provides distributed data frames, lists and loops.

Using Distributed R constructs, data can be loaded in parallel from any data source. Distributed R also provides a parallel data loader from the Vertica database. Please see vRODBC repository.

Installing from binaries

Distributed R is delivered in a single, easy-to-install tar file. The installation tool "distributedR_install" installs the platform and all parallel algorithm R packages. You can register and get the tar file here.

You can also get a Virtual Machine with everything installed here.

Installing from source

  1. Install dependencies:
  • On Ubuntu:

      $ sudo apt-get install -y make gcc g++ libxml2-dev rsync bison byacc flex
    
  • On CentOS:

      $ sudo yum install -y make gcc gcc-c++ libxml2-devel rsync bison byacc flex
    
  1. Install R:
  • On Ubuntu:

      $ echo "deb http://cran.r-project.org//bin/linux/ubuntu trusty/" | sudo tee /etc/apt/sources.list.d/r.list
      $ sudo apt-get update
      $ sudo apt-get install -y --force-yes r-base-core
    
  • On CentOS:

      $ curl -O http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
      $ sudo rpm -i epel-release-latest-7.noarch.rpm
      $ sudo yum update
      $ sudo yum install R R-devel
    
  1. Install R dependencies:

     $ sudo R  # to install globally
     R> install.packages(c('Rcpp','RInside','XML','randomForest','data.table'))
    
  2. Compile and install Distributed R:

     $ R CMD INSTALL platform/executor
     $ R CMD INSTALL platform/master
    
  3. Or directly from the R console:

     R> devtools::install_github('vertica/DistributedR',subdir='platform/executor')
     R> devtools::install_github('vertica/DistributedR',subdir='platform/master')
    
  4. Open R and run an example:

     library(distributedR)
     distributedR_start()  # start DR
     distributedR_status()
    
     B <- darray(dim=c(9,9), blocks=c(3,3), sparse=FALSE) # create a darray
     foreach(i, 1:npartitions(B),
       init<-function(b = splits(B,i), index=i) {
       b <- matrix(index, nrow=nrow(b), ncol=ncol(b))
       update(b)
     })  # initialize it
    
     getpartition(B) # collect darray data
    
     distributedR_shutdown() # stop DR
    

How to Contribute

You can help us in different ways:

  1. Reporting issues.
  2. Contributing code and sending a Pull Request.

In order to contribute the code base of this project, you must agree to the Developer Certificate of Origin (DCO) 1.1 for this project under GPLv2+:

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I have the 
    right to submit it under the open source license indicated in the file; or
(b) The contribution is based upon previous work that, to the best of my 
    knowledge, is covered under an appropriate open source license and I 
    have the right under that license to submit that work with modifications, 
    whether created in whole or in part by me, under the same open source 
    license (unless I am permitted to submit under a different license), 
    as indicated in the file; or
(c) The contribution was provided directly to me by some other person who 
    certified (a), (b) or (c) and I have not modified it.
(d) I understand and agree that this project and the contribution are public and
    that a record of the contribution (including all personal information I submit 
    with it, including my sign-off) is maintained indefinitely and may be 
    redistributed consistent with this project or the open source license(s) involved.

To indicate acceptance of the DCO you need to add a Signed-off-by line to every commit. E.g.:

Signed-off-by: John Doe <[email protected]>

To automatically add that line use the -s switch when running git commit:

$ git commit -s

More Repositories

1

vertica-python

Official native Python client for the Vertica Analytics Database.
Python
379
star
2

VerticaPy

VerticaPy is a Python library that exposes sci-kit like functionality to conduct data science projects on data stored in Vertica, thus taking advantage Verticaā€™s speed and built-in analytics and machine learning capabilities.
Python
219
star
3

Vertica-Extension-Packages

User Defined Extensions (UDX) to the Vertica Analytic Database
HTML
119
star
4

ddR

Standard API for Distributed Data Structures in R
R
118
star
5

vertica-sql-go

Official native Go client for the Vertica Analytics Database.
Go
61
star
6

Graph-Analytics----Triangle-Counting

Use Big data tools such as Vertica, Hadoop and PIG to count triangles in a graph. Experimentally compare their performance.
Java
49
star
7

vertica-kubernetes

Operator, container and Helm chart to deploy Vertica in Kubernetes
Go
44
star
8

Vertica-Hadoop-Connector

Vertica Hadoop Connector
Java
38
star
9

awesome-vertica

A curated list of awesome Vertica libraries, tools and resources
29
star
10

dbt-vertica

Official dbt adapter for Vertica
Python
25
star
11

Machine-Learning-Examples

Vertica Machine Learning examples and example data.
Python
24
star
12

vertica.dplyr

R
22
star
13

dblink

A Vertica User Defined Transform function to interact with other databases via ODBC
C++
20
star
14

spark-connector

This component acts as a bridge between Spark and Vertica, allowing the user to either retrieve data from Vertica for processing in Spark, or store processed data from Spark into Vertica.
Scala
20
star
15

VerticaPyLab

Use Docker to install a simple demo Vertica.
Jupyter Notebook
19
star
16

vertica-containers

Docker containers for building and testing Vertica extensions
Shell
14
star
17

UDx-Examples

Vertica User Defined Extension (UDx) examples.
C++
12
star
18

vertica-nodejs

Official native node.js client for the Vertica Analytics Database.
JavaScript
12
star
19

vertica-grafana-datasource-.2.0

This project defines the official Vertica datasource plugin for Grafana.
JavaScript
12
star
20

ODBC-Loader

C++
11
star
21

PSTL

Parallel Streaming Transformation Loader
Java
9
star
22

vertica-prometheus-exporter

Official open source Vertica Prometheus Exporter
Go
8
star
23

Vertica-Geospatial

Vertica-Geospatial Examples
HTML
7
star
24

r-dataconnector

Distributed Data Connector R package
C++
7
star
25

Social-Media-Connector

Java
5
star
26

client-application-examples

Examples of how to write a client application to connect to vertica
C#
4
star
27

Vertica-AMI-Packer-Templates

Packer templates and provision scripts for creating Vertica AMIs
Shell
4
star
28

aws-lambda-vertica-loader

Automatic HP Vertica Database Loader for AWS S3
JavaScript
4
star
29

hackathon

A getting started kit for Hackathons using Vertica!
Python
4
star
30

vertica-grafana-datasource

Official Vertica datasource plugin for Grafana.
TypeScript
3
star
31

grafana-dashboards

Dashboards that can be imported into Grafana that use the Prometheus metrics Vertica exports
Python
2
star
32

vertica.github.io

Vertica Open Source
HTML
2
star
33

FlexTable

HPE Vertica Flex Table source code of VMap APIs and parsers
C++
2
star
34

vertica-sqlalchemy-dialect

Official Vertica SQLAlchemy dialect to connect Vertica and SQLAlchemy applications.
Python
2
star
35

vcluster

Go library to administer a vertica cluster
Go
2
star
36

charts

1
star
37

vertica-airbyte

Python
1
star
38

DistributedR-demos

CSS
1
star
39

data2ddl

command line tool guessing a table's matching CREATE TABLE statement with optimal data types from a CSV file
C
1
star
40

integrators-guide

A community repo for users that integrate with Vertica with provisioning, monitoring, management, storage, clients, loaders, and other tools.
SCSS
1
star
41

vertica-testenv

Docker-based testing environment for Vertica.
Shell
1
star
42

vertica-highcharts

Python
1
star