• This repository has been archived on 20/Dec/2022
  • Stars
    star
    240
  • Rank 168,229 (Top 4 %)
  • Language
    Java
  • License
    Apache License 2.0
  • Created over 7 years ago
  • Updated over 5 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

This is archive of SparkRDMA project. The new repository with RDMA shuffle acceleration for Apache Spark is here: https://github.com/Nvidia/sparkucx

SparkRDMA ShuffleManager Plugin

SparkRDMA is a high performance ShuffleManager plugin for Apache Spark that uses RDMA (instead of TCP) when performing Shuffle data transfers in Spark jobs.

This open-source project is developed, maintained and supported by Mellanox Technologies.

Performance results

Terasort

TeraSort results

Running 320GB TeraSort workload with SparkRDMA is x2.63 faster than standard Spark (runtime in seconds)

Test environment:

7 Spark standalone workers on Azure "h16mr" VM instance, Intel Haswell E5-2667 V3,

224GB RAM, 2000GB SSD for temporary storage, Mellanox InfiniBand FDR (56Gb/s)

Also featured at the Spark+AI Summit 2018, please see more info on our session: https://databricks.com/session/accelerated-spark-on-azure-seamless-and-scalable-hardware-offloads-in-the-cloud

Pagerank

PageRank results

Running 19GB Pagerank with SparkRDMA is x2.01 faster than standard Spark (runtime in seconds)

Test environment:

5 Spark standalone workers, 2x Intel Xeon E5-2697 v3 @ 2.60GHz, 25 cores per Worker, 150GB RAM, non-flash storage (HDD)

Mellanox ConnectX-5 network adapter with 100GbE RoCE fabric, connected with a Mellanox Spectrum switch

Wiki pages

For more information on configuration, performance tuning and troubleshooting, please visit the SparkRDMA GitHub Wiki

Runtime requirements

  • Apache Spark 2.0.0/2.1.0/2.2.0/2.3.0/2.4.0
  • Java 8
  • An RDMA-supported network, e.g. RoCE or Infiniband

Installation

Obtain SparkRDMA and DiSNI binaries

Please use the "Releases" page to download pre-built binaries.
If you would like to build the project yourself, please refer to the "Build" section below.

The pre-built binaries are packed as an archive that contains the following files:

  • spark-rdma-3.1-for-spark-2.0.0-jar-with-dependencies.jar
  • spark-rdma-3.1-for-spark-2.1.0-jar-with-dependencies.jar
  • spark-rdma-3.1-for-spark-2.2.0-jar-with-dependencies.jar
  • spark-rdma-3.1-for-spark-2.3.0-jar-with-dependencies.jar
  • spark-rdma-3.1-for-spark-2.4.0-jar-with-dependencies.jar
  • libdisni.so

libdisni.so must be in java.library.path on every Spark Master and Worker (usually in /usr/lib)

Configuration

Provide Spark the location of the SparkRDMA plugin jars by using the extraClassPath option. For standalone mode this can be added to either spark-defaults.conf or any runtime configuration file. For client mode this must be added to spark-defaults.conf. For Spark 2.0.0 (Replace with 2.1.0, 2.2.0, 2.3.0, 2.4.0 according to your Spark version):

spark.driver.extraClassPath   /path/to/SparkRDMA/target/spark-rdma-3.1-for-spark-2.0.0-jar-with-dependencies.jar
spark.executor.extraClassPath /path/to/SparkRDMA/target/spark-rdma-3.1-for-spark-2.0.0-jar-with-dependencies.jar

Running

To enable the SparkRDMA Shuffle Manager plugin, add the following line to either spark-defaults.conf or any runtime configuration file:

spark.shuffle.manager   org.apache.spark.shuffle.rdma.RdmaShuffleManager

Build

Building the SparkRDMA plugin requires Apache Maven and Java 8

  1. Obtain a clone of SparkRDMA

  2. Build the plugin for your Spark version (either 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0), e.g. for Spark 2.0.0:

mvn -DskipTests clean package -Pspark-2.0.0
  1. Obtain a clone of DiSNI for building libdisni:
git clone https://github.com/zrlio/disni.git
cd disni
git checkout tags/v1.7 -b v1.7
  1. Compile and install only libdisni (the jars are already included in the SparkRDMA plugin):
cd libdisni
autoprepare.sh
./configure --with-jdk=/path/to/java8/jdk
make
make install

Community discussions and support

For any questions, issues or suggestions, please use our Google group: https://groups.google.com/forum/#!forum/sparkrdma

Contributions

Any PR submissions are welcome

More Repositories

1

libvma

Linux user space library for network socket acceleration based on RDMA compatible network adaptors
C++
581
star
2

sockperf

Network Benchmarking Utility
C++
567
star
3

nv_peer_memory

C
234
star
4

network-operator

Mellanox Network Operator
Go
207
star
5

k8s-rdma-shared-dev-plugin

Go
188
star
6

mlxsw

C
167
star
7

nccl-rdma-sharp-plugins

RDMA and SHARP plugins for nccl library
C
157
star
8

mstflint

Mstflint - an open source version of MFT (Mellanox Firmware Tools)
C
114
star
9

k8s-rdma-sriov-dev-plugin

Kubernetes Rdma SRIOV device plugin
Go
110
star
10

gpu_direct_rdma_access

example code for using DC QP for providing RDMA READ and WRITE operations to remote GPU memory
C
99
star
11

mlnx-tools

Mellanox userland tools and scripts
Python
95
star
12

docker-sriov-plugin

Docker networking plugin for SRIOV and passthrough interfaces
Go
76
star
13

rdmamap

RDMA library for mapping associate netdevice and character devices
Go
58
star
14

ib-kubernetes

Go
57
star
15

libxlio

C++
41
star
16

ofed-docker

Shell
41
star
17

linux-sysinfo-snapshot

Linux Sysinfo Snapshot
Python
39
star
18

scalablefunctions

All about Scalable functions
39
star
19

SAI-Implementation

This repository contains SAI implementation for Mellanox hardware.
C
37
star
20

SAI-P4-BM

C++
36
star
21

SwitchRouterSDK-interfaces

C
32
star
22

mkt

Mellanox Kernel developers Toolset (MKT)
Python
25
star
23

mlx_steering_dump

Mellanox Steering Dump Tool for SWS and HWS acceleration
Python
24
star
24

ovs-tests

A collection of tests for the Open vSwitch HW offload.
Shell
23
star
25

R4H

RDMA for HDFS
Java
23
star
26

bfb-build

BFB (BlueField boot stream and OS installer) build environment
Shell
22
star
27

ufm_sdk_3.0

Python
19
star
28

ibdump

C
19
star
29

scapy-ui

Scapy UI - Web based scapy tools
Python
18
star
30

DCTrafficGen

Data Center Traffic Generator Library
C++
17
star
31

rshim-user-space

Linux based user-space RSHIM driver for the Mellanox BlueField SoC
C
17
star
32

vnf_acceleration_example

C
16
star
33

nvidia-k8s-ipam

IPAM plugin for kubernetes
Go
14
star
34

hw_offload_api_examples

Examples of usage for Mellanox HW offloads
C
14
star
35

pcx

Persistent Collectives X- A collective communication library for high performance, low cost persistent collectives over RDMA devices.
C++
13
star
36

hw-mgmt

Shell
13
star
37

rshim

BlueField RSHIM driver
C
12
star
38

rdma_fc

Demonstration of flow control over RDMA fabric
C
11
star
39

ngc_multinode_perf

Performance tests for multinode NGC.Ready certification
Shell
11
star
40

ipoib-cni

IP Over Infiniband (IPoIB) CNI Plugin
Go
11
star
41

pka

Mellanox BlueField PKA support
C
11
star
42

UDA

Unstructured Data Accelerator (RDMA) for Hadoop MapReduce
C++
10
star
43

mlxdevm-go

mlxdevm library for for device management in go language
Go
10
star
44

bfscripts

Collection of scripts used for BlueField SoC system management.
Shell
10
star
45

k8s-images

Dockerfile
10
star
46

devx

Objective-C
9
star
47

MT.ComB

Multi-Threaded (MT) Communication Benchmark
C
8
star
48

container_tools

Few useful container orchestration, deployment tools when using RDMA
Go
8
star
49

kubernetes-ci

CI for Kubernetes with Mellanox features
Shell
8
star
50

libpsample

C
8
star
51

config-tools

Mellanox Configuration tool for Linux Host
Shell
7
star
52

tls-af_ktls_tool

C
7
star
53

tls-offload

C
7
star
54

OVS

C
7
star
55

EC

!!! NOTICE: DEPRECATED !!! Java Erasure Coding NIC Offload library. For the C level EC offloads, use MLNX_OFED libraries and documentation.
C
6
star
56

TFDeploy

TensorFlow deploy script to easily run on multiple servers
Python
6
star
57

NVMEoF-P2P

A fork of the Linux kernel for NVMEoF target driver using PCI P2P capabilities for full I/O path offloading.
C
6
star
58

napalm

Network Automation and Programmability Abstraction Layer with Multivendor support
Python
5
star
59

containerized-ovs-forwarder

Python
5
star
60

bluefield-linux

Linux kernel to support Mellanox BlueField SoCs
C
5
star
61

kmtracker

Linux Kernel memory tracker
Go
5
star
62

bf-release

BlueField release files, configuration files and post-installation steps
Python
5
star
63

mofed_dockerfiles

MOFED Docker files
Roff
5
star
64

docker-nmos-cpp

Shell
4
star
65

wjh-linux

Python
4
star
66

ALVS

C
4
star
67

Switch-SDK-drivers

Switch SDK Driver
C
4
star
68

container_scripts

Some container scripts
Shell
4
star
69

ipmb-host

IPMB driver to send requests from the BlueField to the BMC on CentOS
C
4
star
70

mlnx_lib

C
4
star
71

nic-configuration-operator

Nvidia Networking NIC Configuration Operator For Kubernetes
Go
4
star
72

mellanox-netdev-stdlib-mlnxos

MLNX_OS specific Provider code for "netdev-stdlib". Netdev provides a set of network resource abstractions for automating network device configuration using Puppet
Ruby
4
star
73

libmlxdevm

Mellanox device management C library
C
3
star
74

virtio-emulation

C
3
star
75

dpdk-mlx4

DPDK.org tree with enhanced librte_pmd_mlx4
Objective-C
3
star
76

sai_p4_compiler

C++
3
star
77

mlnx-project-config

Python
3
star
78

ATC

C
3
star
79

regex

C
3
star
80

network-operator-docs

NVIDIA Network Operator documentation sources
PowerShell
3
star
81

nic-kernel

Nvidia NBU integration kernel
C
3
star
82

DPDK-18.11-for-Ubuntu-18.04

C
3
star
83

nagios4mlnxos

Nagios Plugin for Mellanox's Switches
Perl
3
star
84

meta-bluefield

Shell
3
star
85

NNT-Linux-driver

NNT Linux driver for MFT & MSTFLINT packages
C
3
star
86

OpenAI.recipe

Recommended configuration for large-scale setup - OpenAI
2
star
87

Kubespray-role-for-RDMA-shared-DP

2
star
88

ci-demo

Groovy
2
star
89

libdpcp

C++
2
star
90

doca-driver-build

Shell
2
star
91

ceilometer_sriov_counters

Plugin for Ceilometer SRIOV traffic counters
Python
2
star
92

mlnx-openstack

Puppet manifests for deploying Mellanox OpenStack plugins
Puppet
2
star
93

QAT_Engine

C
2
star
94

iproute2

2
star
95

nic-feature-discovery

NVIDIA NIC feature discovery
Go
2
star
96

eswitchd

Python
2
star
97

mlx-strongswan

Mellanox version of strongswan cloned from strongswan-5.9.0.tar.gz
C
2
star
98

nginx_automation

This is simple Python automation for Nginx - VMA related activity
Python
2
star
99

ipsec-offload

2
star
100

dpdk-utest

Rust
2
star