Discover oap-project/pmem-shuffle Open Source project

This repository has been archived on 18/Sep/2023
Stars
14
Rank 1,391,724 (Top 29 %)
Language
C++
License
Apache License 2.0
Created over 3 years ago
Updated 8 months ago

oap-project

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Spark* Shuffle plugin for support shuffling through remote persistent memory over fabrics, which leverages the RDMA network and remote persistent memory (for read) to provide extremely high performance and low latency shuffle solutions for Spark*.

gluten

Gluten: Plugin to Double SparkSQL's Performance

Scala

920

raydp

RayDP provides simple APIs for running Spark on Ray and integrating Spark with AI libraries.

Python

284

gazelle_plugin

Native SQL Engine plugin for Spark SQL with vectorized SIMD optimizations.

Scala

256

Gluten-Trino

Gluten: Plugin to Boost Trino's Performance

Java

sql-ds-cache

Spark* plug-in for accelerating Spark* SQL performance by using cache and index at SQL data source layer.

Scala

oap-mllib

Optimized Spark package to accelerate machine learning algorithms in Apache Spark MLlib.

Scala

remote-shuffle

Spark* shuffle plugin for support shuffling data through a remote Hadoop-compatible file system, as opposed to vanilla Spark's local-disks.

Scala

oap-tools

Tools for building, packaging, and OAP public cloud integrations such as AWS EMR, Google Dataproc and K8S.

Jupyter Notebook

pmem-spill

Spark plug-in package for accelerating Spark runtime spill functions using PMem such as RDD cache PMem extension.

Scala

arrow-data-source

Spark DataSouce plugin for reading files from various formats like Parquet into Arrow compatible columnar vectors.

Scala

cloudtik

Cloud scaling platform for distributed analytics and AI on Spark

Python

pmem-common

Common library for accessing PMEM native library functions including memkind, vmemcache and so on.

Java

text2sql-gluten

Python

recdp

Python

oap-project/pmem-shuffle

oap-project

Reviews

Repository Details

More Repositories