Discover a0x8o/data-platform-ai Open Source project by Alex (@a0x8o)

Scalable Batch and Stream Data Processing

flink

fbtftp is Facebook's implementation of a dynamic TFTP server framework

fbtftp

Unified programming model to create a data processing pipelines for batch and streaming models

beam

Caffe: a fast open framework for deep learning

caffe

Mixed Incremental Cross-Entropy REINFORCE ICLR 2016

mixer

Lua

fboss

Facebook Open Switching System Software for controlling network switches

Library for fast text representation and classification

fasttext

HTML

felix

Project Calico core repository

Bistro is a flexible distributed scheduler, a high-performance framework supporting multiple paradigms while retaining ease of configuration, management, and monitoring.

bistro

Compute Resource Usage Analysis and Monitoring of Container Clusters

heapster

hudi

Upserts And Incremental Processing on Big Data

The Kubernetes Package Manager

helm

darkforestgo

DarkForest, the Facebook Go engine

charts

Curated applications for Kubernetes using Helm charts with integrated Deployment Manager templates

A platform for developing AI systems as described in A Roadmap towards Machine Intelligence - http://arxiv.org/abs/1511.08130

commai-env

A Torch implementation of the object detection network from "A MultiPath Network for Object Detection" (https://arxiv.org/abs/1604.02135)

multipathnet

Lua

deepmask

Torch implementation of DeepMask and SharpMask

Lua

kubernetes

Production-Grade Container Scheduling and Management

kubernetes-cluster-federation

Kubernetes cluster federation tutorial

Schema-free SQL for Hadoop, NoSQL and Cloud Storage

drill

A Scientific Computing Framework for Luajit

torch

Jupyter Notebook

pysparnn

Approximate Nearest Neighbor Search for Sparse Data in Python

avro

airflow

horovod

Distributed training framework for TensorFlow, Keras, PyTorch, and MXNet.

hive

Apache Hive

Distributed SQL query engine for big data https://prestodb.io

presto

Protocol Buffers - Google's data interchange format

protobuf

Fast, compact, schema-less, binary serialization and deserialization oriented towards dynamic languages

sereal

kuryr

Container and Orchestration remote driver for OpenStack Neutron

Node.js client for the Kafka REST proxy

kafka-rest-node

JavaScript

kafka-examples

Applications, templates and code examples for Apache Kafka

Apache Druid (Incubating) - Column oriented distributed data store ideal for powering interactive applications

spark

Apache Spark

Scala

druid

Rokku project. This projects acts as a proxy on top of any S3 storage solution providing services like authentication, authorisation, short-term tokens and lineage.

rokku

Scala

kafka-ansible

Ansible playbooks for the Kafka

workspaces

Workspaces 2.0 demo

Apache Spark docker image

docker-spark

Message queue backbone for event handling

message-backbone

docker-xserver

Docker Image with Xserver, OpenBLAS and correct user settings

calcite

Apache Calcite

Kafka Connect connector for JDBC-compatible databases

kafka-connect-jdbc

Shared software among connectors that target distributed filesystems and cloud storage

kafka-connect-storage-common

Generic Data Ingestion & Dispersal Library for Hadoop

marmaray

Clone all of your Github repositories from the command line using Python

gitolly

parquet-mr

kafka-python

docker-torch-jupyter

Docker image for deep learning with Torch and Jupyter

kafka-connect-elasticsearch

Kafka Connect Elasticsearch connector

Deploy manifests for the single-container version of Netsil AOC

manifests

Apache Pinot - A realtime distributed OLAP datastore

pinot

Kafka Connect HDFS connector

kafka-connect-hdfs

Orchestrate microservice-based process flows

baker

Scala

docker-deeplearning

Information and scripts to run and develop Deep Learning Docker containers

Caffe2 is a lightweight, modular, and scalable deep learning framework

caffe2

Jupyter Notebook

clocks

Time, Clocks, and the Ordering of Events

Utilities and a small framework for building REST services with Jersey, Jackson, and Jetty.

kafka-rest-utils

Schema registry for Kafka

kafka-schema-registry

Confluent REST Proxy for Kafka

kafka-rest

Python interface to Hive and Presto.

pyhive

Efficient, reusable RNNs and LSTMs for torch

torch-nn

kafka-go

utils

xgboost

Scalable, portable, and distributed Gradient Boosting (GBDT, GBRT or GBM) library, for Python, R, Java, Scala, C++ and more. Runs on single host, Hadoop, Spark, Flink and DataFlow

Unified Resource Scheduler to co-schedule mixed types of workloads such as batch, stateless and stateful jobs in a single cluster for better resource utilization.

peloton

petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

ARX is a comprehensive open source data anonymization tool aiming to provide scalability and usability. It supports various anonymization techniques, methods for analyzing data quality and re-identification risks and it supports well-known privacy models, such as k-anonymity, l-diversity, t-closeness and differential privacy.

arx

Kafka Connect suite of connectors for Cloud storage (currently including Amazon S3)

kafka-connect-storage-cloud