• This repository has been archived on 10/Feb/2021
  • Stars
    star
    136
  • Rank 267,670 (Top 6 %)
  • Language
    Python
  • License
    BSD 3-Clause "New...
  • Created almost 9 years ago
  • Updated almost 4 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A wrapper for libhdfs3 to interact with HDFS from Python

hdfs3

Build Status

This project is not undergoing development

Pyarrow's JNI hdfs interface is mature and stable. It also has fewer problems with configuration and various security settings, and does not require the complex build process of libhdfs3. Therefore, all users who have trouble with hdfs3 are recommended to try pyarrow.

Old README

hdfs3 is a lightweight Python wrapper for libhdfs3, a native C/C++ library to interact with the Hadoop File System (HDFS).

View the documentation for hdfs3.

More Repositories

1

dask

Parallel computing with task scheduling
Python
12,531
star
2

dask-tutorial

Dask tutorial
Jupyter Notebook
1,832
star
3

distributed

A distributed task scheduler for Dask
Python
1,576
star
4

dask-ml

Scalable Machine Learning with Dask
Python
898
star
5

dask-examples

Easy-to-run example notebooks for Dask
Jupyter Notebook
373
star
6

dask-kubernetes

Native Kubernetes integration for Dask
Python
311
star
7

dask-labextension

JupyterLab extension for Dask
TypeScript
311
star
8

dask-searchcv

dask-searchcv is now part of dask-ml: https://github.com/dask/dask-ml
Python
240
star
9

dask-jobqueue

Deploy Dask on job schedulers like PBS, SLURM, and SGE
Python
234
star
10

dask-docker

Docker images for dask
Jupyter Notebook
231
star
11

dask-image

Distributed image processing
Python
210
star
12

dask-xgboost

Python
162
star
13

cachey

Caching based on computation time and storage space
Python
137
star
14

dask-gateway

A multi-tenant server for securely deploying and managing Dask clusters.
Python
136
star
15

dask-cloudprovider

Cloud provider cluster managers for Dask. Supports AWS, Google Cloud Azure and more...
Python
134
star
16

dask-ec2

Start a cluster in EC2 for dask.distributed
Python
106
star
17

partd

Concurrent appendable key-value storage
Python
105
star
18

dask-tensorflow

Python
93
star
19

helm-chart

Helm charts for Dask
YAML
91
star
20

dask-expr

Python
86
star
21

dask-lightgbm

Python
79
star
22

dask-glm

Python
76
star
23

zict

Useful Mutable Mappings
Python
69
star
24

dask-yarn

Deploy dask on YARN clusters
Python
69
star
25

dask-gke

kubernetes setup to bootstrap distributed on google container engine
Python
67
star
26

old-dask-examples

Collection of dask example notebooks
Jupyter Notebook
57
star
27

knit

Deprecated, please use https://github.com/jcrist/skein or https://github.com/dask/dask-yarn instead
Python
53
star
28

dask-mpi

Deploy Dask using MPI4Py
Python
52
star
29

dask-stories

Python
40
star
30

dask-drmaa

Deploy Dask on DRMAA clusters
Python
40
star
31

dask-blog

Dask development blog
HTML
30
star
32

crick

Streaming and approximate algorithms. WIP, use at own risk.
Python
24
star
33

community

For general discussion and community planning. Discussion issues welcome.
20
star
34

mtprof

Thread-aware Python profiler hack
Python
17
star
35

dask-benchmarks

asv benchmarks for dask projects
Python
17
star
36

pandas-streaming

Python
16
star
37

dask-tutorial-infrastructure

Cluster for the Dask Tutorial.
Dockerfile
11
star
38

old-dask-yarn

Deprecated, please use https://github.com/jcrist/skein or https://github.com/dask/dask-yarn instead
Python
7
star
39

governance

The governance process and model for Dask
7
star
40

dask-sphinx-theme

Sphinx theme for Dask documentation
Python
6
star
41

dask-ml-benchmarks

Python
5
star
42

dask.github.io

Dask Website
HTML
5
star
43

scipy-tutorials-2018

5
star
44

design-docs

Experimental repo for proposals of future work
2
star
45

.github

2
star
46

dask-org

General dask resources that aren't code
Jupyter Notebook
2
star
47

marketing

Resources and guidelines for marketing Dask
Python
1
star
48

dask-gateway-helm-repo

Repository holding published dask-gateway helm charts
1
star
49

parquet-integration

Integration tests for various parquet readers and writers
Python
1
star