< Python

Share Share Share

Awesome Python Distributed Computing

• dask 12,531

  

  | updated 28 days ago | BSD 3-Clause "New...

  Parallel computing with task scheduling

• faust 6,683

  

  | updated almost 1 year ago | Other

  Python Stream Processing

• luigi 17,796

  

  | updated about 2 months ago | Apache License 2.0

  Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

• mrjob 2,615

  

  | updated over 1 year ago | Other

  Run MapReduce jobs on Hadoop or Amazon Web Services

• ray 33,272

  

  | updated about 2 months ago | Apache License 2.0

  Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

• streamparse 1,487

  

  | updated over 1 year ago | Apache License 2.0

  Run Python in Apache Storm topologies. Pythonic API, CLI tooling, and a topology DSL.