• Stars
    star
    205
  • Rank 191,264 (Top 4 %)
  • Language
  • Created over 7 years ago
  • Updated about 5 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Resources for Data Science Process management

Data Science Process Management

This repository compiles resources for data science process management.

It is challenging to establish good process around data science tasks. Many data science tasks end up with one-off solutions that cannot be used in other tasks. Data science tasks don't fit well to either software engineering or product engineering process. Also different data science tasks require different processes.

Yet, it is crucial to have well defined process, because, without it, organizations cannot accumulate knowledge internally, and have to rely on individuals to perform critical tasks. People come and go, while process sticks around.

While there is no good standard so far, there have been active conversations around the issue.

By compiling those, we hope that it helps us get closer to the best practices in data science process management.

Understand Data Science Tasks

Data Analytics Process

Data Science Product Development Process

Use Cases

Articles

Videos

Presentation Slides

Data Science Knowledge Sharing Process

Tools

Data Analytics Process Templates

Knowledge Sharing

Data/Modeling/Deployment Pipelines

  • Pachyderm - Version control of data and language-agnostic data pipelines
  • Kubernetes - Automating deployment, scaling, and management of containerized applications
  • AWS Data Pipeline by Amazon - Data pipeline management on AWS
  • Azure Data Factory by Microsoft - Data pipeline management on Azure
  • AirFlow by AirBnB
  • Oryx - Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning
  • Luigi by Spotify
  • MLflow by Databricks - an open source platform for managing the end-to-end machine learning lifecycle.
  • Kubeflow - An open-source model training, deployment, and serving tool leveraging the kubernetes ecosystem.
  • Polyaxon - Another open-source model training, deployment, and serving tool.
  • Algorithmia A SaaS solution for managing the complete ML workflow cycle.
  • Lore Framework to structure ML projects and pipelines

Model Management

Open source:

Experiment Management