• Stars
    star
    74
  • Rank 393,519 (Top 9 %)
  • Language
    R
  • License
    Other
  • Created almost 2 years ago
  • Updated 3 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A distributed worker launcher

crew: a distributed worker launcher framework

CRAN status check codecov lint

In computationally demanding analysis projects, statisticians and data scientists asynchronously deploy long-running tasks to distributed systems, ranging from traditional clusters to cloud services. The NNG-powered mirai R package is a sleek and sophisticated scheduler that efficiently processes these intense workloads. The crew package extends mirai with a unifying interface for third-party worker launchers. Inspiration also comes from packages future, rrq, clustermq, and batchtools.

📣 Request for community contributions 📣

The launcher plugin framework aims to extend crew to modern platforms and services like Google Cloud Run, Kubernetes, and beyond. At the time of writing, plugins for cloud computing do not yet exist. So if you have access to these services and know how to use them, please consider contributing a package with plugins of your own. The tutorial at https://wlandau.github.io/crew/articles/plugins.html explains how. The maintainer of crew would love to work with you!

Installation

Type Source Command
Release CRAN install.packages("crew")
Development GitHub remotes::install_github("wlandau/crew")
Development R-universe install.packages("crew", repos = "https://wlandau.r-universe.dev")

Documentation

The documentation website at https://wlandau.github.io/crew/ includes a function reference and tutorial vignettes linked below.

  1. Introduction to crew
  2. Controller groups
  3. Integration with Shiny
  4. How to write launcher plugins
  5. Known risks of crew

Risks

The crew package has unavoidable risks, and the user is responsible for safety, security, and computational resources. Please read the software license and the vignette about specific known risks.

Similar work

  • mirai: a powerful R framework for asynchronous tasks built on NNG. The purpose of crew is to extend mirai to different computing platforms for distributed workers.
  • rrq: a task queue for R based on Redis.
  • rrqueue: predecessor of rrq.
  • clustermq: sends R function calls as jobs to computing clusters.
  • future: a unified interface for asynchronous evaluation of single tasks and map-reduce calls on a wide variety of backend technologies.
  • batchtools: tools for computation on batch systems.
  • targets: a Make-like pipeline tool for R.
  • later: delayed evaluation of synchronous tasks.
  • promises: minimally-invasive asynchronous programming for a small number of tasks within Shiny apps.
  • callr: initiates R process from other R processes.
  • High-performance computing CRAN task view.

Thanks

The crew package incorporates insightful ideas from the following people.

Code of Conduct

Please note that the crew project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

Citation

To cite packagecrewin publications use:

  Landau WM (2023). _crew: A Distributed Worker Launcher Framework_.
  https://wlandau.github.io/crew/, https://github.com/wlandau/crew.

A BibTeX entry for LaTeX users is

  @Manual{,
    title = {crew: A Distributed Worker Launcher Framework},
    author = {William Michael Landau},
    year = {2023},
    note = {https://wlandau.github.io/crew/, https://github.com/wlandau/crew},
  }

More Repositories

1

learndrake

An interactive free online short course on the drake R package
HTML
150
star
2

targets-tutorial

Short course on the targets R package
R
89
star
3

targets-shiny

Example Shiny app containing a targets pipeline.
R
59
star
4

drake-examples

Example workflows for the drake R package
R
58
star
5

targets-minimal

A minimal example data analysis project with the targets R package
R
57
star
6

targets-stan

An example project to validate a Stan model in a targets pipeline
R
24
star
7

txtq

A small message queue for interprocess communication
R
23
star
8

targets-four-minutes

Get started with the {targets} R package in four minutes
R
20
star
9

drakeplanner

A web app to create new drake projects
R
20
star
10

instantiate

Pre-compiled CmdStan models in R packages
R
17
star
11

targetsketch

Sketch a pipeline of targets in an interactive web app
R
14
star
12

dotfiles

my .bashrc and related files
R
14
star
13

gpu

Materials for the Iowa State University Statistics Department fall 2012 lecture series on general purpose GPU computing.
TeX
13
star
14

appPackage

This Shiny app is a package! This package is a Shiny app!
R
11
star
15

targets-keras

An example Keras pipeline with the targets R package
R
11
star
16

grapes

Turn R functions into binary operators.
R
10
star
17

parallelRemake

Accelerate make/remake workflows in R.
HTML
10
star
18

crew.cluster

crew launcher plugins for traditional high-performance computing clusters
R
10
star
19

JeffLeekMeme

True facts about Jeff Leek: https://yihui.name/en/2017/04/jeff-leek-facts/
R
9
star
20

rmedicine2021-pipeline

Example code for a possible talk at R/Medicine 2021 (submitted and under review, accepted talks not yet determined)
HTML
8
star
21

workers

Persistent and transient parallel workers for R
R
7
star
22

historicalborrowlong

Longitudinal Bayesian historical borrowing models
R
7
star
23

targetopia

An R package ecosystem for democratized reproducible pipelines at scale
CSS
6
star
24

fbseq

A fully Bayesian, CUDA-accelerated method of analyzing RNA-seq data
R
6
star
25

targets-debug

Slides and example code for debugging {targets} pipelines
R
5
star
26

targets-intro

Introduction to the {targets} R package
HTML
5
star
27

remakeGenerator

Generate large remake-style workflows with minimal code.
HTML
5
star
28

larug2020

Presentation about the targets R package at the Los Angeles R Users Group Meetup
HTML
4
star
29

fbseqCUDA

CUDA-accelerated backend of fbseq for the MCMC.
C
3
star
30

downsize

Toggle the test and production versions of your workflow with the flip of a global option.
R
3
star
31

wlandau.github.io

Will Landau's portfolio
JavaScript
3
star
32

rna-seq-stan

A comparison of Eric's method to other methods
R
3
star
33

stantargets-talk

Presentation about the {stantargets} R package
R
3
star
34

fbseqStudies

comparison study for fbseq
R
2
star
35

wildcard

Templating for data frames.
R
2
star
36

rcppcuda

A skeleton R package that uses Rcpp, CUDA, and s4 classes. This is a learning exercise.
C++
2
star
37

r-lille-2021

Presentation about {targets} to the Lille R User Meetup Group
JavaScript
2
star
38

rmedicine2021-slides

Slides for a possible talk at R/Medicine 2021 (submitted and under review, accepted talks not yet determined)
HTML
2
star
39

stantargets-example-validation

A minimal example of the stantargets package
R
2
star
40

condo

This is a tutorial on how to log in and run R jobs on the Iowa State condo cluster.
2
star
41

ncb2023

Short course on Bayesian methods for nonclinical statisticians: https://github.com/wlandau/ncb2023
HTML
1
star
42

rpharma2023

R/Pharma 2023 workshop: leveraging {targets} and {crew} to simulate clinical trials
HTML
1
star
43

rpharma2023-pipeline

R/Pharma 2023 workshop example code: leveraging {targets} and {crew} to simulate clinical trials
R
1
star
44

sdss2019

Venue: 2019 ASA Symposium on Data Science and Statistics
HTML
1
star
45

rpharma2020

Slides and source code for a talk about targets at R/Pharma 2020.
HTML
1
star
46

nyhackr2020

Presentation on targets at the New York Open Statistical Programming Meetup
HTML
1
star
47

rmedicine2020

Slides and source code for a talk about drake at R/Medicine 2020.
HTML
1
star
48

bayes-lund-2021

A talk on Target Markdown and {stantargets} at Bayes@Lund
HTML
1
star
49

stat305

TeX
1
star
50

targets-ohdsi

Presentation on {targets} to OHDSI
HTML
1
star
51

workflowHelper

Analyze multiple datasets in multiple ways with a smooth, efficient, parallelized, reproducible R workflow.
R
1
star
52

historicalborrow

Non-longitudinal Bayesian historical borrowing models
R
1
star