• Stars
    star
    103
  • Rank 325,825 (Top 7 %)
  • Language
    R
  • License
    Other
  • Created over 2 years ago
  • Updated 16 days ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A distributed worker launcher

crew: a distributed worker launcher framework

CRAN status check codecov lint pkgdown

In computationally demanding analysis projects, statisticians and data scientists asynchronously deploy long-running tasks to distributed systems, ranging from traditional clusters to cloud services. The NNG-powered mirai R package is a sleek and sophisticated scheduler that efficiently processes these intense workloads. The crew package extends mirai with a unifying interface for third-party worker launchers. Inspiration also comes from packages future, rrq, clustermq, and batchtools.

📣 Request for community contributions 📣

The launcher plugin framework aims to extend crew to modern platforms and services like Google Cloud Run, Kubernetes, and beyond. At the time of writing, plugins for cloud computing do not yet exist. So if you have access to these services and know how to use them, please consider contributing a package with plugins of your own. The tutorial at https://wlandau.github.io/crew/articles/plugins.html explains how. The maintainer of crew would love to work with you!

Installation

Type Source Command
Release CRAN install.packages("crew")
Development GitHub remotes::install_github("wlandau/crew")
Development R-universe install.packages("crew", repos = "https://wlandau.r-universe.dev")

Documentation

The documentation website at https://wlandau.github.io/crew/ includes a function reference and tutorial vignettes linked below.

  1. Introduction to crew
  2. Controller groups
  3. Integration with Shiny
  4. How to write launcher plugins
  5. Known risks of crew

Risks

The crew package has unavoidable risks, and the user is responsible for safety, security, and computational resources. Please read the software license and the vignette about specific known risks.

Similar work

  • mirai: a powerful R framework for asynchronous tasks built on NNG. The purpose of crew is to extend mirai to different computing platforms for distributed workers.
  • rrq: a task queue for R based on Redis.
  • rrqueue: predecessor of rrq.
  • clustermq: sends R function calls as jobs to computing clusters.
  • future: a unified interface for asynchronous evaluation of single tasks and map-reduce calls on a wide variety of backend technologies.
  • batchtools: tools for computation on batch systems.
  • targets: a Make-like pipeline tool for R.
  • later: delayed evaluation of synchronous tasks.
  • promises: minimally-invasive asynchronous programming for a small number of tasks within Shiny apps.
  • callr: initiates R process from other R processes.
  • High-performance computing CRAN task view.

Thanks

The crew package incorporates insightful ideas from the following people.

Code of Conduct

Please note that the crew project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

Citation

To cite packagecrewin publications use:

  Landau WM (2023). _crew: A Distributed Worker Launcher Framework_.
  https://wlandau.github.io/crew/, https://github.com/wlandau/crew.

A BibTeX entry for LaTeX users is

  @Manual{,
    title = {crew: A Distributed Worker Launcher Framework},
    author = {William Michael Landau},
    year = {2023},
    note = {https://wlandau.github.io/crew/, https://github.com/wlandau/crew},
  }

More Repositories

1

learndrake

An interactive free online short course on the drake R package
HTML
146
star
2

targets-tutorial

Short course on the targets R package
R
94
star
3

targets-shiny

Example Shiny app containing a targets pipeline.
R
61
star
4

targets-minimal

A minimal example data analysis project with the targets R package
R
58
star
5

drake-examples

Example workflows for the drake R package
R
57
star
6

targets-four-minutes

Get started with the {targets} R package in four minutes
R
36
star
7

targets-stan

An example project to validate a Stan model in a targets pipeline
R
27
star
8

txtq

A small message queue for interprocess communication
R
23
star
9

instantiate

Pre-compiled CmdStan models in R packages
R
21
star
10

drakeplanner

A web app to create new drake projects
R
20
star
11

crew.cluster

crew launcher plugins for traditional high-performance computing clusters
R
20
star
12

targetsketch

Sketch a pipeline of targets in an interactive web app
R
17
star
13

dotfiles

my .bashrc and related files
R
15
star
14

gpu

Materials for the Iowa State University Statistics Department fall 2012 lecture series on general purpose GPU computing.
TeX
13
star
15

targets-keras

An example Keras pipeline with the targets R package
R
12
star
16

appPackage

This Shiny app is a package! This package is a Shiny app!
R
11
star
17

crew.aws.batch

A {crew} launcher plugin for AWS Batch
R
11
star
18

rpharma2023-pipeline

R/Pharma 2023 workshop example code: leveraging {targets} and {crew} to simulate clinical trials
R
10
star
19

grapes

Turn R functions into binary operators.
R
10
star
20

parallelRemake

Accelerate make/remake workflows in R.
HTML
10
star
21

targets-debug

Slides and example code for debugging {targets} pipelines
R
9
star
22

JeffLeekMeme

True facts about Jeff Leek: https://yihui.name/en/2017/04/jeff-leek-facts/
R
9
star
23

rmedicine2021-pipeline

Example code for a possible talk at R/Medicine 2021 (submitted and under review, accepted talks not yet determined)
HTML
8
star
24

targetopia

An R package ecosystem for democratized reproducible pipelines at scale
CSS
8
star
25

historicalborrowlong

Longitudinal Bayesian historical borrowing models
R
8
star
26

workers

Persistent and transient parallel workers for R
R
7
star
27

rpharma2023

R/Pharma 2023 workshop: leveraging {targets} and {crew} to simulate clinical trials
HTML
6
star
28

fbseq

A fully Bayesian, CUDA-accelerated method of analyzing RNA-seq data
R
6
star
29

targets-intro

Introduction to the {targets} R package
HTML
5
star
30

remakeGenerator

Generate large remake-style workflows with minimal code.
HTML
5
star
31

larug2020

Presentation about the targets R package at the Los Angeles R Users Group Meetup
HTML
4
star
32

fbseqCUDA

CUDA-accelerated backend of fbseq for the MCMC.
C
3
star
33

fbseqStudies

comparison study for fbseq
R
3
star
34

downsize

Toggle the test and production versions of your workflow with the flip of a global option.
R
3
star
35

wlandau.github.io

Will Landau's portfolio
JavaScript
3
star
36

rmedicine2021-slides

Slides for a possible talk at R/Medicine 2021 (submitted and under review, accepted talks not yet determined)
HTML
3
star
37

historicalborrow

Non-longitudinal Bayesian historical borrowing models
R
3
star
38

rna-seq-stan

A comparison of Eric's method to other methods
R
3
star
39

stantargets-talk

Presentation about the {stantargets} R package
R
3
star
40

wildcard

Templating for data frames.
R
2
star
41

rmedicine2020

Slides and source code for a talk about drake at R/Medicine 2020.
HTML
2
star
42

r-lille-2021

Presentation about {targets} to the Lille R User Meetup Group
JavaScript
2
star
43

stantargets-example-validation

A minimal example of the stantargets package
R
2
star
44

condo

This is a tutorial on how to log in and run R jobs on the Iowa State condo cluster.
2
star
45

rcppcuda

A skeleton R package that uses Rcpp, CUDA, and s4 classes. This is a learning exercise.
C++
2
star
46

ncb2023

Short course on Bayesian methods for nonclinical statisticians: https://github.com/wlandau/ncb2023
HTML
1
star
47

sdss2019

Venue: 2019 ASA Symposium on Data Science and Statistics
HTML
1
star
48

nyhackr2020

Presentation on targets at the New York Open Statistical Programming Meetup
HTML
1
star
49

stat305

TeX
1
star
50

bayes-lund-2021

A talk on Target Markdown and {stantargets} at Bayes@Lund
HTML
1
star
51

targets-ohdsi

Presentation on {targets} to OHDSI
HTML
1
star
52

rpharma2020

Slides and source code for a talk about targets at R/Pharma 2020.
HTML
1
star
53

workflowHelper

Analyze multiple datasets in multiple ways with a smooth, efficient, parallelized, reproducible R workflow.
R
1
star