• Stars
    star
    121
  • Rank 293,108 (Top 6 %)
  • Language
    R
  • License
    Other
  • Created over 2 years ago
  • Updated about 1 month ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A distributed worker launcher

crew: a distributed worker launcher framework

CRAN status check codecov lint pkgdown

In computationally demanding analysis projects, statisticians and data scientists asynchronously deploy long-running tasks to distributed systems, ranging from traditional clusters to cloud services. The NNG-powered mirai R package is a sleek and sophisticated scheduler that efficiently processes these intense workloads. The crew package extends mirai with a unifying interface for third-party worker launchers. Inspiration also comes from packages future, rrq, clustermq, and batchtools.

📣 Request for community contributions 📣

The launcher plugin framework aims to extend crew to modern platforms and services like Google Cloud Run, Kubernetes, and beyond. At the time of writing, plugins for cloud computing do not yet exist. So if you have access to these services and know how to use them, please consider contributing a package with plugins of your own. The tutorial at https://wlandau.github.io/crew/articles/plugins.html explains how. The maintainer of crew would love to work with you!

Installation

Type Source Command
Release CRAN install.packages("crew")
Development GitHub remotes::install_github("wlandau/crew")
Development R-universe install.packages("crew", repos = "https://wlandau.r-universe.dev")

Documentation

The documentation website at https://wlandau.github.io/crew/ includes a function reference and tutorial vignettes linked below.

  1. Introduction to crew
  2. Controller groups
  3. Integration with Shiny
  4. How to write launcher plugins
  5. Known risks of crew

Risks

The crew package has unavoidable risks, and the user is responsible for safety, security, and computational resources. Please read the software license and the vignette about specific known risks.

Similar work

  • mirai: a powerful R framework for asynchronous tasks built on NNG. The purpose of crew is to extend mirai to different computing platforms for distributed workers.
  • rrq: a task queue for R based on Redis.
  • rrqueue: predecessor of rrq.
  • clustermq: sends R function calls as jobs to computing clusters.
  • future: a unified interface for asynchronous evaluation of single tasks and map-reduce calls on a wide variety of backend technologies.
  • batchtools: tools for computation on batch systems.
  • targets: a Make-like pipeline tool for R.
  • later: delayed evaluation of synchronous tasks.
  • promises: minimally-invasive asynchronous programming for a small number of tasks within Shiny apps.
  • callr: initiates R process from other R processes.
  • High-performance computing CRAN task view.

Thanks

The crew package incorporates insightful ideas from the following people.

Code of Conduct

Please note that the crew project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

Citation

To cite package ‘crew’ in publications use:

  Landau WM (2023). _crew: A Distributed Worker Launcher Framework_.
  https://wlandau.github.io/crew/, https://github.com/wlandau/crew.

A BibTeX entry for LaTeX users is

  @Manual{,
    title = {crew: A Distributed Worker Launcher Framework},
    author = {William Michael Landau},
    year = {2023},
    note = {https://wlandau.github.io/crew/, https://github.com/wlandau/crew},
  }

More Repositories

1

learndrake

An interactive free online short course on the drake R package
HTML
146
star
2

targets-tutorial

Short course on the targets R package
R
95
star
3

targets-shiny

Example Shiny app containing a targets pipeline.
R
62
star
4

targets-minimal

A minimal example data analysis project with the targets R package
R
58
star
5

drake-examples

Example workflows for the drake R package
R
56
star
6

targets-four-minutes

Get started with the {targets} R package in four minutes
R
36
star
7

targets-stan

An example project to validate a Stan model in a targets pipeline
R
27
star
8

crew.cluster

crew launcher plugins for traditional high-performance computing clusters
R
24
star
9

txtq

A small message queue for interprocess communication
R
23
star
10

instantiate

Pre-compiled CmdStan models in R packages
R
22
star
11

drakeplanner

A web app to create new drake projects
R
20
star
12

targetsketch

Sketch a pipeline of targets in an interactive web app
R
17
star
13

dotfiles

my .bashrc and related files
R
15
star
14

crew.aws.batch

A {crew} launcher plugin for AWS Batch
R
15
star
15

posit2024

posit::conf(2024) presentation about {mirai} and {crew}
JavaScript
13
star
16

gpu

Materials for the Iowa State University Statistics Department fall 2012 lecture series on general purpose GPU computing.
TeX
13
star
17

targets-keras

An example Keras pipeline with the targets R package
R
12
star
18

appPackage

This Shiny app is a package! This package is a Shiny app!
R
11
star
19

rpharma2023-pipeline

R/Pharma 2023 workshop example code: leveraging {targets} and {crew} to simulate clinical trials
R
10
star
20

grapes

Turn R functions into binary operators.
R
10
star
21

parallelRemake

Accelerate make/remake workflows in R.
HTML
10
star
22

targets-debug

Slides and example code for debugging {targets} pipelines
R
9
star
23

JeffLeekMeme

True facts about Jeff Leek: https://yihui.name/en/2017/04/jeff-leek-facts/
R
9
star
24

rmedicine2021-pipeline

Example code for a possible talk at R/Medicine 2021 (submitted and under review, accepted talks not yet determined)
HTML
8
star
25

targetopia

An R package ecosystem for democratized reproducible pipelines at scale
CSS
8
star
26

historicalborrowlong

Longitudinal Bayesian historical borrowing models
R
8
star
27

workers

Persistent and transient parallel workers for R
R
7
star
28

rpharma2023

R/Pharma 2023 workshop: leveraging {targets} and {crew} to simulate clinical trials
HTML
6
star
29

fbseq

A fully Bayesian, CUDA-accelerated method of analyzing RNA-seq data
R
6
star
30

targets-intro

Introduction to the {targets} R package
HTML
5
star
31

remakeGenerator

Generate large remake-style workflows with minimal code.
HTML
5
star
32

larug2020

Presentation about the targets R package at the Los Angeles R Users Group Meetup
HTML
4
star
33

fbseqCUDA

CUDA-accelerated backend of fbseq for the MCMC.
C
3
star
34

downsize

Toggle the test and production versions of your workflow with the flip of a global option.
R
3
star
35

wlandau.github.io

Will Landau's portfolio
JavaScript
3
star
36

rmedicine2021-slides

Slides for a possible talk at R/Medicine 2021 (submitted and under review, accepted talks not yet determined)
HTML
3
star
37

fbseqStudies

comparison study for fbseq
R
3
star
38

historicalborrow

Non-longitudinal Bayesian historical borrowing models
R
3
star
39

rna-seq-stan

A comparison of Eric's method to other methods
R
3
star
40

stantargets-talk

Presentation about the {stantargets} R package
R
3
star
41

stantargets-example-validation

A minimal example of the stantargets package
R
2
star
42

wildcard

Templating for data frames.
R
2
star
43

rmedicine2020

Slides and source code for a talk about drake at R/Medicine 2020.
HTML
2
star
44

r-lille-2021

Presentation about {targets} to the Lille R User Meetup Group
JavaScript
2
star
45

condo

This is a tutorial on how to log in and run R jobs on the Iowa State condo cluster.
2
star
46

rcppcuda

A skeleton R package that uses Rcpp, CUDA, and s4 classes. This is a learning exercise.
C++
2
star
47

ncb2023

Short course on Bayesian methods for nonclinical statisticians: https://github.com/wlandau/ncb2023
HTML
1
star
48

nyhackr2020

Presentation on targets at the New York Open Statistical Programming Meetup
HTML
1
star
49

sdss2019

Venue: 2019 ASA Symposium on Data Science and Statistics
HTML
1
star
50

mirai

mirai - minimalist async evaluation framework for R
R
1
star
51

stat305

TeX
1
star
52

bayes-lund-2021

A talk on Target Markdown and {stantargets} at Bayes@Lund
HTML
1
star
53

targets-ohdsi

Presentation on {targets} to OHDSI
HTML
1
star
54

rpharma2020

Slides and source code for a talk about targets at R/Pharma 2020.
HTML
1
star
55

workflowHelper

Analyze multiple datasets in multiple ways with a smooth, efficient, parallelized, reproducible R workflow.
R
1
star