• Stars
    star
    100
  • Rank 326,326 (Top 7 %)
  • Language
    R
  • License
    Other
  • Created about 2 years ago
  • Updated 14 days ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A distributed worker launcher

crew: a distributed worker launcher framework

CRAN status check codecov lint pkgdown

In computationally demanding analysis projects, statisticians and data scientists asynchronously deploy long-running tasks to distributed systems, ranging from traditional clusters to cloud services. The NNG-powered mirai R package is a sleek and sophisticated scheduler that efficiently processes these intense workloads. The crew package extends mirai with a unifying interface for third-party worker launchers. Inspiration also comes from packages future, rrq, clustermq, and batchtools.

📣 Request for community contributions 📣

The launcher plugin framework aims to extend crew to modern platforms and services like Google Cloud Run, Kubernetes, and beyond. At the time of writing, plugins for cloud computing do not yet exist. So if you have access to these services and know how to use them, please consider contributing a package with plugins of your own. The tutorial at https://wlandau.github.io/crew/articles/plugins.html explains how. The maintainer of crew would love to work with you!

Installation

Type Source Command
Release CRAN install.packages("crew")
Development GitHub remotes::install_github("wlandau/crew")
Development R-universe install.packages("crew", repos = "https://wlandau.r-universe.dev")

Documentation

The documentation website at https://wlandau.github.io/crew/ includes a function reference and tutorial vignettes linked below.

  1. Introduction to crew
  2. Controller groups
  3. Integration with Shiny
  4. How to write launcher plugins
  5. Known risks of crew

Risks

The crew package has unavoidable risks, and the user is responsible for safety, security, and computational resources. Please read the software license and the vignette about specific known risks.

Similar work

  • mirai: a powerful R framework for asynchronous tasks built on NNG. The purpose of crew is to extend mirai to different computing platforms for distributed workers.
  • rrq: a task queue for R based on Redis.
  • rrqueue: predecessor of rrq.
  • clustermq: sends R function calls as jobs to computing clusters.
  • future: a unified interface for asynchronous evaluation of single tasks and map-reduce calls on a wide variety of backend technologies.
  • batchtools: tools for computation on batch systems.
  • targets: a Make-like pipeline tool for R.
  • later: delayed evaluation of synchronous tasks.
  • promises: minimally-invasive asynchronous programming for a small number of tasks within Shiny apps.
  • callr: initiates R process from other R processes.
  • High-performance computing CRAN task view.

Thanks

The crew package incorporates insightful ideas from the following people.

Code of Conduct

Please note that the crew project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

Citation

To cite package ‘crew’ in publications use:

  Landau WM (2023). _crew: A Distributed Worker Launcher Framework_.
  https://wlandau.github.io/crew/, https://github.com/wlandau/crew.

A BibTeX entry for LaTeX users is

  @Manual{,
    title = {crew: A Distributed Worker Launcher Framework},
    author = {William Michael Landau},
    year = {2023},
    note = {https://wlandau.github.io/crew/, https://github.com/wlandau/crew},
  }

More Repositories

1

learndrake

An interactive free online short course on the drake R package
HTML
147
star
2

targets-tutorial

Short course on the targets R package
R
92
star
3

targets-shiny

Example Shiny app containing a targets pipeline.
R
60
star
4

drake-examples

Example workflows for the drake R package
R
57
star
5

targets-minimal

A minimal example data analysis project with the targets R package
R
56
star
6

targets-four-minutes

Get started with the {targets} R package in four minutes
R
29
star
7

targets-stan

An example project to validate a Stan model in a targets pipeline
R
26
star
8

txtq

A small message queue for interprocess communication
R
23
star
9

instantiate

Pre-compiled CmdStan models in R packages
R
21
star
10

drakeplanner

A web app to create new drake projects
R
20
star
11

crew.cluster

crew launcher plugins for traditional high-performance computing clusters
R
16
star
12

targetsketch

Sketch a pipeline of targets in an interactive web app
R
15
star
13

dotfiles

my .bashrc and related files
R
15
star
14

targets-keras

An example Keras pipeline with the targets R package
R
13
star
15

gpu

Materials for the Iowa State University Statistics Department fall 2012 lecture series on general purpose GPU computing.
TeX
13
star
16

appPackage

This Shiny app is a package! This package is a Shiny app!
R
11
star
17

grapes

Turn R functions into binary operators.
R
10
star
18

parallelRemake

Accelerate make/remake workflows in R.
HTML
10
star
19

JeffLeekMeme

True facts about Jeff Leek: https://yihui.name/en/2017/04/jeff-leek-facts/
R
9
star
20

crew.aws.batch

A {crew} launcher plugin for AWS Batch
R
9
star
21

rpharma2023-pipeline

R/Pharma 2023 workshop example code: leveraging {targets} and {crew} to simulate clinical trials
R
8
star
22

rmedicine2021-pipeline

Example code for a possible talk at R/Medicine 2021 (submitted and under review, accepted talks not yet determined)
HTML
8
star
23

targetopia

An R package ecosystem for democratized reproducible pipelines at scale
CSS
8
star
24

historicalborrowlong

Longitudinal Bayesian historical borrowing models
R
8
star
25

targets-debug

Slides and example code for debugging {targets} pipelines
R
7
star
26

workers

Persistent and transient parallel workers for R
R
7
star
27

fbseq

A fully Bayesian, CUDA-accelerated method of analyzing RNA-seq data
R
6
star
28

rpharma2023

R/Pharma 2023 workshop: leveraging {targets} and {crew} to simulate clinical trials
HTML
5
star
29

targets-intro

Introduction to the {targets} R package
HTML
5
star
30

remakeGenerator

Generate large remake-style workflows with minimal code.
HTML
5
star
31

larug2020

Presentation about the targets R package at the Los Angeles R Users Group Meetup
HTML
4
star
32

fbseqCUDA

CUDA-accelerated backend of fbseq for the MCMC.
C
3
star
33

fbseqStudies

comparison study for fbseq
R
3
star
34

downsize

Toggle the test and production versions of your workflow with the flip of a global option.
R
3
star
35

wlandau.github.io

Will Landau's portfolio
JavaScript
3
star
36

historicalborrow

Non-longitudinal Bayesian historical borrowing models
R
3
star
37

rna-seq-stan

A comparison of Eric's method to other methods
R
3
star
38

stantargets-talk

Presentation about the {stantargets} R package
R
3
star
39

wildcard

Templating for data frames.
R
2
star
40

r-lille-2021

Presentation about {targets} to the Lille R User Meetup Group
JavaScript
2
star
41

rmedicine2021-slides

Slides for a possible talk at R/Medicine 2021 (submitted and under review, accepted talks not yet determined)
HTML
2
star
42

stantargets-example-validation

A minimal example of the stantargets package
R
2
star
43

condo

This is a tutorial on how to log in and run R jobs on the Iowa State condo cluster.
2
star
44

rcppcuda

A skeleton R package that uses Rcpp, CUDA, and s4 classes. This is a learning exercise.
C++
2
star
45

ncb2023

Short course on Bayesian methods for nonclinical statisticians: https://github.com/wlandau/ncb2023
HTML
1
star
46

sdss2019

Venue: 2019 ASA Symposium on Data Science and Statistics
HTML
1
star
47

nyhackr2020

Presentation on targets at the New York Open Statistical Programming Meetup
HTML
1
star
48

rmedicine2020

Slides and source code for a talk about drake at R/Medicine 2020.
HTML
1
star
49

bayes-lund-2021

A talk on Target Markdown and {stantargets} at Bayes@Lund
HTML
1
star
50

stat305

TeX
1
star
51

targets-ohdsi

Presentation on {targets} to OHDSI
HTML
1
star
52

rpharma2020

Slides and source code for a talk about targets at R/Pharma 2020.
HTML
1
star
53

workflowHelper

Analyze multiple datasets in multiple ways with a smooth, efficient, parallelized, reproducible R workflow.
R
1
star