Reproducible, scalable, and shareable analysis pipelines with bioinformatics workflow managers

Workflow managers provide an easy and intuitive way to simplify pipeline development. Here we provide basic proof-of-concept implementations for selected workflow managers. The analysis workflow is based on a small portion of an RNA-seq pipeline, using fastqc for quality controls and salmon for transcript quantification. These implementations are designed for basic illustrations. Workflow managers provide many more powerful features than what we use here, please visit the official documentations to explore those in detail.

The RNA-Seq workflow

The RNA-Seq analysis workflow performs quality controls with fastqc and quantifies transcripts expression using Salmon. Here we will use local installation (see documentation for salmon and fastqc). For the local installations you can add a symbolic link to the executables to your $PATH:

sudo ln -s /absolute/path/salmon/bin/salmon /usr/local/bin/salmon

sudo ln -s /absolute/path/FastQC/fastqc /usr/local/bin/fastqc

You can test the installation using the help function of the two tools (i.e. salmon -h and fastqc -h).

Test Data

This repository contains a simulated test data set which can be used to run the example implementations. The test data contains RNA-Seq reads (reads_1.fq.gz and reads_2.fq.gz), a transcriptome reference file (transcriptome.fa) and the true counts from the simulation experiments (truth.tsv)

Basic proof-of-concept implementations

Each workflow manager folder in this repository has a README detailing how to run the proof-of-concept pipeline. Implementations that are contributed and reviewed by developers from these workflow management systems are marked with ⭐ (see Acknowledgements):

• Galaxy ⭐
• Nextflow ⭐
• Snakemake ⭐
• SciPipe ⭐
• GenPipes ⭐
• Bpipe ⭐
• Toil (WDL/CWL/Python) ⭐
• cromwell (WDL)

Online Documentation for Workflow managers

Workflow managers have many more features which are not used in these implementations, and there are many additional workflow managers. You can read more about each workflow manager in their official documentation:

• Galaxy
• KNIME
• Nextflow
• Snakemake
• GenPipes
• Bpipe
• Pachyderm
• SciPipe
• Luigi
• WDL
• CWL
• Toil

About this repository

The repository was created to illustrate features of workflow managers that are discussed in detail in this manuscript:

Wratten, L., Wilm, A. & Göke, J. Reproducible, scalable, and shareable analysis pipelines with bioinformatics workflow managers. Nat Methods (2021). https://doi.org/10.1038/s41592-021-01254-9 (Full text link: https://rdcu.be/cyjRN)

Contact and Call for Contribution

This repository was created by Laura Wratten. We very much encourage contributions by users of these workflows. If you would like to add an implementation for any of these workflow managers you can follow the template. If you would like to suggest changes to any of the existing implementations, please raise an issue and submit a pull request.

Acknowledgements

We would like to thank the following people for their contribution to this repository:

• Rob Patro for creating the RNA-Seq test data set
• Paolo Di Tommaso for the Nextflow workflow
• Johannes Köster for the Snakemake workflow
• Samuel Lampa for the SciPipe workflow
• Marius van den Beek for the Galaxy workflow
• José Héctor Gálvez López, Pierre-Olivier Quirion, Edouard Henrion, and Mathieu Bourgey for the GenPipes workflow
• Simon Sadedin for the Bpipe workflow
• Lukas Koziol Adam Novak, Benedict Paten, and Lon Blauvelt for the Toil workflows