• Stars
    star
    411
  • Rank 105,247 (Top 3 %)
  • Language
    Python
  • License
    Other
  • Created over 7 years ago
  • Updated about 2 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Sequence correction provided by ONT Research

Oxford Nanopore Technologies logo

Medaka

install with bioconda

medaka is a tool to create consensus sequences and variant calls from nanopore sequencing data. This task is performed using neural networks applied a pileup of individual sequencing reads against a draft assembly. It provides state-of-the-art results outperforming sequence-graph based methods and signal-based methods, whilst also being faster.

© 2018- Oxford Nanopore Technologies Ltd.

Features

  • Requires only basecalled data. (.fasta or .fastq)
  • Improved accuracy over graph-based methods (e.g. Racon).
  • 50X faster than Nanopolish (and can run on GPUs).
  • Includes extras for implementing and training bespoke correction networks.
  • Works on Linux and MacOS.
  • Open source (Mozilla Public License 2.0).

For creating draft assemblies we recommend Flye.

Installation

Medaka can be installed in one of several ways.

Installation with conda

Perhaps the simplest way to start using medaka is through conda; medaka is available via the bioconda channel:

conda create -n medaka -c conda-forge -c bioconda medaka

Occasionally the conda releases lag behind the source code and PyPI releases.

Installation with pip

For those who prefer Python's native pacakage manager, medaka is also available on pypi and can be installed using pip:

pip install medaka

On Linux platforms this will install a precompiled binary, on MacOS (and other) platforms this will fetch and compile a source distribution.

We recommend using medaka within a virtual environment, viz.:

virtualenv medaka --python=python3 --prompt "(medaka) "
. medaka/bin/activate
pip install --upgrade pip
pip install medaka

Using this method requires the user to provide several binaries:

and place these within the PATH. samtools/bgzip/tabix version 1.14 and minimap2 version 2.17 are recommended as these are those used in development of medaka.

Installation from source

This method is useful for macOS M1 devices as it will assist in building dependencies which will fail with the other methods above.

Medaka can be installed from its source quite easily on most systems.

Before installing medaka it may be required to install some prerequisite libraries, best installed by a package manager. On Ubuntu theses are:

bzip2 g++ zlib1g-dev libbz2-dev liblzma-dev libffi-dev libncurses5-dev
libcurl4-gnutls-dev libssl-dev curl make cmake wget python3-all-dev
python-virtualenv

In addition it is required to install and set up git LFS before cloning the repository.

A Makefile is provided to fetch, compile and install all direct dependencies into a python virtual environment. To set-up the environment run:

# Note: certain files are stored in git-lfs, https://git-lfs.github.com/,
#       which must therefore be installed first.
git clone https://github.com/nanoporetech/medaka.git
cd medaka
make install
. ./venv/bin/activate

Using this method both samtools and minimap2 are built from source and need not be provided by the user.

Using a GPU

Since version 1.1.0 medaka uses Tensorflow 2.2, prior versions used Tensorflow 1.4. For medaka 1.1.0 and higher installation from source or using pip can make immediate use of GPUs. However, note that the tensorflow package is compiled against specific versions of the NVIDIA CUDA and cuDNN libraries; users are directed to the tensorflow installation pages for further information. cuDNN can be obtained from the cuDNN Archive, whilst CUDA from the CUDA Toolkit Archive.

For medaka prior to version 1.1.0, to enable the use of GPU resource it is necessary to install the tensorflow-gpu package. Using the source code from github a working GPU-powered medaka can be configured with:

# Note: certain files are stored in git-lfs, https://git-lfs.github.com/,
#       which must therefore be installed first.
git clone https://github.com/nanoporetech/medaka.git
cd medaka
sed -i 's/tensorflow/tensorflow-gpu/' requirements.txt
make install

GPU Usage notes

Depending on your GPU, medaka may show out of memory errors when running. To avoid these the inference batch size can be reduced from the default value by setting the -b option when running medaka_consensus. A value -b 100 is suitable for 11Gb GPUs.

For users with RTX series GPUs it may be required to additionally set an environment variable to have medaka run without failure:

export TF_FORCE_GPU_ALLOW_GROWTH=true

In this situation a further reduction in batch size may be required.

Using Docker

The source code repository contains a Dockerfile which can be used to create a GPU compatible Docker container image with the appropriate CUDA and cuDNN library versions for running medaka. The image is built on top of images provided by NVIDIA designed to run with the NVIDIA Container Toolkit. With the toolkit setup on your host computer the following command can be used to run the latest version of medaka:

docker run --rm --gpus 0 ontresearch/medaka:latest medaka --help

(The --gpus option can be amended as appropriate for your environment). Versioned tags are also available.

Usage

medaka can be run using its default settings through the medaka_consensus program. An assembly in .fasta format and basecalls in .fasta or .fastq formats are required. The program uses both samtools and minimap2. If medaka has been installed using the from-source method these will be present within the medaka environment, otherwise they will need to be provided by the user.

source ${MEDAKA}  # i.e. medaka/venv/bin/activate
NPROC=$(nproc)
BASECALLS=basecalls.fa
DRAFT=draft_assm/assm_final.fa
OUTDIR=medaka_consensus
medaka_consensus -i ${BASECALLS} -d ${DRAFT} -o ${OUTDIR} -t ${NPROC} -m r941_min_high_g303

The variables BASECALLS, DRAFT, and OUTDIR in the above should be set appropriately. For the selection of the model (-m r941_min_high_g303 in the example above) see the Model section following.

When medaka_consensus has finished running, the consensus will be saved to ${OUTDIR}/consensus.fasta.

Bacterial (ploidy-1) variant calling

Variant calling for monoploid samples is enabled through the medaka_haploid_variant workflow:

medaka_haploid_variant -i <reads.fastq> -r <ref.fasta>

which requires the reads as a .fasta or .fastq and a reference sequence as a .fasta file.

Diploid variant calling

The diploid variant calling workflow medaka_variant that was historically implemented within the medaka package has been surpassed in accuracy and compute performance by other methods, it has therefore been deprecated. Our current recommendation for performing this task is to use Clair3 either directly or through the Oxford Nanopore Technologies provided Nextflow implementation available through EPI2ME Labs.

Models

For best results it is important to specify the correct model, -m in the above, according to the basecaller used. Allowed values can be found by running medaka tools list\_models.

Medaka models are named to indicate i) the pore type, ii) the sequencing device (MinION or PromethION), iii) the basecaller variant, and iv) the basecaller version, with the format:

{pore}_{device}_{caller variant}_{caller version}

For example the model named r941_min_fast_g303 should be used with data from MinION (or GridION) R9.4.1 flowcells using the fast Guppy basecaller version 3.0.3. By contrast the model r941_prom_hac_g303 should be used with PromethION data and the high accuracy basecaller (termed "hac" in Guppy configuration files). Where a version of Guppy has been used without an exactly corresponding medaka model, the medaka model with the highest version equal to or less than the guppy version should be selected.

Improving parallelism

The medaka_consensus program is good for simple datasets but perhaps not optimal for running large datasets at scale. A higher level of parallelism can be achieved by running independently the component steps of medaka_consensus. The program performs three tasks:

  1. alignment of reads to input assembly (via mini_align which is a thin veil over minimap2)
  2. running of consensus algorithm across assembly regions (medaka consensus, note no underscore!)
  3. aggregation of the results of 2. to create consensus sequences (medaka stitch)

The three steps are discrete, and can be split apart and run independently. In most cases, Step 2. is the bottleneck and can be trivially parallelized. The medaka consensus program can be supplied a --regions argument which will restrict its action to particular assembly sequences from the .bam file output in Step 1. Therefore individual jobs can be run for batches of assembly sequences simultaneously. In the final step, medaka stitch can take as input one or more of the .hdf files output by Step 2.

So in summary something like this is possible:

.. code-block:: bash

# align reads to assembly
mini_align -i basecalls.fasta -r assembly.fasta -P -m \
    -p calls_to_draft.bam -t <threads>
# run lots of jobs like this, change model as appropriate
mkdir results
medaka consensus calls_to_draft.bam results/contigs1-4.hdf \
    --model r941_min_fast_g303 --batch 200 --threads 8 \
    --region contig1 contig2 contig3 contig4
...
# wait for jobs, then collate results
medaka stitch results/*.hdf polished.assembly.fasta

It is not recommended to specify a value of --threads greater than 2 for medaka consensus since the compute scaling efficiency is poor beyond this. Note also that medaka consensus may been seen to use resources equivalent to <threads> + 4 as an additional 4 threads are used for reading and preparing input data.

Origin of the draft sequence

Medaka has been trained to correct draft sequences output from the Flye assembler.

Processing a draft sequence from alternative sources (e.g. the output of canu or wtdbg2) may lead to different results.

Historical correction models in medaka were trained to correct draft sequences output from the canu assembler with racon applied either once, or four times iteratively. For contemporary models this is not the case and medaka should be used directly on the output of Flye.

Acknowledgements

We thank Joanna Pineda and Jared Simpson for providing htslib code samples which aided greatly development of the optimised feature generation code, and for testing the version 0.4 release candidates.

We thank Devin Drown for working through use of medaka with his RTX 2080 GPU.

Help

Licence and Copyright

© 2018- Oxford Nanopore Technologies Ltd.

medaka is distributed under the terms of the Mozilla Public License 2.0.

Research Release

Research releases are provided as technology demonstrators to provide early access to features or stimulate Community development of tools. Support for this software will be minimal and is only provided directly by the developers. Feature requests, improvements, and discussions are welcome and can be implemented by forking and pull requests. However much as we would like to rectify every issue and piece of feedback users may have, the developers may have limited resource for support of this software. Research releases may be unstable and subject to rapid iteration by Oxford Nanopore Technologies.

More Repositories

1

dorado

Oxford Nanopore's Basecaller
C++
493
star
2

bonito

A PyTorch Basecaller for Oxford Nanopore Reads
Python
392
star
3

tombo

Tombo is a suite of tools primarily for the identification of modified nucleotides from raw nanopore sequencing data.
Python
230
star
4

megalodon

Megalodon is a research command line tool to extract high accuracy modified base and sequence variant calls from raw nanopore reads by anchoring the information rich basecalling neural network output to a reference genome/transriptome.
Python
197
star
5

fast-ctc-decode

Blitzing Fast CTC Beam Search Decoder
Rust
176
star
6

remora

Methylation/modified base calling separated from basecalling.
Python
156
star
7

ont_fast5_api

Oxford Nanopore Technologies fast5 API software
Python
146
star
8

modkit

A bioinformatics tool for working with modified bases
Rust
137
star
9

pod5-file-format

Pod5: a high performance file format for nanopore reads.
C++
131
star
10

taiyaki

Training models for basecalling Oxford Nanopore reads
Python
114
star
11

pipeline-structural-variation

Pipeline for calling structural variations in whole genomes sequencing Oxford Nanopore data
Python
113
star
12

pipeline-transcriptome-de

Pipeline for differential gene expression (DGE) and differential transcript usage (DTU) analysis using long reads
Python
106
star
13

read_until_api

Read Until client library for Nanopore Sequencing
Python
102
star
14

rerio

Research release basecalling models and configurations
Python
102
star
15

flappie

Flip-flop basecaller for Oxford Nanopore reads
C
98
star
16

pomoxis

Analysis components from Oxford Nanopore Research
Python
94
star
17

scrappie

Scrappie is a technology demonstrator for the Oxford Nanopore Research Algorithms group
C
91
star
18

ont-assembly-polish

ONT assembly and Illumina polishing pipeline
Makefile
91
star
19

pychopper

A tool to identify, orient, trim and rescue full length cDNA reads
Python
79
star
20

qcat

qcat is a Python command-line tool for demultiplexing Oxford Nanopore reads from FASTQ files.
Python
77
star
21

jmespath-ts

Typescript translation of the jmespath.js package
TypeScript
63
star
22

wub

Tools and software library developed by the ONT Applications group
Python
61
star
23

minknow_api

Protobuf and gRPC specifications for the MinKNOW API
Python
55
star
24

pore-c

Pore-C support
Python
53
star
25

kmer_models

Predictive kmer models for development use
53
star
26

katuali

Analysis pipelines from Oxford Nanopore Technologies' Research Division
Python
50
star
27

duplex-tools

Splitting of sequence reads by internal adapter sequence search
Python
49
star
28

pinfish

Tools to annotate genomes using long read transcriptomics data
Go
45
star
29

sockeye

Single Cell Transcriptomics
Python
40
star
30

vbz_compression

VBZ compression plugin for nanopore signal data
C++
38
star
31

pipeline-nanopore-ref-isoforms

Pipeline for annotating genomes using long read transcriptomics data with stringtie and other tools
Python
36
star
32

Pore-C-Snakemake

Python
33
star
33

bwapy

Python bindings to bwa mem
Python
31
star
34

ont_tutorial_basicqc

A bioinformatics tutorial demonstrating a best-practice workflow to review a flowcell's sequence_summary.txt
TeX
30
star
35

pyguppyclient

Python client library for Guppy
Python
30
star
36

pipeline-umi-amplicon

Workflow to prepare high accuracy single molecule consensus sequences from amplicon data using unique molecular identifiers
Python
28
star
37

pipeline-pinfish-analysis

Pipeline for annotating genomes using long read transcriptomics data with pinfish
Python
27
star
38

pipeline-nanopore-denovo-isoforms

Pipeline for de novo clustering of long transcriptomic reads
Python
26
star
39

sloika

Sloika is Oxford Nanopore Technologies' software for training neural network models for base calling
Python
25
star
40

fast5_research

Fast5 API provided by ONT Research
Python
21
star
41

pyspoa

Python bindings to spoa
Python
18
star
42

DTR-phage-pipeline

Python
16
star
43

minimappy

Python bindings to minimap2
Python
16
star
44

isONclust2

A tool for de novo clustering of long transcriptomic reads
C++
14
star
45

jmespath-plus

JMESPath with extended collection of built-in functions
TypeScript
14
star
46

minknow_lims_interface

Protobuff and gRPC specifications for the MinKNOW LIMS Interface
13
star
47

fast5mod

Extract modifed base call information from Guppy Fast5 files.
Python
13
star
48

ont_h5_validator

Python
12
star
49

dRNA-paper-scripts

Direct RNA publication scripts
Python
11
star
50

currennt

Modified fork of CURRENNT https://sourceforge.net/projects/currennt/
C++
11
star
51

pipeline-polya-diff

Pipeline for testing shifts in poly(A) tail lengths estimated by nanopolish
Python
9
star
52

ont-open-datasets

Website describing data releases, and providing additional resources.
HTML
9
star
53

pipeline-polya-ng

Pipeline for calling poly(A) tail lengths from nanopore direct RNA data using nanopolish
Python
9
star
54

ts-runtime-typecheck

A collection of common types for TypeScript along with dynamic type cast methods.
TypeScript
9
star
55

epi2me-api

API for communicating with the EPI2ME Platform for nanopore data analysis. Used by EPI2ME Agent & CLI.
TypeScript
9
star
56

cronkite

One **hell** of a reporter
TypeScript
8
star
57

mako

Analyte identification via squiggles.
Python
7
star
58

marine-phage-paper-scripts

Python
6
star
59

homebrew-tap

Homebrew casks for applications from Oxford Nanopore Technologies PLC and Metrichor Ltd.
Ruby
6
star
60

barcoding

Naïve barcode deconvolution for amplicons
Perl
6
star
61

ont-minimap2

Cross platform builds for minimap2
CMake
5
star
62

plasmid-map

Plasmid map visualisations for Metrichor reports
TypeScript
5
star
63

spliced_bam2gff

Go
5
star
64

hammerpede

A package for training strand-specific profile HMMs for primer sets from real Nanopore data
Python
5
star
65

hatch-protobuf

Hatch plugin for generating Python files from Protocol Buffers .proto files
Python
4
star
66

fastq-filter

Quality and length filter for FastQ data
Python
4
star
67

bripy

Bam Read Index for python
C
3
star
68

pipeline-pychopper

Utility pipeline for running pychopper, a tool to identify full length cDNA reads
Python
3
star
69

lamprey

GUI for desktop basecalling
JavaScript
3
star
70

panga

Python
2
star
71

data-rambler

An experimental language for a JSON query, transformation and streaming
TypeScript
2
star
72

getopt-win32

C
2
star
73

ts-argue

TypeScript
1
star
74

onesie

A Linux device-driver for the MinION-mk1C
C
1
star
75

vbz-h5py-plugin

Python
1
star
76

fs-inspect

node.js library for indexing the contents of a folder
TypeScript
1
star