NOTE: This is LUMPY 0.2.13 with additional changes to allow lumpyexpress to function when the main file is a CRAM, not a BAM. The splitters and discordants must still be BAM files as LUMPY itself doesn't yet support CRAM as an input. This requires the hexdump command be available.
For questions and discussion about LUMPY please visit the forum at:
https://groups.google.com/forum/#!forum/lumpy-discuss
LUMPY
A probabilistic framework for structural variant discovery.
Ryan M Layer, Colby Chiang, Aaron R Quinlan, and Ira M Hall. 2014. "LUMPY: a Probabilistic Framework for Structural Variant Discovery." Genome Biology 15 (6): R84. doi:10.1186/gb-2014-15-6-r84.
Table of Contents
- Quick start
- Installation
- LUMPY Express usage: Automated breakpoint detection for standard analyses.
- LUMPY (traditional) usage: Flexible and customizable breakpoint detection for advanced users.
- Example workflows
- Test data
- Troubleshooting
Quick start
Note that smoove is the recommended way to run lumpy as it collects the
best-practices of lumpy and associated tools and will have a shorter run-time and lower false-positive rate than
lumpyexpress described below.
Download and install
git clone --recursive https://github.com/arq5x/lumpy-sv.git
cd lumpy-sv
make
cp bin/* /usr/local/bin/.
Run LUMPY Express
lumpyexpress \
-B my.bam \
-S my.splitters.bam \
-D my.discordants.bam \
-o output.vcf
Installation
Requirements
- LUMPY
- g++ compiler
- CMake
- LUMPY Express (optional)
- Samtools (0.1.18+) (htslib.org/)
- SAMBLASTER (0.1.19+) (github repo)
- Python 2.7 (python.org/) with pysam (0.8.3+) and NumPy (1.8.1+)
- sambamba (gihub repo)
- gawk (GNU project)
Install
Default method to install:
git clone --recursive [email protected]:arq5x/lumpy-sv.git
cd lumpy-sv
make
cp bin/* /usr/local/bin/.
Installing with costom zlib (gzopen64 compile error):
git clone --recursive [email protected]:arq5x/lumpy-sv.git
cd lumpy-sv
export ZLIB_PATH="/usr/lib/x86_64-linux-gnu/"; #when /usr/lib/x86_64-linux-gnu/libz.so
make
cp bin/* /usr/local/bin/.
LUMPY Express usage
Automated breakpoint detection for standard analyses.
usage: lumpyexpress [options]
Required arguments
-B FILE coordinate-sorted BAM file(s) (comma separated)
-S FILE split reads BAM file(s) (comma separated)
-D FILE discordant reads BAM files(s) (comma separated)
Optional arguments
-o STR output [fullBam.bam.vcf]
-x FILE BED file to exclude
-P output probability curves for each variant
-m INT minimum sample weight for a call [4]
-r FLOAT trim threshold [0]
-T DIR temp directory [./output_prefix.XXXXXXXXXXXX]
-k keep temporary files
-K FILE path to lumpyexpress.config file
(default: same directory as lumpyexpress)
-v verbose
-h show this message
Configuration
LUMPY Express runs several external program whose paths are specified in scripts/lumpyexpress.config. This config must reside in the same directory as lumpyexpress, or be specified explicitly with the -K flag.
The installation Makefile auto-generates a lumpyexpress.config file and places it in the "bin" directory.
Input
LUMPY Express expects BWA-MEM aligned BAM files as input. It automatically parses sample, library, and read group information using the @RG tags in the BAM header. Each BAM file is expected to contain exactly one sample.
The minimum input is a coordinate-sorted BAM file (-B), from which LUMPY Express extracts splitters and discordants using SAMBLASTER before running LUMPY. Optionally, users may supply coordinate-sorted splitter (-S) and discordant (-D) BAM files which will bypass SAMBLASTER extraction for faster analysis.
Output
LUMPY Express produces a VCF file according to VCF spec 4.2.
LUMPY (traditional) usage
Flexible and customizable breakpoint detection for advanced users.
usage: lumpy [options]
Options
-g Genome file (defines chromosome order)
-e Show evidence for each call
-w File read windows size (default 1000000)
-mw minimum weight across all samples for a call
-msw minimum per-sample weight for a call
-tt trim threshold
-x exclude file bed file
-t temp file prefix, must be to a writeable directory
-P output probability curve for each variant
-b output as BEDPE instead of VCF
-sr bam_file:<file name>,
id:<sample name>,
back_distance:<distance>,
min_mapping_threshold:<mapping quality>,
weight:<sample weight>,
min_clip:<minimum clip length>,
read_group:<string>
-pe bam_file:<file name>,
id:<sample name>,
histo_file:<file name>,
mean:<value>,
stdev:<value>,
read_length:<length>,
min_non_overlap:<length>,
discordant_z:<z value>,
back_distance:<distance>,
min_mapping_threshold:<mapping quality>,
weight:<sample weight>,
read_group:<string>
-bedpe bedpe_file:<bedpe file>,
id:<sample name>,
weight:<sample weight>
Example workflows
Pre-processing
We recommend aligning data with SpeedSeq, which performs BWA-MEM alignment, marks duplicates and extracts split and discordant read-pairs.
speedseq align -R "@RG\tID:id\tSM:sample\tLB:lib" \
human_g1k_v37.fasta \
sample.1.fq \
sample.2.fq
Otherwise, data may be aligned with BWA-MEM.
# Align the data
bwa mem -R "@RG\tID:id\tSM:sample\tLB:lib" human_g1k_v37.fasta sample.1.fq sample.2.fq \
| samblaster --excludeDups --addMateTags --maxSplitCount 2 --minNonOverlap 20 \
| samtools view -S -b - \
> sample.bam
# Extract the discordant paired-end alignments.
samtools view -b -F 1294 sample.bam > sample.discordants.unsorted.bam
# Extract the split-read alignments
samtools view -h sample.bam \
| scripts/extractSplitReads_BwaMem -i stdin \
| samtools view -Sb - \
> sample.splitters.unsorted.bam
# Sort both alignments
samtools sort sample.discordants.unsorted.bam sample.discordants
samtools sort sample.splitters.unsorted.bam sample.splitters
Running LUMPY
LUMPY has two distinct execution alternatives. LUMPY Express is a simplified wrapper for standard analyses. LUMPY (traditional) is more customizable, for advanced users and specialized experiments.
LUMPY Express
-
Run LUMPY Express on a single sample with pre-extracted splitters and discordants
lumpyexpress \ -B sample.bam \ -S sample.splitters.bam \ -D sample.discordants.bam \ -o sample.vcf -
Run LUMPY Express jointly on multiple samples with pre-extracted splitters and discordants
lumpyexpress \ -B sample1.bam,sample2.bam,sample3.bam \ -S sample1.splitters.bam,sample2.splitters.bam,sample3.splitters.bam \ -D sample1.discordants.bam,sample2.discordants.bam,sample3.discordants.bam \ -o multi_sample.vcf -
Run LUMPY Express on a tumor-normal pair
lumpyexpress \ -B tumor.bam,normal.bam \ -S tumor.splitters.bam,normal.splitters.bam \ -D tumor.discordants.bam,normal.discordants.bam \ -o tumor_normal.vcf
LUMPY (traditional)
First, generate empirical insert size statistics on each library in the BAM file
samtools view -r readgroup1 sample.bam \
| tail -n+100000 \
| scripts/pairend_distro.py \
-r 101 \
-X 4 \
-N 10000 \
-o sample.lib1.histo
The above script (scripts/pairend_distro.py) will display mean and stdev to screen. For these examples we will assume the mean is 500 and the stdev is 50.
-
Run LUMPY with paired-end and split-reads.
lumpy \ -mw 4 \ -tt 0 \ -pe id:sample,bam_file:sample.discordants.bam,histo_file:sample.lib1.histo,mean:500,stdev:50,read_length:101,min_non_overlap:101,discordant_z:5,back_distance:10,weight:1,min_mapping_threshold:20 \ -sr id:sample,bam_file:sample.splitters.bam,back_distance:10,weight:1,min_mapping_threshold:20 \ > sample.vcf -
Run LUMPY on a BAM file with multiple libraries.
lumpy \ -mw 4 \ -tt 0 \ -pe id:sample,read_group:rg1,bam_file:sample.discordants.bam,histo_file:sample.lib1.histo,mean:500,stdev:50,read_length:101,min_non_overlap:101,discordant_z:5,back_distance:10,weight:1,min_mapping_threshold:20 \ -pe id:sample,read_group:rg2,bam_file:sample.discordants.bam,histo_file:sample.lib2.histo,mean:500,stdev:50,read_length:101,min_non_overlap:101,discordant_z:5,back_distance:10,weight:1,min_mapping_threshold:20 \ -sr id:sample,bam_file:sample.splitters.bam,back_distance:10,weight:1,min_mapping_threshold:20 \ > sample.vcf -
Run LUMPY on multiple samples with multiple libraries.
lumpy \ -mw 4 \ -tt 0 \ -pe id:sample1,bam_file:sample1.discordants.bam,read_group:rg1,read_group:rg2,histo_file:sample1.lib1.histo,mean:500,stdev:50,read_length:101,min_non_overlap:101,discordant_z:5,back_distance:10,weight:1,min_mapping_threshold:20 \ -pe id:sample1,bam_file:sample1.discordants.bam,read_group:rg3,histo_file:sample1.lib2.histo,mean:500,stdev:50,read_length:101,min_non_overlap:101,discordant_z:5,back_distance:10,weight:1,min_mapping_threshold:20 \ -pe id:sample2,bam_file:sample2.discordants.bam,read_group:rg4,histo_file:sample2.lib1.histo,mean:500,stdev:50,read_length:101,min_non_overlap:101,discordant_z:5,back_distance:10,weight:1,min_mapping_threshold:20 \ -sr id:sample1,bam_file:sample1.splitters.bam,back_distance:10,weight:1,min_mapping_threshold:20 \ -sr id:sample2,bam_file:sample2.splitters.bam,back_distance:10,weight:1,min_mapping_threshold:20 \ > multi_sample.vcf -
Run LUMPY with low complexity regions excluded.
Heng Li provides a set of low complexity regions in the supplementary information of his paper, "Toward better understanding of artifacts in variant calling from high-coverage samples" at https://doi.org/10.1093/bioinformatics/btu356.unzip btu356_Supplementary_Data.zip unzip btu356-suppl_data.zip lumpy \ -mw 4 \ -tt 0.0 \ -x btu356_LCR-hs37d5.bed/btu356_LCR-hs37d5.bed \ -pe bam_file:sample.discordants.bam,histo_file:sample.pe.histo,mean:500,stdev:50,read_length:101,min_non_overlap:101,discordant_z:5,back_distance:10,weight:1,id:sample,min_mapping_threshold:1 \ -sr bam_file:sample.sr.sort.bam,back_distance:10,weight:1,id:sample,min_mapping_threshold:1 \ > sample.exclude.vcf -
Run LUMPY with regions of very high coverage excluded.
We can direct lumpy to ignore certain regions by using the exclude region option. In this example we find and then exclude regions that have very high coverage. First we use the get_coverages.py script to find the min, max, and mean coverages of the the sr and pe bam files, and to create coverage profiles for both files.python ../scripts/get_coverages.py \ sample.pe.sort.bam \ sample.sr.sort.bam # sample.pe.sort.bam.coverage min:1 max:14 mean(non-zero):2.35557521272 # sample.sr.sort.bam.coverage min:1 max:7 mean(non-zero):1.08945936729From this output, we will choose to exclude regions that have more than 10x coverage. To create the exclude file we will use the get_exclude_regions.py script to create the exclude.bed file
python ../scripts/get_exclude_regions.py \ 10 \ exclude.bed \ sample.pe.sort.bam \ sample.sr.sort.bamWe now rerun lumpy with the exclude (-x) option
lumpy \ -mw 4 \ -tt 0.0 \ -x exclude.bed \ -pe bam_file:sample.discordants.bam,histo_file:sample.pe.histo,mean:500,stdev:50,read_length:101,min_non_overlap:101,discordant_z:5,back_distance:10,weight:1,id:sample,min_mapping_threshold:1 \ -sr bam_file:sample.sr.sort.bam,back_distance:10,weight:1,id:sample,min_mapping_threshold:1 \ > sample.exclude.vcf
Post-processing
SVTyper can call genotypes on LUMPY output VCF files using a Bayesian maximum likelihood algorithm.
svtyper \
-B sample.bam \
-S sample.splitters.bam \
-i sample.vcf
> sample.gt.vcf
Test data
The test/test.sh script executes lumpy against several simulated data sets
and compares the results to the known correct result. The sample data sets can
be found at http://layerlab.org/lumpy/data.tar.gz. This tar ball should be
extracted into the top-level lumpy directory. The script test/test.sh checks
for the the existence of this directory before running LUMPY.
Troubleshooting
All of the bam files that lumpy processes must be position sorted. To check if your bams are sorted correctly, use the check_sorting.py script
python ../scripts/check_sorting.py \
pe.pos_sorted.bam \
sr.pos_sorted.bam \
pe.name_sorted.bam
# pe.pos_sorted.bam
# in order
# sr.pos_sorted.bam
# in order
# pe.name_sorted.bam
# out of order: chr10 102292476 occurred after chr10 102292893