- About Singularity
- How using Singularity
- Where finding Singularity images : BioContainer
- Docker 2 Singularity
- A Full Example
- The script slurm/sbatch
- sbatch
- Result
About Singularity
Singularity is a free, cross-platform and open-source computer program that performs operating-system-level virtualization also known as containerization.
One of the main uses of Singularity is to bring containers and reproducibility to scientific computing and the high-performance computing (HPC) world.
The need for reproducibility requires the ability to use containers to move applications from system to system.
Using Singularity containers, developers can work in reproducible environments of their choosing and design, and these complete environments can easily be copied and executed on other platforms.
(source)
How using Singularity
$ srun singularity run shub://GodloveD/lolcow
Progress |===================================| 100.0%
________________________________________
/ RTFM: \
\ Read The Fucking Manual /
----------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
The documentation is here: User Guide
A selection for survival: - Executing Commands - Working with Files
Where finding Singularity images : BioContainer
BioContainers is a community-driven project that provides the infrastructure and basic guidelines to create, manage and distribute bioinformatics packages (e.g conda) and containers (e.g docker, singularity).
BioContainers is based on the popular frameworks Conda, Docker and Singularity. (source)
For each BioConda packages, a Docker container is build and push on Quay.io and DockerHub (example: Bioconda/Fastqc)
(source)
BioContainer repositories:
- Quay.io: https://quay.io/repository/biocontainers/fastqc
- DockerHub: https://hub.docker.com/r/biocontainers/fastqc
Docker 2 Singularity
Singularity accept to run Docker containers
$ singularity pull docker://godlovedc/lolcow
$ srun ./lolcow_latest.sif
____________________________________
/ You're currently going through a \
| difficult transition period called |
\ "Life." /
------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||A Full Example
The script slurm/sbatch
singularity_bowtie2.slurm
#!/bin/bash
#SBATCH -o slurm.%N.%j.out
#SBATCH -e slurm.%N.%j.err
#SBATCH --partition fast
#SBATCH --cpus-per-task 2
#SBATCH --mem 10GB
SIMAGES=$PWD/singularity_images
#### Do it once
#mkdir -p $SIMAGES; cd $SIMAGES
#singularity pull docker://quay.io/biocontainers/sickle-trim:1.33--hed695b0_5
#singularity pull docker://quay.io/biocontainers/bowtie2:2.4.1--py37h8270d21_3
#singularity pull docker://quay.io/biocontainers/samtools:1.6--h9dace67_6
#cd -
mkdir fastq
wget http://denis.puthier.perso.luminy.univ-amu.fr/siNT_ER_E2_r3_chr21.fastq.gz
gunzip siNT_ER_E2_r3_chr21.fastq.gz
mv siNT_ER_E2_r3_chr21.fastq fastq
mkdir trimmed
$SIMAGES/sickle-trim_1.33--hed695b0_5.sif sickle se -f fastq/siNT_ER_E2_r3_chr21.fastq -t sanger -o trimmed/siNT_ER_E2_r3_chr21_trim.fastq
mkdir index
wget http://hgdownload.soe.ucsc.edu/goldenPath/hg19/chromosomes/chr21.fa.gz
gunzip chr21.fa.gz
mv chr21.fa index
$SIMAGES/bowtie2_2.4.1--py37h8270d21_3.sif bowtie2-build index/chr21.fa index/chr21_hg19
mkdir bam
$SIMAGES/bowtie2_2.4.1--py37h8270d21_3.sif bowtie2 --threads $SLURM_CPUS_PER_TASK -x index/chr21_hg19 -U trimmed/siNT_ER_E2_r3_chr21_trim.fastq -S bam/siNT_ER_E2_r3_chr21_trim.sam
$SIMAGES/samtools_1.6--h9dace67_6.sif samtools view -hbS -q 30 bam/siNT_ER_E2_r3_chr21_trim.sam | $SIMAGES/samtools_1.6--h9dace67_6.sif samtools sort > bam/siNT_ER_E2_r3_chr21_trim.bam
$SIMAGES/samtools_1.6--h9dace67_6.sif samtools index bam/siNT_ER_E2_r3_chr21_trim.bam
$SIMAGES/samtools_1.6--h9dace67_6.sif samtools view -h bam/siNT_ER_E2_r3_chr21_trim.bam | head -20
sbatch
sbatch singularity_bowtie2.slurmResult
$ cat slurm.*.out
@HD VN:1.0 SO:coordinate
@SQ SN:chr21 LN:48129895
@PG ID:bowtie2 PN:bowtie2 VN:2.3.3.1 CL:"/usr/local/bin/bowtie2-align-s --wrapper basic-0 -x /data/index/chr21_hg19 -S /data/bam/siNT_ER_E2_r3_chr21_trim.sam -U /data/trimmed/siNT_ER_E2_r3_chr21_trim.fastq"
SRR540192.3916166 16 chr21 9411377 42 37M * 0 0 CTCTGTTCCTTGCTGACCTCCAATTTTGTCTATCCTT GHFFFEGBHHHHHBHHHHHHHHFFDEHHHHHHHHHHH AS:i:0 XN:i:0 XM:i:0 XO:i:0 XG:i:0 NM:i:0 MD:Z:37 YT:Z:UU
SRR540192.7673103 0 chr21 9411551 42 37M * 0 0 CCGGCCCTTTATCAAGCCCTGGCCACCATGATAGTCA HHHHFHHHHHHHHHHDHHHHGGGGFHGDHHGGGGEGG AS:i:0 XN:i:0 XM:i:0 XO:i:0 XG:i:0 NM:i:0 MD:Z:37 YT:Z:UU
SRR540192.25047223 16 chr21 9411593 42 37M * 0 0 TCCAATTGTTGTCTATGCAGGCCTACCAGATTTCTAA IIEIIIIDIIIIIIIIIIIIIIIIIIIIIIIIIIIII AS:i:0 XN:i:0 XM:i:0 XO:i:0 XG:i:0 NM:i:0 MD:Z:37 YT:Z:UU
SRR540192.11503770 0 chr21 9411670 42 37M * 0 0 CAAATGTATCCAAATGAAAGGCTGTGGAGAATGTTGA IIIIIIIIIIIIIIIIIIIIIIIIGIIIIHIFHIHII AS:i:0 XN:i:0 XM:i:0 XO:i:0 XG:i:0 NM:i:0 MD:Z:37 YT:Z:UU
SRR540192.21957275 0 chr21 9411756 42 37M * 0 0 TCTGGATGCTTTGATTGCTATCAGAAGCCGTTAAATA IIIIIIIIIIIIIIIIIIIHIIIHHIIIIIIIIIIHG AS:i:0 XN:i:0 XM:i:0 XO:i:0 XG:i:0 NM:i:0 MD:Z:37 YT:Z:UU
SRR540192.2371243 0 chr21 9412286 42 37M * 0 0 CATAAAATGAGTTCTAGAGTTTATTTCTTTACTGCAT GDG<GEAAEA=5;==DDBB?EBED@=@E@@ADBGG>< AS:i:0 XN:i:0 XM:i:0 XO:i:0 XG:i:0 NM:i:0 MD:Z:37 YT:Z:UU
SRR540192.16647265 0 chr21 9412303 42 37M * 0 0 AGTTTATTTCTTTACTGCATCATTCTATTTTCAAGTC IIIIIIIIIIIIIIIIIIIIIIIIIIHIIIIIIIIII AS:i:0 XN:i:0 XM:i:0 XO:i:0 XG:i:0 NM:i:0 MD:Z:37 YT:Z:UU
SRR540192.8808648 16 chr21 9412397 42 37M * 0 0 TTCATATTTTATTTTTTATTTACTGTATAATTCAGTA IIIHHIGIIIGIIIIIGHIHHIIIIIIIIIIIIIIII AS:i:0 XN:i:0 XM:i:0 XO:i:0 XG:i:0 NM:i:0 MD:Z:37 YT:Z:UU
SRR540192.24964502 0 chr21 9412572 42 37M * 0 0 CAAATTTGGAAAACTGGAAAAAATATACATGGCAACA IIIIIIIIIIIIIIIIIGIIIIIIIIIIIIIIIIIII AS:i:0 XN:i:0 XM:i:0 XO:i:0 XG:i:0 NM:i:0 MD:Z:37 YT:Z:UU
SRR540192.14173679 0 chr21 9412633 42 37M * 0 0 CAAACAACTATAAATATTGTTCCACCCAAACAACTAT IIIIIIIIIIIIIIHIIIHIII3IIIIIIIIIIIIHH AS:i:-3 XN:i:0 XM:i:1 XO:i:0 XG:i:0 NM:i:1 MD:Z:22T14 YT:Z:UU
SRR540192.28855342 0 chr21 9412728 42 37M * 0 0