Pseudo PR - Do not merge!

.github/workflows/awsfulltest.yml

@@ -15,9 +15,6 @@ jobs:
     steps:
       - name: Launch workflow via tower
         uses: seqeralabs/action-tower-launch@v1
-        # TODO nf-core: You can customise AWS full pipeline tests as required
-        # Add full size test data (but still relatively small datasets for few samples)
-        # on the `test_full.config` test runs with only one set of parameters
         with:
           workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }}
           access_token: ${{ secrets.TOWER_ACCESS_TOKEN }}

.github/workflows/ci.yml

@@ -26,6 +26,15 @@ jobs:
         NXF_VER:
           - "22.10.1"
           - "latest-everything"
+        profile:
+          - "test"
+          - "test_cram"
+          - "test_chr"
+          - "test_no_bai"
+          - "test_no_crai"
+          - "test_no_stats"
+          - "test_no_qc"
+          - "test_collate_fast"
     steps:
       - name: Check out pipeline code
         uses: actions/checkout@v3
@@ -36,8 +45,5 @@ jobs:
           version: "${{ matrix.NXF_VER }}"
 
       - name: Run pipeline with test data
-        # TODO nf-core: You can customise CI pipeline run tests as required
-        # For example: adding multiple test runs with different parameters
-        # Remember that you can parallelise this by using strategy.matrix
         run: |
-          nextflow run ${GITHUB_WORKSPACE} -profile test,docker --outdir ./results
+          nextflow run ${GITHUB_WORKSPACE} -profile ${{ matrix.profile }},docker --outdir ./results

.gitignore

@@ -6,3 +6,5 @@ results/
 testing/
 testing*
 *.pyc
+*.fasta
+*.fai

CHANGELOG.md

@@ -3,14 +3,55 @@
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## v2.0.0dev - [date]
+## v2.0.0 - Annie Easley
 
 Initial release of nf-core/bamtofastq, created with the [nf-core](https://nf-co.re/) template.
 
 ### `Added`
 
+- [#49](https://github.com/nf-core/bamtofastq/pull/49) Add descriptions to main options
+- [#48](https://github.com/nf-core/bamtofastq/pull/48) Add igenomes
+- [#45](https://github.com/nf-core/bamtofastq/pull/45) Add `test.yml` files with md5sums
+- [#44](https://github.com/nf-core/bamtofastq/pull/44) DSL2 conversion
+
+### `Changed`
+
+- [#55](https://github.com/nf-core/bamtofastq/pull/55) Code review suggestions & formatting
+- [#54](https://github.com/nf-core/bamtofastq/pull/54) Code review changes
+- [#53](https://github.com/nf-core/bamtofastq/pull/53) Code review & updated modules
+- [#52](https://github.com/nf-core/bamtofastq/pull/52) Code review changed resources in configs
+- [#47](https://github.com/nf-core/bamtofastq/pull/47) Sync TEMPLATE with tools 2.8
+
 ### `Fixed`
 
+- [#49](https://github.com/nf-core/bamtofastq/pull/49) Fixed release version
+- [#45](https://github.com/nf-core/bamtofastq/pull/45) Minor bugfix with chromosome extraction
+
 ### `Dependencies`
 
+| Dependency | Old version | New version |
+| ---------- | ----------- | ----------- |
+| `multiqc`  | 1.9         | 1.14        |
+| `samtools` | 1.10        | 1.17        |
+
 ### `Deprecated`
+
+- Option `--cram_files` not needed anymore due to automatic format detection.
+
+## v1.2.0 - Anna Winlock
+
+- [#36](https://github.com/qbic-pipelines/bamtofastq/pull/36) Add options `--cram_files` and `--reference_fasta` to add support for CRAM files.
+- [#31](https://github.com/qbic-pipelines/bamtofastq/pull/31) Add option `--samtools_collate_fast` and improve speed of cat.
+- [#32](https://github.com/qbic-pipelines/bamtofastq/pull/32) Added `--samtools_collate_fast` to sortExtractMapped and changed cat command to append.
+- [#33](https://github.com/qbic-pipelines/bamtofastq/pull/33) Added flag `--reads_in_memory` to specify how many reads shall be stored in memory.
+
+## v1.1.0 - Katherine Johnson
+
+- [#21](https://github.com/qbic-pipelines/bamtofastq/21) Allows bam indices as additional input files
+- [#23](https://github.com/qbic-pipelines/bamtofastq/23) Fix documentation: `--bam` is `--input` now
+- [#23](https://github.com/qbic-pipelines/bamtofastq/23) Make stats computation optional with parameter `--no_stats`. Use at own risk.
+- [#26](https://github.com/qbic-pipelines/bamtofastq/26) `BAM` index is now also used for processes dealing with separating reads based on the mapping status of both mates
+
+## v1.0.0 - Ada Lovelace
+
+Initial release of qbic-pipelines/bamtofastq, created with the [nf-core](http://nf-co.re/) template.

CITATIONS.md

@@ -12,9 +12,18 @@
 
 - [FastQC](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/)
 
+  > Andrews, S. (2010). FastQC: A Quality Control Tool for High Throughput Sequence Data [Online]. Available online https://www.bioinformatics.babraham.ac.uk/projects/fastqc/.
+
 - [MultiQC](https://pubmed.ncbi.nlm.nih.gov/27312411/)
+
   > Ewels P, Magnusson M, Lundin S, Käller M. MultiQC: summarize analysis results for multiple tools and samples in a single report. Bioinformatics. 2016 Oct 1;32(19):3047-8. doi: 10.1093/bioinformatics/btw354. Epub 2016 Jun 16. PubMed PMID: 27312411; PubMed Central PMCID: PMC5039924.
 
+- [SAMtools](https://doi.org/10.1093/gigascience/giab008)
+
+  > Danecek, P., Bonfield, J. K., Liddle, J., Marshall, J., Ohan, V., Pollard, M. O., Whitwham, A., Keane, T., McCarthy, S. A., Davies, R. D., Li, H., (2021) Twelve years of SAMtools and BCFtools, GigaScience, Volume 10, Issue 2, giab008. doi: 10.1093/gigascience/giab008
+
+  > Li H, Handsaker B, Wysoker A, Fennell T, Ruan J, Homer N, Marth G, Abecasis G, Durbin R; 1000 Genome Project Data Processing Subgroup. The Sequence Alignment/Map format and SAMtools. Bioinformatics. 2009 Aug 15;25(16):2078-9. doi: 10.1093/bioinformatics/btp352. Epub 2009 Jun 8. PubMed PMID: 19505943; PubMed Central PMCID: PMC2723002.
+
 ## Software packaging/containerisation tools
 
 - [Anaconda](https://anaconda.com)
@@ -31,5 +40,8 @@
 
 - [Docker](https://dl.acm.org/doi/10.5555/2600239.2600241)
 
+  > Merkel, D. (2014). Docker: lightweight linux containers for consistent development and deployment. Linux Journal, 2014(239), 2. doi: 10.5555/2600239.2600241.
+
 - [Singularity](https://pubmed.ncbi.nlm.nih.gov/28494014/)
+
   > Kurtzer GM, Sochat V, Bauer MW. Singularity: Scientific containers for mobility of compute. PLoS One. 2017 May 11;12(5):e0177459. doi: 10.1371/journal.pone.0177459. eCollection 2017. PubMed PMID: 28494014; PubMed Central PMCID: PMC5426675.

README.md

@@ -1,6 +1,6 @@
 # ![nf-core/bamtofastq](docs/images/nf-core-bamtofastq_logo_light.png#gh-light-mode-only) ![nf-core/bamtofastq](docs/images/nf-core-bamtofastq_logo_dark.png#gh-dark-mode-only)
 
-[![AWS CI](https://img.shields.io/badge/CI%20tests-full%20size-FF9900?labelColor=000000&logo=Amazon%20AWS)](https://nf-co.re/bamtofastq/results)[![Cite with Zenodo](http://img.shields.io/badge/DOI-10.5281/zenodo.XXXXXXX-1073c8?labelColor=000000)](https://doi.org/10.5281/zenodo.XXXXXXX)
+[![AWS CI](https://img.shields.io/badge/CI%20tests-full%20size-FF9900?labelColor=000000&logo=Amazon%20AWS)](https://nf-co.re/bamtofastq/results)[![Cite with Zenodo](http://img.shields.io/badge/DOI-10.5281/zenodo.4022138-1073c8?labelColor=000000)](https://doi.org/10.5281/zenodo.4022138)
 
 [![Nextflow](https://img.shields.io/badge/nextflow%20DSL2-%E2%89%A522.10.1-23aa62.svg)](https://www.nextflow.io/)
 [![run with conda](http://img.shields.io/badge/run%20with-conda-3EB049?labelColor=000000&logo=anaconda)](https://docs.conda.io/en/latest/)
@@ -12,20 +12,26 @@
 
 ## Introduction
 
-**nf-core/bamtofastq** is a bioinformatics pipeline that ...
+**nf-core/bamtofastq** is a bioinformatics best-practice analysis pipeline that converts (un)mapped `.bam` files or `.cram` files into `fq.gz` files. Initially, it auto-detects, whether the input file contains single-end or paired-end reads. Following this step, the reads are sorted using `samtools collate` and extracted with `samtools fastq`. Furthermore, for mapped bam/cram files it is possible to only convert reads mapping to a specific region or chromosome. The obtained FastQ files can then be used to further process with other pipelines.
 
-<!-- TODO nf-core:
-   Complete this sentence with a 2-3 sentence summary of what types of data the pipeline ingests, a brief overview of the
-   major pipeline sections and the types of output it produces. You're giving an overview to someone new
-   to nf-core here, in 15-20 seconds. For an example, see https://github.com/nf-core/rnaseq/blob/master/README.md#introduction
--->
+The pipeline is built using [Nextflow](https://www.nextflow.io), a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It uses Docker/Singularity containers making installation trivial and results highly reproducible. The [Nextflow DSL2](https://www.nextflow.io/docs/latest/dsl2.html) implementation of this pipeline uses one container per process which makes it much easier to maintain and update software dependencies. Where possible, these processes have been submitted to and installed from [nf-core/modules](https://github.com/nf-core/modules) in order to make them available to all nf-core pipelines, and to everyone within the Nextflow community!
 
-<!-- TODO nf-core: Include a figure that guides the user through the major workflow steps. Many nf-core
-     workflows use the "tube map" design for that. See https://nf-co.re/docs/contributing/design_guidelines#examples for examples.   -->
-<!-- TODO nf-core: Fill in short bullet-pointed list of the default steps in the pipeline -->
+On release, automated continuous integration tests run the pipeline on a full-sized dataset on the AWS cloud infrastructure. This ensures that the pipeline runs on AWS, has sensible resource allocation defaults set to run on real-world datasets, and permits the persistent storage of results to benchmark between pipeline releases and other analysis sources.The results obtained from the full-sized test can be viewed on the [nf-core website](https://nf-co.re/bamtofastq/results).
 
-1. Read QC ([`FastQC`](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/))
-2. Present QC for raw reads ([`MultiQC`](http://multiqc.info/))
+## Pipeline summary
+
+By default, the pipeline currently performs the following steps:
+
+1. Quality control (QC) of input (bam/cram) files ([`FastQC`](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/)).
+2. Check if input files are single- or paired-end ([`Samtools`](https://www.htslib.org/)).
+3. Compute statistics on input files ([`Samtools`](https://www.htslib.org/)).
+4. Convert to fastq reads ([`Samtools`](https://www.htslib.org/)).
+5. QC of converted fastq reads ([`FastQC`](https://www.bioinformatics.babraham.ac.uk/projects/fastqc/)).
+6. Summarize QC and statistics before and after format conversion ([`MultiQC`](http://multiqc.info/)).
+
+<p align="center">
+    <img title="Bamtofastq Workflow" src="docs/images/nf-core-bamtofastq-subway.png" width=60%>
+</p>
 
 ## Usage
 
@@ -34,26 +40,26 @@
 > to set-up Nextflow. Make sure to [test your setup](https://nf-co.re/docs/usage/introduction#how-to-run-a-pipeline)
 > with `-profile test` before running the workflow on actual data.
 
-<!-- TODO nf-core: Describe the minimum required steps to execute the pipeline, e.g. how to prepare samplesheets.
-     Explain what rows and columns represent. For instance (please edit as appropriate):
+Download the pipeline and test it on a minimal dataset with a single command:
 
-First, prepare a samplesheet with your input data that looks as follows:
+```bash
+nextflow run nf-core/bamtofastq -profile test,<docker/singularity/.../institute> --outdir './results'
+```
+
+To run your own analysis, start by preparing a samplesheet with your input data that looks as follows:
 
 `samplesheet.csv`:
 
 ```csv
-sample,fastq_1,fastq_2
-CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz
+sample_id,mapped,index,file_type
+test,test1.bam,test1.bam.bai,bam
+test2,test2.bam,test2.bam.bai,bam
 ```
 
-Each row represents a fastq file (single-end) or a pair of fastq files (paired end).
-
--->
+Each row represents a bam/cram file with or without indices.
 
 Now, you can run the pipeline using:
 
-<!-- TODO nf-core: update the following command to include all required parameters for a minimal example -->
-
 ```bash
 nextflow run nf-core/bamtofastq \
    -profile <docker/singularity/.../institute> \
@@ -68,19 +74,25 @@ nextflow run nf-core/bamtofastq \
 
 For more details, please refer to the [usage documentation](https://nf-co.re/bamtofastq/usage) and the [parameter documentation](https://nf-co.re/bamtofastq/parameters).
 
-## Pipeline output
-
-To see the the results of a test run with a full size dataset refer to the [results](https://nf-co.re/bamtofastq/results) tab on the nf-core website pipeline page.
-For more details about the output files and reports, please refer to the
-[output documentation](https://nf-co.re/bamtofastq/output).
+The nf-core/bamtofastq pipeline comes with documentation about the pipeline [usage](https://nf-co.re/bamtofastq/usage), [parameters](https://nf-co.re/bamtofastq/parameters) and [output](https://nf-co.re/bamtofastq/output).
 
 ## Credits
 
-nf-core/bamtofastq was originally written by Friederike Hanssen, Susanne Jodoin.
+nf-core/bamtofastq was originally written by Friederike Hanssen. It was ported to DSL2 by Susanne Jodoin.
 
 We thank the following people for their extensive assistance in the development of this pipeline:
 
-<!-- TODO nf-core: If applicable, make list of people who have also contributed -->
+- [Gisela Gabernet](https://github.com/ggabernet)
+- [Matilda Åslin](https://github.com/matrulda)
+- [Susanne Jodoin](https://github.com/SusiJo)
+- [Bruno Grande](https://github.com/BrunoGrandePhd)
+
+### Resources
+
+The individual steps of this pipeline are based of on the following tutorials and resources:
+
+1.  [Extracting paired FASTQ read data from a BAM mapping file](http://darencard.net/blog/2017-09-07-extract-fastq-bam/)
+2.  [Check if BAM is derived from pair-end or single-end reads](https://www.biostars.org/p/178730/)
 
 ## Contributions and Support
 
@@ -90,10 +102,7 @@ For further information or help, don't hesitate to get in touch on the [Slack `#
 
 ## Citations
 
-<!-- TODO nf-core: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file. -->
-<!-- If you use  nf-core/bamtofastq for your analysis, please cite it using the following doi: [10.5281/zenodo.XXXXXX](https://doi.org/10.5281/zenodo.XXXXXX) -->
-
-<!-- TODO nf-core: Add bibliography of tools and data used in your pipeline -->
+If you use nf-core/bamtofastq for your analysis, please cite it using the following doi: [10.5281/zenodo.4022138](https://doi.org/10.5281/zenodo.4022138)
 
 An extensive list of references for the tools used by the pipeline can be found in the [`CITATIONS.md`](CITATIONS.md) file.
 

assets/methods_description_template.yml

@@ -3,8 +3,6 @@ description: "Suggested text and references to use when describing pipeline usag
 section_name: "nf-core/bamtofastq Methods Description"
 section_href: "https://github.com/nf-core/bamtofastq"
 plot_type: "html"
-## TODO nf-core: Update the HTML below to your prefered methods description, e.g. add publication citation for this pipeline
-## You inject any metadata in the Nextflow '${workflow}' object
 data: |
   <h4>Methods</h4>
   <p>Data was processed using nf-core/bamtofastq v${workflow.manifest.version} ${doi_text} of the nf-core collection of workflows (<a href="https://doi.org/10.1038/s41587-020-0439-x">Ewels <em>et al.</em>, 2020</a>).</p>
@@ -13,7 +11,16 @@ data: |
   <h4>References</h4>
   <ul>
     <li>Di Tommaso, P., Chatzou, M., Floden, E. W., Barja, P. P., Palumbo, E., & Notredame, C. (2017). Nextflow enables reproducible computational workflows. Nature Biotechnology, 35(4), 316-319. <a href="https://doi.org/10.1038/nbt.3820">https://doi.org/10.1038/nbt.3820</a></li>
-    <li>Ewels, P. A., Peltzer, A., Fillinger, S., Patel, H., Alneberg, J., Wilm, A., Garcia, M. U., Di Tommaso, P., & Nahnsen, S. (2020). The nf-core framework for community-curated bioinformatics pipelines. Nature Biotechnology, 38(3), 276-278. <a href="https://doi.org/10.1038/s41587-020-0439-x">https://doi.org/10.1038/s41587-020-0439-x</a></li>
+    <li>Ewels, P. A., Peltzer, A., Fillinger, S., Patel, H., Alneberg, J., Wilm, A., Garcia, M. U., Di Tommaso, P., & Nahnsen, S. (2020). The nf-core framework for community-curated bioinformatics pipelines. Nature Biotechnology, 38(3), 276-278. <br><a href="https://doi.org/10.1038/s41587-020-0439-x">https://doi.org/10.1038/s41587-020-0439-x</a></li>
+    <li>Hanssen, F. (2022) nf-core/bamtofastq <a href="https://doi.org/10.5281/zenodo.4022138">https://doi.org/10.5281/zenodo.4022138</a></li>
+    <li>Andrews, S. (2010). FastQC:  A Quality Control Tool for High Throughput Sequence Data [Online]. Available online <a href="https://www.bioinformatics.babraham.ac.uk/projects/fastqc/">https://www.bioinformatics.babraham.ac.uk/projects/fastqc/</a></li>
+    <li>Ewels, P. A., Magnusson, M., Lundin, S., Käller, S., MultiQC: summarize analysis results for multiple tools and samples in a single report, Bioinformatics, Volume 32, Issue 19, October 2016, Pages 3047–3048. <a href="https://doi.org/10.1093/bioinformatics/btw354">https://doi.org/10.1093/bioinformatics/btw354</a></li>
+    <li>Danecek, P., Bonfield, J. K., Liddle, J., Marshall, J., Ohan, V., Pollard, M. O., Whitwham, A., Keane, T., McCarthy, S. A., Davies, R. D., Li, H., (2021) Twelve years of SAMtools and BCFtools, GigaScience, Volume 10, Issue 2, giab008. <a href="https://doi.org/10.1093/gigascience/giab008">https://doi.org/10.1093/gigascience/giab008</a></li>
+    <li>Anaconda Software Distribution. (2020). Anaconda Documentation. Anaconda Inc. Retrieved from <a href="https://anaconda.com">https://anaconda.com</a></li>
+    <li>Grüning, B., Dale, R., Sjödin, A., Chapman, B. A., Rowe, J., Tomkins-Tinch, C. H., Valieris, R., Köster, J., & Bioconda Team (2018). Bioconda: sustainable and comprehensive software distribution for the life sciences. Nature methods, 15(7), 475–476.  <a href="https://doi.org/10.1038/s41592-018-0046-7">https://doi.org/10.1038/s41592-018-0046-7</a></li>
+    <li>da Veiga Leprevost F., Grüning B., Aflitos S. A., Röst H. L., Uszkoreit J., Barsnes H., Vaudel M., Moreno P., Gatto L., Weber J., Bai M., Jimenez R. C., Sachsenberg T., Pfeuffer J., Alvarez R. V., Griss J., Nesvizhskii A. I., Perez-Riverol Y. (2017). BioContainers: an open-source and community-driven framework for software standardization. Bioinformatics (Oxford, England), 33(16), 2580–2582. <a href="https://doi.org/10.1093/bioinformatics/btx192">https://doi.org/10.1093/bioinformatics/btx192</a></li>
+    <li>Merkel, D. (2014). Docker: lightweight linux containers for consistent development and deployment. Linux Journal, 2014(239), 2. <a href="https://dl.acm.org/doi/10.5555/2600239.2600241">https://dl.acm.org/doi/10.5555/2600239.2600241</a></li>
+    <li>Kurtzer GM., Sochat V., Bauer MW. (2017). Singularity: Scientific containers for mobility of compute. PLoS One, 12(5):e0177459. <a href="https://doi.org/10.1371/journal.pone.0177459">https://doi.org/10.1371/journal.pone.0177459</a></li>
   </ul>
   <div class="alert alert-info">
     <h5>Notes:</h5>

assets/multiqc_config.yml

@@ -11,3 +11,5 @@ report_section_order:
     order: -1002
 
 export_plots: true
+
+fn_clean_sample_names: false

assets/samplesheet.csv

@@ -1,3 +1,3 @@
-sample,fastq_1,fastq_2
-SAMPLE_PAIRED_END,/path/to/fastq/files/AEG588A1_S1_L002_R1_001.fastq.gz,/path/to/fastq/files/AEG588A1_S1_L002_R2_001.fastq.gz
-SAMPLE_SINGLE_END,/path/to/fastq/files/AEG588A4_S4_L003_R1_001.fastq.gz,
+sample_id,mapped,index,file_type
+test,https://github.com/nf-core/test-datasets/raw/bamtofastq/test-datasets/data/genomics/homo_sapiens/illumina/bam/test.paired_end.sorted.bam,https://github.com/nf-core/test-datasets/raw/bamtofastq/test-datasets/data/genomics/homo_sapiens/illumina/bam/test.paired_end.sorted.bam.bai,bam
+test2,https://github.com/nf-core/test-datasets/raw/bamtofastq/test-datasets/data/genomics/homo_sapiens/illumina/bam/test2.paired_end.sorted.bam,https://github.com/nf-core/test-datasets/raw/bamtofastq/test-datasets/data/genomics/homo_sapiens/illumina/bam/test2.paired_end.sorted.bam.bai,bam