IRMA bug fixes & improvements; theiacov_illumina_pe wf updates for Flu

[kapsakcj]

Starting a draft while doing testing in Terra. Will update this message periodically

This PR closes #412 closes #437 and closes #457

🗑️ This dev branch should be deleted after merging to main.

🧠 Aim, Context and Functionality

The aim of this PR is to resolve a few different issues/bugs and make general improvements & upgrades to the TheiaCoV workflows for Flu analysis. Much of these changes impact the IRMA task, used in TheiaCoV_Illumina_PE wf, but some changes impact other workflows like TheiaCoV_ONT for Flu analysis.

🛠️ Impacted Workflows/Tasks & Changes Being Made

This will affect the behavior of the workflow(s) even if users don’t change any workflow inputs relative to the last version : Yes

Running this workflow on different occasions could result in different results, e.g. due to use of a live database, "latest" docker image, or stochastic data processing : No

📋 Workflow/Task Step Changes

🔄 Data Processing

• added many DEBUG statements throughout
• Fixed issue [TheiaCoV] IRMA task runs out of disk space due to default TMP directory #412 by use of irma_config.sh config file by setting $TMP used by IRMA to the present working directory. This impacted samples where FASTQ files were large and is no longer an issue

Docker/software or software versions changed: cdcgov/irma:v1.1.3 ➡️ "us-docker.pkg.dev/general-theiagen/cdcgov/irma:v1.1.5"

Databases or database versions changed: N/A

Data processing/commands changed: lots of changes to IRMA task & data processing

• Fixed usage of mafft --thread flag and updated how version is captured
• added --external-config irma_config.sh to IRMA command to use custom config file created in beginning of task.

File processing changed: lots of changes to output files & file renaming & FASTA header replacement

• FASTA with all segments now contains headers with ~{samplename}

# example where samplename == ERR11656083-FluB-ONT_B
$ grep '>' 20240509_155808_theiacov_ont/call-irma/work/ERR11656083-FluB-ONT.irma.consensus.fasta 
>ERR11656083-FluB-ONT_B_HA
>ERR11656083-FluB-ONT_B_MP
>ERR11656083-FluB-ONT_B_NA
>ERR11656083-FluB-ONT_B_NP
>ERR11656083-FluB-ONT_B_NS
>ERR11656083-FluB-ONT_B_PA
>ERR11656083-FluB-ONT_B_PB1
>ERR11656083-FluB-ONT_B_PB2

• Fix for empty HA segment for Flu B (Issue [IRMA] bug in output FASTA file for HA segment of Flu B samples is empty #437) on line 103. Adjusted sed command to modify file in place instead of redirecting into file.
  • Also applied same change to line 117 for renaming NA FASTA file when it includes the subtype number in output FASTA file name. This only applies to Flu A samples as B do not have subtype numbers int output HA and NA segment FASTA filenames.

Compute resources changed:

• IRMA task reduced default cpus to 4
  • Additionally - updated irma_config.sh file created in task to include 2 variables for multi-threading IRMA
• MAFFT task: reduced cpu to 2, memory to 8 and disk_size to 50

➡️ Inputs

• defaults for docker and cpu were changed

⬅️ Outputs

New IRMA task outputs:

• added seg_np_assembly & seq_ns_assembly which are the FASTA files corresponding with the NP (Nucleoprotein) and NS (nonstructural) segment
• String irma_docker
• String irma_subtype_notes which is either: IRMA does not differentiate Victoria and Yamagata Flu B lineages. See abricate_flu_subtype output column for Flu B samples and blank/empty for non Flu B samples.
  • (EXISTING BEHAVIOR, THIS OUTPUT HAS NOT CHANGED) String irma_subtype will either say H1N1 (or whatever Flu A subtype) for example with Flu A and for Flu B it will say No subtype predicted by IRMA
• adding these "padded" assemblies as outputs to be passed to VADR and MAFFT (antiviral substitutions tasks). "Padded" in this context means that periods . have been replaced with N's.

    File? irma_assembly_fasta_padded = "~{samplename}.irma.consensus.pad.fasta"
    File? seg_ha_assembly_padded = "~{samplename}_HA.pad.fasta"
    File? seg_na_assembly_padded = "~{samplename}_NA.pad.fasta"
    File? seg_pa_assembly_padded = "~{samplename}_PA.pad.fasta"
    File? seg_pb1_assembly_padded = "~{samplename}_PB1.pad.fasta"
    File? seg_pb2_assembly_padded = "~{samplename}_PB2.pad.fasta"
    File? seg_mp_assembly_padded = "~{samplename}_MP.pad.fasta"
    File? seg_np_assembly_padded = "~{samplename}_NP.pad.fasta"
    File? seg_ns_assembly_padded = "~{samplename}_NS.pad.fasta"

New TheiaCoV_Illumina_PE & ONT outputs:

• String irma_docker = irma.irma_docker
• String? irma_subtype_notes = irma.irma_subtype_notes
• File? irma_ha_segment_fasta = irma.seg_ha_assembly
• File? irma_na_segment_fasta = irma.seg_na_assembly
• File? irma_pa_segment_fasta = irma.seg_pa_assembly
• File? irma_pb1_segment_fasta = irma.seg_pb1_assembly
• File? irma_pb2_segment_fasta = irma.seg_pb2_assembly
• File? irma_mp_segment_fasta = irma.seg_mp_assembly
• File? irma_np_segment_fasta = irma.seg_np_assembly
• File? irma_ns_segment_fasta = irma.seg_ns_assembly

⚠️ NOTE: I have opted to NOT output the padded FASTA files to the workflow level (i.e. Terra output column) as it would add lots of clutter and likely confuse the user. They are saved as intermediate files during the IRMA task and can be retrieved from the execution directory if necessary.

🧪 Testing

Test Dataset

Described below.

Commandline Testing with MiniWDL or Cromwell (optional)

tested lots with miniwdl locally, but don't have output saved.

Terra Testing

• TheiaCoV_illumina_PE with H1N1, H3N2, B Victoria, B Yamagata (I don't have data available)
  • 96 Illumina PE samples with H1N1, H3N2, B Victoria, but NO Yamagata samples: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/f47caabd-8ca2-4fe7-8c2e-9d60eb6f5e35
  • ⚠️ Need to review outputs, especially the all-segment FASTA files & the HA segment FASTA in particular
  • outputs look good, HA FASTA files (especially the type B HA segment FASTAs) and all-segment FASTA files look good.
  • for B samples, nextclade datasets are appropriately being selected now that ABRicate-predicted subtype (Victoria or yamagata) is used appropriately to pick the correct nextclade datasets
• TheiaCoV_ONT with H1N1 & B Victoria samples: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-theiagen-validations/job_history/1128454d-f8e1-4558-9fe9-2c1903fea4c4
  • ⚠️ need to review outputs, especially the all-segment FASTA files & the HA segment FASTA in particular
  • outputs look good, HA FASTA files (especially the type B HA segment FASTAs) and all-segment FASTA files look good.
  • for B samples, nextclade datasets are appropriately being selected now that ABRicate-predicted subtype (Victoria or yamagata) is used appropriately to pick the correct nextclade datasets

Suggested Scenarios for Reviewer to Test

Would be good to test as many types/subtypes as possible and even test with poor quality data to see how the workflow behaves when IRMA cannot produce an assembly.

Need to test ONT as I've primarily been testing the IRMA WDL task updates with Illumina data. I tested 5 ONT samples, but would be good to do more if data is available

Theiagen Version Release Testing (optional)

🔬 Final Developer Checklist

• The workflow/task has been tested locally and results, including file contents, are as anticipated
• The workflow/task has been tested on Terra and results, including file contents, are as anticipated
• The CI/CD has been adjusted and tests are passing (to be completed by Theiagen developer)
• Code changes follow the style guide

🎯 Reviewer Checklist

• All impacted workflows/tasks have been tested on Terra with a different dataset than used for development
• All reviewer-suggested scenarios have been tested and any additional
• All changed results have been confirmed to be accurate
• All workflows/tasks impacted by change/s have been tested using a standard validation dataset to ensure no unintended change of functionality
• All code adheres to the style guide
• MD5 sums have been updated
• The PR author has addressed all comments

🗂️ Associated Documentation (to be completed by Theiagen developer)

• Relevant documentation on the Public Health Resources "PHB Main" has been updated
• Workflow diagrams have been updated to reflect changes. Updates to workflow diagram are not necessary

[kapsakcj]

leaving these lines commented out in case we want to revisit in the future when working on this task

[kapsakcj]

OK PR is ready for review.

I'll update the documentation and check the above box when I'm finished.

@sage-wright FYI there was one additional commit made after you forked your branch, you may want to merge that in if you are working on theiacov_ont workflow

[kapsakcj]

docs have been updated 👍 (mostly updates to theiacov inputs and outputs table)

[sage-wright]

TheiaProk_Illumina_PE here success
TheiaProk_ONT here success

[kapsakcj]

I relaunched Sage's tests since they were run prior to the last 2 commits. Same samples, call caching off, both run on the cjk-irma-diskcheck branch

• ILMN PE: https://app.terra.bio/#workspaces/theiagen-validations/sage_viral/job_history/eeff2739-4c20-48bb-9b02-f8677a94cfe2
• ONT: https://app.terra.bio/#workspaces/theiagen-validations/sage_viral/job_history/46e22cb8-846f-4a0b-b86d-826149d54c15

⚠️ Need to review outputs, esp. the sample that previously had a - in the output HA segment FASTA file and all segment FASTA file

[cimendes]

Code changes look solid! 🏅

New outputs present as expected:

Launching a new set of tests as I don't have access to Sage's workspace:

• Illumina PE (10 H1N1): https://app.terra.bio/#workspaces/theiagen-validations/Theiagen_Mendes_Sandbox/job_history/696deea2-d786-4c94-ab78-e11a7d01e0c2 ✅
• ONT (5 A and B): https://app.terra.bio/#workspaces/theiagen-validations/Theiagen_Mendes_Sandbox/job_history/b730d5d8-9d3d-4248-8e01-604e2e04b83b ✅

[cimendes]

My tests were successful and the workflow is working as expected. @jrotieno you're the main reviewer here but you got my okay!