Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for namespaced inputs again. #5317

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,14 @@
+ Fix a bug where zip files with directories could not be imported.
For example a zip with `a.wdl` and `b.wdl` could be imported but one with `sub_workflows/a.wdl`
and `imports/b.wdl` could not.


### WDL
+ Namespaces in inputs JSON files are handled differently. This only affects inputs for
tasks (or subworkflows) in subworkflows. For example `my_workflow.sub_wf_file.sub_wf_name.my_task.my_input`
will change to `my_workflow.sub_wf_name.my_task.my_input`. More information can be
found in [the documentation on inputs](https://cromwell.readthedocs.io/en/stable/Inputs).


## 48 Release Notes

### Womtool Graph for WDL 1.0
Expand Down
131 changes: 131 additions & 0 deletions docs/Inputs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
## Inputs

Inputs are specified as a JSON file. The JSON file can be passed to cromwell by using the
`-i` of `--inputs` command line flag. See the [command line documentation](CommandLine.md)
for more information. Check the [API documentation](api/RESTAPI.md) for information on how
to pass inputs to a cromwell server.

### Example

Given the following workflow

***hello.wdl***
```WDL
version 1.0
task hello {
input {
String addressee
String? optionalText
}
command {
echo "Hello ~{addressee}! Welcome to Cromwell . . . on Google Cloud!"
~{"echo " + optionalText}
}
output {
String message = read_string(stdout())
}
runtime {
docker: "ubuntu:latest"
}
}

workflow wf_hello {
input {
String addressee
}
call hello {
input:
addressee=addressee
}

output {
String message = hello.message
}
}
```

The following inputs file contains all the required inputs.
***hello.inputs***
```JSON
{
"wf_hello.addressee": "World"
}
```

The inputs file can also be used to specify task level inputs.

```json
{
"wf_hello.addressee": "World",
"wf_hello.hello.optionalText": "This is submitted into the task input directly."
}
```

### Advanced: namespacing in inputs files.

Cromwell uses namespacing to make sure all inputs are addressable in a comprehendable manner.
In the above example the `wf_hello` workflow has its own namespace. In this namespace reside:
+ `addressee`, an input string
+ `hello`, a task
+ `message`, an output string

In the inputs file all inputs are addressable by namespace. So `addressee` is accessible as
`wf_hello.addressee`. This also allows access to all inputs of the task `hello` such as
`wf_hello.hello.optionalText`.

If a more complicated workflow with lots of subworkflows is used the namespaces can be used as
well.

```wdl
version 1.0

import "subworkflow1.wdl" as wf_one
import "subworkflow2.wdl" as wf_two

workflow MainWorkflow {
input {
File dataFile
String method
}
call wf_one.Workflow as WorkflowOne {
input:
data = dataFile
}
call wf_two.Workflow as WorkflowTwo {
input:
data = WorkflowOne.analysedData,
method = method
}
output {
File processedData = WorkflowTwo.methodicallyAnalyzedData
File report = WorkflowTwo.report
}
}
```

The following inputs are required
```json
{
"MainWorkflow.dataFile": "/path/to/data",
"MainWorkflow.method": "thorough"
}
```

Using namespaces, subworkflow tasks can also be addressed.
```json
{
"MainWorkflow.dataFile": "/path/to/data",
"MainWorkflow.method": "thorough",
"MainWorkflow.WorkflowOne.nested_task.obscureParameter": 42,
"MainWorkflow.WorkflowTwo.summarize.reportType": "html"
}
```

This feature allows users to specify inputs for deeply nested subworkflows as well:
```json
{
"MainWorkflow.dataFile": "/path/to/data",
"MainWorkflow.method": "thorough",
"MainWorkflow.WorkflowOne.AnotherSubworkflow.DeeplyNestedWorkflow.analysisTask.kmerSize": 22
}
```
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,7 @@ object CallElementToGraphNode {
}

def supplyableInput(definition: Callable.InputDefinition): Boolean = {
!definition.isInstanceOf[FixedInputDefinitionWithDefault] &&
!definition.name.contains(".") // NB: Remove this check when sub-workflows allow pass-through task inputs
!definition.isInstanceOf[FixedInputDefinitionWithDefault]
}

def validInput(name: String, definition: Callable.InputDefinition): Boolean = {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,21 +32,25 @@
"CNVSomaticPairWorkflow.CollectCountsTumor.cpu": "Int? (optional)",
"CNVSomaticPairWorkflow.kernel_approximation_dimension": "Int? (optional)",
"CNVSomaticPairWorkflow.outlier_neutral_segment_copy_ratio_z_score_threshold": "Float? (optional)",
"CNVSomaticPairWorkflow.CNVOncotatorWorkflow.OncotateSegments.basename_called_file": "String? (optional)",
"CNVSomaticPairWorkflow.kernel_variance_copy_ratio": "Float? (optional)",
"CNVSomaticPairWorkflow.CallCopyRatioSegmentsTumor.use_ssd": "Boolean (optional, default = false)",
"CNVSomaticPairWorkflow.additional_args_for_oncotator": "String? (optional)",
"CNVSomaticPairWorkflow.format": "String? (optional)",
"CNVSomaticPairWorkflow.CNVOncotatorWorkflow.OncotateSegments.use_ssd": "Boolean (optional, default = false)",
"CNVSomaticPairWorkflow.DenoiseReadCountsTumor.use_ssd": "Boolean (optional, default = false)",
"CNVSomaticPairWorkflow.mem_gb_for_model_segments": "Int? (optional)",
"CNVSomaticPairWorkflow.mem_gb_for_plotting": "Int? (optional)",
"CNVSomaticPairWorkflow.min_total_allele_count": "Int? (optional)",
"CNVSomaticPairWorkflow.ref_fasta": "File",
"CNVSomaticPairWorkflow.ModelSegmentsNormal.use_ssd": "Boolean (optional, default = false)",
"CNVSomaticPairWorkflow.CNVOncotatorWorkflow.OncotateSegments.machine_mem_mb": "Int? (optional)",
"CNVSomaticPairWorkflow.num_burn_in_allele_fraction": "Int? (optional)",
"CNVSomaticPairWorkflow.smoothing_threshold_allele_fraction": "Float? (optional)",
"CNVSomaticPairWorkflow.ModelSegmentsTumor.cpu": "Int? (optional)",
"CNVSomaticPairWorkflow.mem_gb_for_collect_counts": "Int? (optional)",
"CNVSomaticPairWorkflow.ref_fasta_dict": "File",
"CNVSomaticPairWorkflow.CNVOncotatorWorkflow.OncotateSegments.disk_space_gb": "Int? (optional)",
"CNVSomaticPairWorkflow.minimum_contig_length": "Int? (optional)",
"CNVSomaticPairWorkflow.smoothing_threshold_copy_ratio": "Float? (optional)",
"CNVSomaticPairWorkflow.PlotDenoisedCopyRatiosNormal.output_dir": "String? (optional)",
Expand All @@ -72,6 +76,7 @@
"CNVSomaticPairWorkflow.PreprocessIntervals.use_ssd": "Boolean (optional, default = false)",
"CNVSomaticPairWorkflow.mem_gb_for_collect_allelic_counts": "Int? (optional)",
"CNVSomaticPairWorkflow.normal_bam": "File? (optional)",
"CNVSomaticPairWorkflow.CNVOncotatorWorkflow.OncotateSegments.cpu": "Int? (optional)",
"CNVSomaticPairWorkflow.CollectCountsNormal.cpu": "Int? (optional)",
"CNVSomaticPairWorkflow.CollectAllelicCountsTumor.use_ssd": "Boolean (optional, default = false)",
"CNVSomaticPairWorkflow.num_burn_in_copy_ratio": "Int? (optional)",
Expand Down