From eca2b36c07b4b0096e836f771256ce108ef9808f Mon Sep 17 00:00:00 2001 From: Michael Franklin Date: Tue, 2 Apr 2019 14:12:49 +1100 Subject: [PATCH 1/7] Add runtime attribute override to dev spec --- versions/development/SPEC.md | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/versions/development/SPEC.md b/versions/development/SPEC.md index 36698d61..b0dfc145 100644 --- a/versions/development/SPEC.md +++ b/versions/development/SPEC.md @@ -91,6 +91,8 @@ Table of Contents * [Computing Task Inputs](#computing-task-inputs) * [Computing Workflow Inputs](#computing-workflow-inputs) * [Specifying Workflow Inputs in JSON](#specifying-workflow-inputs-in-json) + * [Specifying / Overriding runtime attributes in JSON](#specifying--overriding-runtime-attributes-in-json) + * [Resolution Order](#resolution-order) * [Optional Inputs](#optional-inputs) * [Declared Inputs: Defaults and Overrides](#declared-inputs-defaults-and-overrides) * [Optional Inputs with Defaults](#optional-inputs-with-defaults) @@ -2646,6 +2648,30 @@ In JSON, the inputs to the workflow in the previous section might be: It's important to note that the type in JSON must be coercible to the WDL type. For example `wf.int_val` expects an integer, but if we specified it in JSON as `"wf.int_val": "three"`, this coercion from string to integer is not valid and would result in a coercion error. See the section on [Type Coercion](#type-coercion) for more details. +## Specifying / Overriding Runtime Attributes in JSON + +Workflow runtime attributes may additionally be specified as key/value pairs within the JSON input. The mapping from JSON or YAML values to WDL values is codified similar to the serialization of task inputs section, however with an additional runtime tag. The runtime attribute does not need to be specified in the task defintion to be overidden by the JSON. + +In JSON, the user may be able to specify task-specific runtime attributes similar to the following: + +```json +{ + "wf.t1.runtime.memory": "16 GB", + "wf.t2.runtime.cpus": 2, + "wf.t2.runtime.disks": "100 GB", + "wf.t3.runtime.arbitrary_key": ["arbitrary", "value"] +} +``` + +As the runtime section consists of key/value pairs, it's up to user to ensure they provide the correct coercible type for the backend they are targeting. See the section on [Type Coercion](#type-coercion) for more details. + +### Resolution Order + +Similar to how inputs are resolved, these runtime parameters are resolved in the following order: + +1. Inputs provided by the inputs JSON +2. The value provided inline on the task, whether that's an expression or literal value. + # Type Coercion WDL values can be created from either JSON values or from native language values. The below table references String-like, Integer-like, etc to refer to values in a particular programming language. For example, "String-like" could mean a `java.io.String` in the Java context or a `str` in Python. An "Array-like" could refer to a `Seq` in Scala or a `list` in Python. @@ -3639,4 +3665,3 @@ task test { This task would assign the one key-value pair map in the echo statement to `my_map`. If the echo statement was instead `echo '["foo", "bar"]'`, the engine MUST fail the task for a type mismatch. - From b413af00d8118652cd9eea5f806f42255b05f614 Mon Sep 17 00:00:00 2001 From: Michael Franklin Date: Wed, 3 Apr 2019 10:15:32 +1100 Subject: [PATCH 2/7] Clarify runtime attributes + no JSON expressions --- versions/development/SPEC.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/versions/development/SPEC.md b/versions/development/SPEC.md index b0dfc145..2c852be3 100644 --- a/versions/development/SPEC.md +++ b/versions/development/SPEC.md @@ -2650,7 +2650,7 @@ It's important to note that the type in JSON must be coercible to the WDL type. ## Specifying / Overriding Runtime Attributes in JSON -Workflow runtime attributes may additionally be specified as key/value pairs within the JSON input. The mapping from JSON or YAML values to WDL values is codified similar to the serialization of task inputs section, however with an additional runtime tag. The runtime attribute does not need to be specified in the task defintion to be overidden by the JSON. +Workflow runtime attributes may additionally be specified as key/value pairs within a JSON input file. Ideally this should be a separate `runtime.json` file to The mapping from JSON or YAML values to WDL values is codified similar to the serialization of task inputs section, however with an additional runtime tag to avoid name collisions and allow for unspecified attributes to be set by the runtime input file. Runtime attributes do not need to be specified in the task defintion to be overidden or set by the JSON. In JSON, the user may be able to specify task-specific runtime attributes similar to the following: @@ -2663,7 +2663,7 @@ In JSON, the user may be able to specify task-specific runtime attributes simila } ``` -As the runtime section consists of key/value pairs, it's up to user to ensure they provide the correct coercible type for the backend they are targeting. See the section on [Type Coercion](#type-coercion) for more details. +As the runtime section consists of key/value pairs, it is the user's responsibility to ensure they provide the correct coercible type for the backend they are targeting. Expressions are explicitly forbidden disallowed within the `runtime.json`, hence no input will be coerced to an expression. See the section on [Type Coercion](#type-coercion) for more details. ### Resolution Order From 233b6634afe5caf187a14ad579a362c2532bc314 Mon Sep 17 00:00:00 2001 From: Michael Franklin Date: Wed, 3 Apr 2019 10:19:09 +1100 Subject: [PATCH 3/7] Remove explicit + forbidden from runtime override --- versions/development/SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/versions/development/SPEC.md b/versions/development/SPEC.md index 2c852be3..5cf02816 100644 --- a/versions/development/SPEC.md +++ b/versions/development/SPEC.md @@ -2663,7 +2663,7 @@ In JSON, the user may be able to specify task-specific runtime attributes simila } ``` -As the runtime section consists of key/value pairs, it is the user's responsibility to ensure they provide the correct coercible type for the backend they are targeting. Expressions are explicitly forbidden disallowed within the `runtime.json`, hence no input will be coerced to an expression. See the section on [Type Coercion](#type-coercion) for more details. +As the runtime section consists of key/value pairs, it is the user's responsibility to ensure they provide the correct coercible type for the backend they are targeting. Expressions are disallowed within the `runtime.json`, hence no input will be coerced to an expression. See the section on [Type Coercion](#type-coercion) for more details. ### Resolution Order From 7d744e30f41fd601298e915633264268ef98532a Mon Sep 17 00:00:00 2001 From: Michael Franklin Date: Thu, 18 Apr 2019 23:55:42 +1000 Subject: [PATCH 4/7] Remove typo and clarify recommendation --- versions/development/SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/versions/development/SPEC.md b/versions/development/SPEC.md index 5cf02816..356f65cb 100644 --- a/versions/development/SPEC.md +++ b/versions/development/SPEC.md @@ -2650,7 +2650,7 @@ It's important to note that the type in JSON must be coercible to the WDL type. ## Specifying / Overriding Runtime Attributes in JSON -Workflow runtime attributes may additionally be specified as key/value pairs within a JSON input file. Ideally this should be a separate `runtime.json` file to The mapping from JSON or YAML values to WDL values is codified similar to the serialization of task inputs section, however with an additional runtime tag to avoid name collisions and allow for unspecified attributes to be set by the runtime input file. Runtime attributes do not need to be specified in the task defintion to be overidden or set by the JSON. +Workflow runtime attributes may additionally be specified as key/value pairs within a JSON input file. It's recommended that this should be a separate inputs file called `runtime.json`. The mapping from JSON or YAML values to WDL values is codified similar to the serialization of task inputs section, however with an additional runtime tag to avoid name collisions and allow for unspecified attributes to be set by the runtime input file. Runtime attributes do not need to be specified in the task defintion to be overidden or set by the JSON. In JSON, the user may be able to specify task-specific runtime attributes similar to the following: From bc9afaa5526d8627f73794746c68495c71609289 Mon Sep 17 00:00:00 2001 From: Michael Franklin Date: Mon, 29 Apr 2019 12:08:23 +1000 Subject: [PATCH 5/7] Remove duplicate TOC entry Per https://github.com/openwdl/wdl/pull/301#discussion_r276810096 --- versions/development/SPEC.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/versions/development/SPEC.md b/versions/development/SPEC.md index 356f65cb..8225e673 100644 --- a/versions/development/SPEC.md +++ b/versions/development/SPEC.md @@ -93,9 +93,6 @@ Table of Contents * [Specifying Workflow Inputs in JSON](#specifying-workflow-inputs-in-json) * [Specifying / Overriding runtime attributes in JSON](#specifying--overriding-runtime-attributes-in-json) * [Resolution Order](#resolution-order) - * [Optional Inputs](#optional-inputs) - * [Declared Inputs: Defaults and Overrides](#declared-inputs-defaults-and-overrides) - * [Optional Inputs with Defaults](#optional-inputs-with-defaults) * [Call Input Blocks](#call-input-blocks) * [Type Coercion](#type-coercion) * [Standard Library](#standard-library) From 26537d5c5058e8278dd157f52759cc5f2c5d135c Mon Sep 17 00:00:00 2001 From: Michael Franklin Date: Mon, 29 Apr 2019 12:13:26 +1000 Subject: [PATCH 6/7] Update SPEC.md --- versions/development/SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/versions/development/SPEC.md b/versions/development/SPEC.md index 8225e673..11a0c4ec 100644 --- a/versions/development/SPEC.md +++ b/versions/development/SPEC.md @@ -2660,7 +2660,7 @@ In JSON, the user may be able to specify task-specific runtime attributes simila } ``` -As the runtime section consists of key/value pairs, it is the user's responsibility to ensure they provide the correct coercible type for the backend they are targeting. Expressions are disallowed within the `runtime.json`, hence no input will be coerced to an expression. See the section on [Type Coercion](#type-coercion) for more details. +As the runtime section consists of key/value pairs, it is the user's responsibility to ensure they provide the correct coercible type for the backend they are targeting. An input value should not be coerced to an expression, hence values provided within the `runtime.json` should be evaluated. See the section on [Type Coercion](#type-coercion) for more details. ### Resolution Order From 362a46a1eb90deb5b2752b7b3eec2ff479d0aa67 Mon Sep 17 00:00:00 2001 From: Michael Franklin Date: Mon, 29 Apr 2019 14:29:15 +1000 Subject: [PATCH 7/7] should -> should not maybe I shouldn't add attention to detail on my resume anymore --- versions/development/SPEC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/versions/development/SPEC.md b/versions/development/SPEC.md index 11a0c4ec..1f911901 100644 --- a/versions/development/SPEC.md +++ b/versions/development/SPEC.md @@ -2660,7 +2660,7 @@ In JSON, the user may be able to specify task-specific runtime attributes simila } ``` -As the runtime section consists of key/value pairs, it is the user's responsibility to ensure they provide the correct coercible type for the backend they are targeting. An input value should not be coerced to an expression, hence values provided within the `runtime.json` should be evaluated. See the section on [Type Coercion](#type-coercion) for more details. +As the runtime section consists of key/value pairs, it is the user's responsibility to ensure they provide the correct coercible type for the backend they are targeting. An input value should not be coerced to an expression, hence values provided within the `runtime.json` should not be evaluated. See the section on [Type Coercion](#type-coercion) for more details. ### Resolution Order