Schema: Support nested parameters (objects)

[ewels]

Nested parameters (objects)

The current iteration of the nextflow_schema.json file does not support nested parameters. It doesn't break, it simply ignores them. This was a conscious decision to limit the complexity and scope when building the schema, but does limit its power.

It would be good to support nested parameters, for example from -with-params example.yml:

foo: bar
baz:
    qux: corge
    grault: garply

The schema structure / format itself should not need modification. It's a case of adding support for the type object. For example (simplified):

"foo": { "type": "string", "default": "bar" },
"baz": {
    "type": "object",
    "properties": {
        "qux": { "type": "string", "default": "corge" },
        "grault": { "type": "string", "default": "garply" }
    }
}

⚠️ NB: These objects can theoretically be infinitely nested! It's turtles all the way down.. ⚠️

Arrays

See #3392

Implication

Although the schema itself should be fine, there is a lot of code that interacts with the schema. All of the following will need to be updated:

• nf-core schema build
• nf-core schema lint
• The web schema build UI
• nf-core launch
• The nf-core web launch UI & validation code
• Pipeline template validation code (nextflow run ... )
• Pipeline template help text (nextflow run ... --help)

I'm hoping that the parameter validation part will be relatively simple, as we are mostly using off-the-shelf JSON Schema libraries to do this. However, anything involving a UI will need quite a bit of thought.

This is an overview issue and each part of the above should be broken into its own issue / task.

[lyin-vir]

Hi @ewels , thanks for raising this issue. I understand the challenge of implementing the nested schema, especially from a UI point of view. And I also echo the concern of having infinite nesting.

For me, having the ability to define separate objects in params is good enough. I think it would be a useful addition to the Tower interface with minimal destruction. And I would argue that rarely would anyone nest objects more than once. Also, the current tower schema interface actually supports the nesting. It's only the side menu that's a bit broken.

When I manually created the nested object, the content renders correctly:

Only the side menu has some glitches when you scroll to the nested part:

The main reason to have objects in my params is to better organize params, and to allow duplicate names. e.g. (metadata.id and user.id).

Btw, this may be an unpopular opinion, but I think the current schema is a bit confusing, since it enforces an object structure. Although the current paradigm requires a flattened params structure, all pipeline params has to be inside an object. I feel this restriction is a bit unnecessary (there might be other reasons I'm not aware of). And it actually tripped me when I'm learning to use tower schema, since intuitively I would put the params at the top level, and nested params as an object. I understand that it may break a lot of things should we decide to change this behavior. But it would simplify the UI a little bit and the nested objects would look more natural.

[ewels]

Ah so Tower does already support it a bit? Interesting! Does parameter validation / launch work and everything?

For me, having the ability to define separate objects in params is good enough.

I don't quite understand this sorry.. As opposed to what?

all pipeline params has to be inside an object

Yeah I struggled with this a bit when I was first writing it. It's because we're validating params, which is itself an object. If you remove this top-level object then the validation gets much messier. With it, we can just scoop up the whole params data-structure and validate it in one go.

I mostly feel that no-one should ever need to edit the JSON manually, so it doesn't bother me too much. Usually when people do directly edit the JSON it's either because they don't know nf-core schema build exists or because they're doing something that's not supported (like nested parameters 👀 😅 ). If you only use the build UI then you don't need to know about or understand the JSON structure.

[lyin-vir]

Thanks for your reply @ewels . Please see inline comment:

Ah so Tower does already support it a bit? Interesting! Does parameter validation / launch work and everything?

Yes, only the UI is a little broken. But validation works and the params are passed correctly.

For me, having the ability to define separate objects in params is good enough.

As opposed to what?

Compared to having lists, which is also nice to have. We use it a lot in our pipelines. But I understand the challenge from the Tower UI side.

[ewels]

Should include support for arrays as well as objects. Similar logic applies to both.

[Edit: Tracked in #3392 ]

[ewels]

@bentsherman has called into question future support of nested parameters, so this issue should be put on hold until that's figured out.

If you'd like nested parameters, please join the Slack discussion thread to voice your opinion.

[bentsherman]

Based on discussions in Slack, what I'm leaning toward is to allow a "map" param which can only be specified as a single map, and overriding the param overrides the entire map (no deep nesting).

The nested params support in nf-schema can then be used to validate the map param as is done for structured files (e.g. samplesheet). So you can still have name and type checking on the map param, but in the script it's just a regular map.

We may be able to expand on this further as we evolve the Nextflow language

[ewels]

Nested parameters are now supported in nf-schema, as of v2.1.0: https://nextflow-io.github.io/nf-schema/latest/nextflow_schema/nextflow_schema_specification/#nested-parameters

We should add support for these everywhere else in the Nextflow parameter schema world.