DFN format

[wpbonelli]

We have arrived at TOML as the target format for definition files. TOML is visually similar to existing definition files, while allowing a structured representation of the block/variable tree.

As @deltamarnix has suggested DFN files should be versioned. We can consider legacy DFNs version 1. The TOML format can be 2(.0.0), then whenever we make a change to the format (e.g. adding/removing an attribute) we bump the version number, so consumers can explicitly decide whether they support a given DFN file without having to try and parse it all.

After chatting on an initial draft in modflowpy/flopy#2333 (comment) it sounds like we want blocks at the top level, rather than variables, and to omit explicit child attributes in favor of attaching children directly to the parent. So the example would become

name = "prt-oc"
multi = false
stress = false
advanced = false

[options.budget_filerecord]
type = "record"

[options.budget_filerecord.budget]
type = "keyword"
reader = "urword"
optional = false
longname = "budget keyword"
description = "keyword to specify that record corresponds to the budget."

[options.budget_filerecord.fileout]
type = "keyword"
reader = "urword"
optional = false
longname = "file keyword"
description = "keyword to specify that an output filename is expected next."

[options.budget_filerecord.budgetfile]
type = "string"
reader = "urword"
optional = false
longname = "file keyword"
description = "name of the output file to write budget information."

...

Period block specifications could look like

...

[period.iper]
type = "integer"
block_index = true
reader = "urword"
optional = false
longname = "stress period number"
description = "integer value specifying the stress period number for which the data specified in the PERIOD block apply. IPER must be less than or equal to NPER in the TDIS Package and greater than zero. The IPER value assigned to a stress period block must be greater than the IPER value assigned for the previous PERIOD block. The information specified in the PERIOD block applies only to that stress period."

[period.perioddata]
type = "list"
reader = "urword"
optional = false

[period.perioddata.releasesetting]
type = "record"
reader = "urword"
optional = false

[period.perioddata.releasesetting.all]
type = "keyword"
reader = "urword"
description = "keyword to indicate release of particles at the start of all time steps in the period."

[period.perioddata.releasesetting.first]
type = "keyword"
reader = "urword"
description = "keyword to indicate release of particles at the start of the first time step in the period. This keyword may be used in conjunction with other keywords to release particles at the start of multiple time steps."

[period.perioddata.releasesetting.frequency]
type = "integer"
reader = "urword"
description = "release particles at the specified time step frequency. This keyword may be used in conjunction with other keywords to release particles at the start of multiple time steps."

[period.perioddata.releasesetting.steps]
type = "integer"
shape = "(<nstp)"
reader = "urword"
description = "release particles at the start of each step specified in STEPS. This keyword may be used in conjunction with other keywords to release particles at the start of multiple time steps."
...

Setting `block_index` for an integer top-level block variable can indicate transient data, otherwise the block is handled normally

This has not been implemented yet in the referenced PR or here. It could maybe be prototyped here, moved to flopy3 when ready, and then moved to devtools once we're confident in it.

[wpbonelli]

The format above has been implemented in devtools:

• feat(dfn): add dfn container, parser, toml conversion script MODFLOW-USGS/modflow-devtools#167
• feat(dfn): toml load support, switch to tomli, add tests MODFLOW-USGS/modflow-devtools#173

I'm presently working it into modflowpy/flopy#2333.

I'm having second thoughts about the "implicit" nesting. Looking elsewhere for precedent, the convention in pyproject.toml files is to name collections explicitly. E.g. [tool.pixi.feature.lint.tasks]. This is more verbose, but clearer (imo) and easier to work with programmatically (no need to filter entries to find children).

E.g. [options.cvoptions.dewatered] is concise but [block.options.cvoptions.fields.dewatered] makes it clearer from the table path that this variable is part of a record, that it's a member of the options block rather than some other notion of options relevant to definition files more generally, etc.

It seems better to have a fixed top-level schema with a blocks table whose entries vary by component rather than some minimal set of top-level attributes and no restrictions on top-level block names.

Explicitness also seems preferable in toml in general because nested structures are not obvious to the eye, as they would be in yaml or json. Issue 781 on the toml repo is an interesting discussion of toml's approach to nested data.

[wpbonelli]

Removed the top-level blocks table in MODFLOW-USGS/modflow-devtools#177 after marinating more. Blocks now sit at the top level. Variables still have explicit tables for children:

• item for lists
• choices for unions
• fields for records

For example

[period.stress_period_data]
name = "stress_period_data"
type = "list"
shape = "(maxbound)"
reader = "urword"
mf6internal = "spd"

[period.stress_period_data.item]
name = "stress_period_data"
type = "record"
reader = "urword"
mf6internal = "spd"

[period.stress_period_data.item.fields.cellid]
name = "cellid"
type = "integer"
shape = "(ncelldim)"
reader = "urword"
longname = "cell identifier"
description = "is the cell identifier, and depends on the type of grid that is used for the simulation.  For a structured grid that uses the DIS input file, CELLID is the layer, row, and column.   For a grid that uses the DISV input file, CELLID is the layer and CELL2D number.  If the model uses the unstructured discretization (DISU) input file, CELLID is the node number for the cell."

IMO less verbose, but still clear, also emphasizes the distinction between blocks and variables — both really can be considered product types but mf6 considers them distinct: blocks sit at the top level of a component definition, contain variables directly, and have no other attributes*. Variables must be inside a block or another variable, and can have arbitrary attributes (for now — eventually lock it down).

*iper is a block_variable, but imo this is an mf6 io format detail and flopy4 can probably consider a transient block a variable with an extra time dimension? MODFLOW-USGS/modflow-devtools#178 moves in this direction.

Lists remain the most awkward type to express. In the current DFNs a variable with type recarray can be a list of records, or of unions of records

• list of records type recarray cellid head aux boundname, where the record fields are defined inline
• list of unions type recarray releasesetting, where releasesetting is a keystring variable

There is no such thing as a list of scalars, that's an array from mf6's perspective.

In the TOML format these are now

• [...<var>.item.fields...]
• [...<var>.item.choices...]

I'm not convinced this is the best way, though.

[wpbonelli]

@deltamarnix we recently had a chat on how to represent subpackages where we arrived at the following:

• a component can declare itself a subpackage by including a top-level attribute which identifies the variable to be substituted into a parent package
• a component can consume subpackages by including a subpackages table, with an entry per subpackage, each containing an attribute specifying the attribute name under which to attach the subpackage data

I thought about this some more. An MF6 simulation is a tree. Likewise the MF6 input specification is a tree too. A simulation is a "materialization" of some subset of the specification. Most components must have one particular parent, e.g. tdis must be attached to the simulation, npf to a gwf model. Some components may relax this requirement (to a set of potential parents, or a set of parent types) or remove it entirely. These are what we have been calling "subpackages".

We might consider making the tree explicit, instead of inferring it from the DFN names? Components might have a top-level parent attribute, either a string or a list of strings, identifying base or concrete parent types e.g. parent = "simulation" or parent = ["simulation", "model"] or parent = "gwf-nam". This would give more flexibility for DFN (re-)naming in the future, if ever useful.

And it generalizes the concept of "subpackage". Flopy could treat anything with one particular parent as normal, binding by path as specified in the parent's namefile, and for anything with no specified (or multiple) parent(s), it could do what it does for subpackages currently, bind by init parameter (i.e. swap it into the consuming class for the identifying "foreign key" filepath variable).

[wpbonelli]

If we want to make parents explicit I think we will need to enforce a 1-1 mapping between DFNs and generated classes. Currently flopy generates 2 classes each for simulation and model DFNs, one for the sim/model, one for the "namefile package". But seems like a flopy implementation detail, not a distinction mf6 makes. We could aim to collapse these in flopy4

[wpbonelli]

Consider changing the shape attribute name for array variables to dims for consistency with xarray?

[deltamarnix]

Both are fine names to me.

[wpbonelli]

I like both too, but maybe dims makes it clearer that the shape is unresolved? i.e. dimension names not integers?

[wpbonelli]

Consider distinguishing abstract vs concrete DFNs?

Currently namefile definitions contain record representations of their abstract subcomponents, e.g. sim-nam.dfn has blocks for models and exchanges, which are recarrays with columns type, filename, namea, nameb. This is information about how to "bind" components, i.e. create a parent-child relationship between them.

Binding is a problem that MF6 and flopy solve independently. The abstract relationship is what matters — a simulation "has" a model which "has" a package — how to express that most easily depends whether one is writing input files by hand or using python/flopy.

It seems like a code-generation concern how to "interpret" DFNs for a given program. DFNs could abstractly represent the input spec, with mf6 and flopy tooling parsing that into the appropriate paradigms. E.g. flopy's object-oriented interface, where the user might not know the ASCII input files expected until they call write_simulation(). Or, the current "subpackage" mechanism with children bound by __init__ method parameter instead of a separate input file. Or in future read a simulation from a single netcdf file, where the structure is in one file.

[wpbonelli]

The file binding problem is the reason flopy currently makes two classes for simulations and models, one for the thing itself and one for its namefile. But IMO that's unduly influenced by the MF6 paradigm. If DFNs encoded the component tree in a more abstract way, flopy could keep input file binding details out of the component classes?