ADIOS plugin

[dayalsoap]

From our discussion the other week, we want to start testing out the ADIOS plugin with some of the in-situ/staging methods for ADIOS, Flexpath in particular.. From going over the code, and during the discussion, we found a few issues.

1. At each output epoch, a new file name is created. For the staging methods file names are used to establish connectivity between different processes. Re-establishing the communication graphs at each output epoch cause a lot of overhead.

2. At each output epoch, there a new set of variables is defined. This causes some issues for us because we use variable names to create serialization formats, so for each output epoch, this translates into a new format.

3. Array dimensions and offsets are passed in as numeric values and not as variables. ADIOS allows you to specify variables that will hold the array dimensions and offsets. The only caveat is that those variables have to be written via adios_write before the array is written.

I am working on some of this now, but also wanted to know if using the XML document was a better option? Using define vars is perfectly fine, but there are some advantages to using an XML document. Using an XML document however would require some bigger changes to the plugin I believe. Basically, I don't think you get variable ID's with the xml document.

[ax3l]

Thank you for collecting all the points in the issue! ✨

1. and 2) are basically since we were afraid, from our experiences with HDF5, that a crash in the code could corrupt previous outputs if the file handles are still open. In addition, we can both dump checkpoints and regular outputs with the same routines (but containing different data sets). This requires some refactoring but will be possible!

2. If we would use the xml API: you are right we can use variables for dimensions. But that requires us to write a XML file at runtime with all the information, adding an additional XML writer and introducing additional overhead, adding an additional layer of abstracting that information too and adds XML lib dependencies. As a side note, variable IDs are currently only used (in a non-fancy way) to spare to define some information twice between calculating the buffers, opening the files and actually writing the data.

Since XML requires a lot of restructuring for us and does not naturally fit into our implementation: can we go with the C-API, too?

We would also need to write actually two files, one for checkpointing and one for dumps... And we would have to expose a lot of information in variables such as moving window simulations, additional variables that are dumped in other intervals etc. We can try to add this in the long term but it will make the whole thing way more complex on our side (and from my experience during the implementation of #679 it would take some time without scientific output on our side).

[dayalsoap]

So in regards to the XML document, why would it have to be written programmatically? I'm guessing this is because the variable names aren't the same across configurations?

The no_xml is perfectly fine, anyways. We can still define array dimensions/offsets as variables. This way, the defines would only need to be called once after the adios_init call. Does the set of output variables change within a run?

[ax3l]

the xml documents would have to be written programatically, since the users can specify for the default core but also for various plugins how often and what they should dump (and calculate). An example is the input file fileOutput.param which is completely free to user input and additionally compiled-in for each run. User-side command-line input might also influence the output.

If no_xml is possible we can probably start way faster, that is great! The set of output variables are currently determined by either compiled-in or by command-line arguments. But we have plugins such as our 3D live rendering, where a connected scientist can actually select at run time which variables he/she would like to analyse (really selecting an other source during the run; ideally creating a new source from existing combinations in the future as in the ParaView Calculator pipeline element).

Nevertheless, for the start and in the current implementation (fileOutput.param + checkpoints) we can assume the set is known for the whole sim at the point of the adios_init call. Can we still vary the output periodicity reacting on non-fixed in-simulation events?

[dayalsoap]

Using the no_xml API is definitely fine. I guess the only real issue there is changing the I/O method, and the parameters given to the I/O method, would require a recompile, or perhaps they could be passed in via the command line or some other input file.

Can we still vary the output periodicity reacting on non-fixed in-simulation events?

Absolutely.

[dayalsoap]

So, there definitely is a risk in terms of crashes and losing the whole BP file. The solution for this I've been told, by the ADIOS developers, is to use separate files, which is how the ADIOS plugin currently works. Maybe a better option for me then is to just write a separate ADIOS plug in geared towards the in-situ/staging transports?

[ax3l]

thank you for the information!

there is an other aspect with using just a "single" file that I got from
upstream: the parallel "append" is not even close to the performance of
regular writes, unfortunately.

update: afaik, append performance was drastically improved in ADIOS 1.9 or 1.10 - needs dbl check.