bug: plot_rmse_xxx_evolution.m and plot_evolution not working with 1D location

[timhoar]

Describe the bug

plot_rmse_xxx_evolution.m and plot_evolution.m (and maybe more) no longer work with obs_diag_output.nc files from the models that use the oned_location_mod.f90

Run the lorenz_63 workshop_setup.csh and then run the obs_diag program with the default values. It generates an obs_diag_output.nc file with 200 timesteps and 3 observations (one for each state variable).

Error Message

>> plot_rmse_xxx_evolution('obs_diag_output.nc','totalspread')
...
Error using reshape
To RESHAPE the number of elements must not change.

Error in plot_rmse_xxx_evolution>plot_quantity (line 638)
    data = reshape([prior              posterior         ]',2*plotdat.Nbins,1);

Error in plot_rmse_xxx_evolution>myplot (line 344)
[hrmse,  legstr_rmse ] = plot_quantity(            'rmse', plotdat);

Error in plot_rmse_xxx_evolution (line 306)
            myplot(plotdat);
 
>> plot_evolution('obs_diag_output.nc','totalspread')         
  1 is RAW_STATE_VARIABLE

 RAW_STATE_VARIABLE_guess all regions level 1 all times

Which model(s) are you working with?

Anything that uses the oned_location_mod.f90

Version of DART

v9.10.4

Have you modified the DART code?

No

Build information

Not a compiler problem, but Matlab is R2019a Update 7 (9.6.0.1307630) 64-bit (maci64)

Any Idea on the solution?

The 1D obs_diag_output.nc files do not have a 'level' dimension, so it is (now) not interpreting the time,copy,region correctly.
The 3D obs_diag_output.nc files have variables dimensioned time,copy,level,region. I think the solution is to detect that one has read 1D output files and reshape the variable to have a singleton dimension for level. Right about line 265 in the case of plot_rmse_xxx_evolution.m ... I have not tested this solution, BTW.

[hkershaw-brown]

This is trying to reshape a 3x3 into 400x 1

638     data = reshape([prior              posterior         ]',2*plotdat.Nbins,1);
K>> [prior              posterior         ]

ans =

     3     3

K>> 2*plotdat.Nbins,1

ans =

   400


ans =

     1

K>> data = reshape([prior              posterior         ]',2*plotdat.Nbins,1);
Error using reshape
Number of elements must not change. Use [] as one of the size inputs to automatically calculate
the appropriate size for that dimension.
 
K>> 

[hkershaw-brown]

lorenz_63 workshop_setup.sh (input.workshop.nml) is out-of-date (v10 vs v11 qceff)

[jlaucar]

This could be fixed by changing the many plotting programs to be smarter in detecting the lack of a 'level' for fields generated by models that use the oned location module. It can also be fixed as Tim suggested by modifying the oned/obs_diag_mod.f90 so that it outputs fields with an additional dimension 'level' that only has a single legal value.
The first approach would require extensive testing with threed models. The latter approach is slightly less nice given that it adds an extra meaningless singleton dimension to the netCDF output files. However, it has no potential footprint on anything other than oned models and is straightforward to test.
I have implemented this version and tested it with output from Lorenz_96. Is this solution acceptable?

Note that these matlab scripts are generating an annoying error message when they create postscript output files which are apparently going to be no longer supported sometime after the matlab 2024a release.

[jlaucar]

Implemented a fix to oned/obs_diag.f90 as discussed above. Results in output nc files where the fields have an extra singleton dimension with the only legal value -1 for level. This has been tested with Lorenz_96, Lorenz_63, Lorenz_84 and Lorenz_96_tracer_advection with matlab programs plot_rank_histogram.m, plot_evolution.m, and plot_rmse_xxxx_evolution.m. These had never been tested with Lorenz_96_tracer_advection which has multiple observation types, but all the features worked nicely.