gcmpost.script is corrupted

[yvikhlya]

Submitting archive job is commented out.
https://github.com/GEOS-ESM/GMAO_Shared/blob/main/GEOS_Util/post/gcmpost.script#L1018
This results in daily output remaining in holding directory in coupled run, overfilling disk quota eventually. If I uncomment this line, the script will fill archive disk with daily data.

[mathomp4]

@yvikhlya I believe @sdrabenh commented this out because the AGCM moved to use monthly history for pretty much all HISTORY output in:

• Use the monthly attribute by default GEOSgcm_App#316
• Use the monthly attribute by default #259

You might need to do something similar for the AOGCM histories if you move up to "recent" GMAO_Shared.

Of course, if you need the daily files, then we might need to figure out a way to at-run-time say whether archiving is needed.

[yvikhlya]

@mathomp4 I don't need dailies, I NEED to get rid of them from holding location in a safe way. I can't switch to monthly output in my ongoing runs. We will switch to monthly output in coupled runs, but what should I do with ongoing runs? Deleting dailies manually leaves a chance of missing data.

[yvikhlya]

Once again, the issue here is accumulation of dailies in holding location. They has to be deleted when month is processed, otherwise left in holding.

[yvikhlya]

@mathomp4 @sdrabenh There should be a code in gcmpost.script which prevents deleting daily data from holding if archive job is not done. This code better be removed.

[yvikhlya]

I found a way to process monthlies from dailies for my ongoing runs. All future coupled runs will be with monthly collections. But still, if someone tries to run with daily collections, they will not be moved out of holding.

[sdrabenh]

@yvikhlya I agree there is a potential for problems and unexpected results by commenting out this line. However, we have been asked not to write our data to $ARCHIVE anymore and this was the simplest fix to stop that from happening automatically. Uncommenting this line allows a user to easily revert to the old method if they really want to override this default and write to $ARCHIVE anyway (there have been requests for this). Unfortunately, gcmpost.script and related scripts were designed to use $ARCHIVE, so they really should be refactored moving forward. Regarding the issue of accumulated dailies in holding, it is not reasonable to automatically remove files from holding if the user has requested daily files in their HISTORY.rc file. We cannot assume all the user wants to do is average them and then throw away the dailies - they could very well wish to keep them. It should be the user's responsibility to manually remove the very files they requested to be written out if there is a storage limitation. If the monthly attribute is set in the HISTORY, then we know all the user expects is monthlies and there won't be any daily files to remove anyway.

[yvikhlya]

@sdrabenh Clear enough. So, if storing dailies in holding is desirable behavior, so be it, I am going to close this issue. I have a way to continue my ongoing runs the old way (archive, clear holding). Will edit coupled history templates to use monthly collections.

[yvikhlya]

Closing issue. Current behavior of gcmpost.script is intentional.

[sdrabenh]

@yvikhlya please beware that time_ave.x writes out the field variances and can additionally write out diurnal files. However, using the monthly attribute in HISTORY will not produce either of those. I believe the SI team @mathomp4 @bena-nasa is working on a way to output diurnal files through MAPL, but I'm unsure where that currently stands. I also don't know if there are plans to write out the variances through MAPL, but many of @lltakacs plots depend on the variances. For this reason, I did not default to monthly output for specific collections (IAU, SURF, etc) since time_ave.x is still needed to process them even though I discard the dailies after averaging. Anyway, this is just something to keep in mind before switching all your collections to use monthly.