Fixes requested by NCO on some of the resources used in operations.

[aerorahul]

Description
NCO requested clarification and updates to the resources for some of the jobs run in operations.
This PR addresses those comments and concerns.
Some of the changes are still being tested and may need further refinement.

Tagging @WeiWei-NCO

This work is part of #398

Type of change

• Bug fix (non-breaking change)

How Has This Been Tested?
The only way to test this is with NCO running the jobs in this PR and validating it meets their acceptance criteria.

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes need updates to the documentation. I have made corresponding changes to the documentation
• My changes generate no new warnings
• New and existing tests pass with my changes
• Any dependent changes have been merged and published

[WalterKolczynski-NOAA]

Mostly looks good as long as they meet NCO timing requirements and don't crash due to OOM. There are a couple that were reduced to less than a full node but still are requesting exclusive mode. Seems like they either shouldn't use exclusive, or you might as well throw all the CPUs at it since they are reserved anyway as long as the overhead doesn't actually make the job slower.

[aerorahul]

Mostly looks good as long as they meet NCO timing requirements and don't crash due to OOM. There are a couple that were reduced to less than a full node but still are requesting exclusive mode. Seems like they either shouldn't use exclusive, or you might as well throw all the CPUs at it since they are reserved anyway as long as the overhead doesn't actually make the job slower.

I haven't gone through all jobs, but we shouldn't request more CPU's than required or actually used.
This came about from conversations w/ @WeiWei-NCO

[WalterKolczynski-NOAA]

Mostly looks good as long as they meet NCO timing requirements and don't crash due to OOM. There are a couple that were reduced to less than a full node but still are requesting exclusive mode. Seems like they either shouldn't use exclusive, or you might as well throw all the CPUs at it since they are reserved anyway as long as the overhead doesn't actually make the job slower.

I haven't gone through all jobs, but we shouldn't request more CPU's than required or actually used. This came about from conversations w/ @WeiWei-NCO

I agree. My point was just that if we are requesting exclusive mode, we are "using" the other CPUs on that node, even if we aren't requesting them explicitly.

[aerorahul]

Mostly looks good as long as they meet NCO timing requirements and don't crash due to OOM. There are a couple that were reduced to less than a full node but still are requesting exclusive mode. Seems like they either shouldn't use exclusive, or you might as well throw all the CPUs at it since they are reserved anyway as long as the overhead doesn't actually make the job slower.

I haven't gone through all jobs, but we shouldn't request more CPU's than required or actually used. This came about from conversations w/ @WeiWei-NCO

I agree. My point was just that if we are requesting exclusive mode, we are "using" the other CPUs on that node, even if we aren't requesting them explicitly.

Actually, you are not "using" them. If one tries to do mpiexec -n number_exceeding_requested_cores, the the job will fail.

[KateFriedman-NOAA]

Changes look good, I'll need to incorporate into rocoto workflow and test. Left questions on the waveinit and wavepostsbs resources changes: need excl or memory setting per Wei's request.

[aerorahul]

@NOAA-EMC/global-workflow-cms
I have added a few more updates since the last review following changes from NCO.

[aerorahul]

@KateFriedman-NOAA @WeiWei-NCO
I think this is current as of this evening of testing and can be merged.
There may be a few more later, but I think we are on solid ground now.

[KateFriedman-NOAA]

Changes look good and conform to expected optimization outcomes between NCO/EMC.