Automatic resubmit does not work on several platforms

[jeff-cohere]

(via Peter Caldwell)

The following link defines a case that runs successfully but fails to restart on Edison:

https://gist.github.com/jnjohnsonlbl/dfc2e5828d02f859d4cf

[rljacob]

Looks like this is basically an FC5 case with ne30_g16 resolution. Anything else needed to cause the restart fail?

[rljacob]

An extact restart test with FC5 and ne30_g16 passes on edison with latest master. By default, it does an 11 day run, makes a restart at day 6, then picks that up and does a 5 day run, then compares results.

[rljacob]

Tagging @PeterCaldwell

[mt5555]

Is this the issue we are using to track the failure to resubmit on all platforms? (original bug report says fails to restart, but @PeterCaldwell confirmed that he meant the job fails to resubmit a restart run).

[rljacob]

Yes. Updated the title.

[abigailgaddis]

Not sure if this is helpful, but my resubmits are working on Titan in the usual way (continue_run = true in env_run.xml, along with setting the number of days and resubmits). I'm using a slightly older version of ACME that still has the cime/scripts structure. Git tag: v0.4-298-ga47f412

[rljacob]

Thanks. machines known of so far are: Edison, Mira, Constance (PNNL). A fix is available, I just need time to merge it.

[kaizhangpnl]

Thanks for the information. Would you mind sharing the fix (in a feature branch?) with us (PNNL team) now?

[rljacob]

I mispoke: I don't have a branch either. Its a patch for standalone cime that I still need to put on an ACME feature branch. I'll try to at least get it pushed tomorrow.

[kaizhangpnl]

Great. Thanks!

[PeterCaldwell]

Is this fixed on the latest master now? The previous situation was that the first time you submit a job it fails to resubmit. But if you manually resubmit the model once, it would automatically resubmit itself from then on. @golaz mentioned yesterday that tags this week now not only fail to resubmit initially, but also fail to resubmit themselves subsequently on titan. I'm wondering if this is a symptom of your cure...

[mt5555]

Probably not.

ideally, if this is fixed, the pull request would say Fixes #XXXX, and then this issue would automatically be closed when the pull request is merged.

[golaz]

Yes, I was getting errors with the automatic resubmission on titan last week. The error was

"""
Submitting CESM job script: case_scripts.run
: qsub ./case_scripts.run

Job not submitted

qsub input from STDIN is not permitted.
Please use "qsub batch.script".

Job submission failed
"""

Eventually, I ended up disabling automatic resubmission and manually submitting chained jobs with dependencies. That's actually probably more efficient in terms of getting through the queues.

[abigailgaddis]

When I submit on Titan, I've been using ./CASENAME.submit rather than the old qsub *.run. I'm not sure if that's the issue, @golaz?

[cameronsmith1]

I get the same message as @golaz .
When the run is started, it is done using ./CASENAME.submit . The error @golaz provides above occurs after the first run has finished and it is trying to submit a follow-on run.

[ghost]

Resubmission works fo me for Edison@Nersc with @amametjanov commit:

c4501ed
Author: Azamat Mametjanov azamat@mcs.anl.gov
Date: Fri Jan 8 19:23:45 2016 -0800

The way it works is that after a run and, possibly, archiving are done,, it sends another run to the queue.
Ideally, you want to specify the number of runs and submit them all with dependencies so that when the first is done, the dependants are ready to go without have to climb the queue again.
Notice that each queue has limits on the number of jobs that can simultaneously be both running or holding on.

[cameronsmith1]

I have found that the problem goes away if short-term archiving is switched on (dout_s=true in env_run.xml). I have no idea why this should be, other than the logic in the script is probably a bit different.

[bmayerornl]

I am having a failure to resubmit on Titan. I have setup a case while using v1.0.0-alpha.5. The case creation command is: create_newcase -mach titan -compiler pgi -res ne16_g37 -compset F1850C5 -project cli115 -case F1850C5_ne16_g37_p32

After running a 2 month test case, I have modified env_run.xml from the default by:
RESUBMIT = 6
CONTINUE_RUN = TRUE
And of course the ndays -> nmonths and STOP_N to 8

The case runs fine, but does not resubmit itself. When I manually resubmit the case run fine again, but again does not resubmit itself.

[cameronsmith1]

@bmayerornl , have you do you have short-term archiving turned on? That has solved the problem for me in the past (dout_s=true in env_run.xml).

[helenhe40]

@bmayerornl For the resubmit issue, could you please test the following?

I tested that in your *.run batch script, if you add the "cd $CASEROOT" line below before the if ($RESUBMIT > 0) logic, it will work.

cd $CASEROOT ** notice this is the added line **

if ($RESUBMIT > 0) then
@ RESUBMIT = $RESUBMIT - 1
echo RESUBMIT is now $RESUBMIT
...

Could you please give it a try?

[rljacob]

The new version of CIME has a test for resubmit.

Clone it from https://github.com/ESMCI/cime/, cd to cime/scripts and try:
./create_test ERR_Ld3.f45_g37_rx1.A