MPAS-O running on mira / cetus

[douglasjacobsen]

MPAS-O has not been confirmed to run correctly on mira or cetus yet.

[douglasjacobsen]

@jayeshkrishna I only assigned you because you're the POC. I'll help make sure it's working, but we can have the discussions about it here.

[jayeshkrishna]

Thanks.
(FYI: Please refer to earlier discussions on MPAS0 + Mira/Cetus in Issue #173)

[jayeshkrishna]

Unfortunately the 2-day run on Cetus failed.
The logs are here -
https://gist.github.com/f726e12d3f637ccaa82b

@douglasjacobsen , Anything in particular that stands out to you (in the logs) as the source of the failure?

[douglasjacobsen]

@jayeshkrishna Not really, but it looks like it's dying in finalization (since It's getting through almost two days worth of time steps).

What command did you use to create the case? I want to try it out on cetus, but previously it was dying when it was trying to init the model for me.

[jayeshkrishna]

I used the following command to create the case (this is the same error that I got when I was interacting with Mark for #173 ),

./create_newcase -compset CMPASO-NYF -mach cetus -res T62_oEC60to30 -case mpaso_testing_CMPASO-NYF_cetus_T62_oEC60to30_062615_01

[jayeshkrishna]

I found two versions of ESMF_TimeMod in the repo (see below) that is possibly the source of one of the error messages in the cesm log (format specification error),

[jayesh@cetuslac1 ACME_mpaso_testing (douglasjacobsen/mpas-o/add-model=)]$ find . -name ESMF_TimeMod.F90
./models/ocn/mpas-o/model/src/external/esmf_time_f90/ESMF_TimeMod.F90
./models/utils/esmf_wrf_timemgr/ESMF_TimeMod.F90
[jayesh@cetuslac1 ACME_mpaso_testing (douglasjacobsen/mpas-o/add-model=)]$ diff ./models/utils/esmf_wrf_timemgr/ESMF_TimeMod.F90 ./models/ocn/mpas-o/model/src/external/esmf_time_f90/ESMF_TimeMod.F90 | more
839c839
<            "(I", yearWidth, ".", yearWidth, "'-',I2.2,'-',I2.2,'_',I2.2,':',I2.2,':',I2.
2)"
---
>            "(I", yearWidth, ".", yearWidth, ",'-',I2.2,'-',I2.2,'_',I2.2,':',I2.2,':',I2
.2)"

[douglasjacobsen]

@jayeshkrishna Yeah, the one in the ocean isn't built as part of this. (i.e. models/ocn/mpas-o/model/src/external/esmf_time_f90). Instead, we use the one from models/utils/esmf_wrf_timemgr.

So, unless the bottom format specifier looks like the incorrect one, I'd be surprised if this is actually causing an issue.

[douglasjacobsen]

@jayeshkrishna I had a run using the same command you used that made it through the queue over the long weekend, but it didn't make it as far as your run did.
https://gist.github.com/douglasjacobsen/3fbde002f0a1b9cb201c

I'm going to keep looking into it, but I'm not sure what to do about this. I had built this run in debug mode, but since it failed different from how yours failed, I'm going to try it without debug mode.

[douglasjacobsen]

@jayeshkrishna I can get through a run, but now when it's deallocating everything, I get the following error message:

Error encountered while attempting to deallocate a data object

It specifically happens on this line: https://github.com/ACME-Climate/MPAS/blob/ocean/develop/src/framework/mpas_pool_routines.F#L252

I tried changing the compiler version to see if it was a bug, but that didn't work. Any ideas?

[jayeshkrishna]

Was (dptr % r2) NULLified before use? Some compilers don't nullify ptrs by default.

[douglasjacobsen]

Yes: https://github.com/ACME-Climate/MPAS/blob/ocean/develop/src/framework/mpas_pool_types.inc#L26

[douglasjacobsen]

Also, I added an if associated test right before the deallocate to see if it is something that it should be able to dealloacte, and it is.

[jayeshkrishna]

How about r2 % array (Not NULLiefied in the decl - mpas_field_types.inc)?

[worleyph]

I haven't been paying any attention to this, so please excuse the possibly stupid comment, but have you tried running with fewer processes per node (and more nodes), just to eliminate any possibility that this is a memory corruption issue due to maxing out the memory?

[douglasjacobsen]

@jayeshkrishna r2 % array is deallocated fine right before that. I could try nullifying it after it's deallocated, just as a test though.

@worleyph I've not had good luck trying to change the number of processes on mira / cetus. So, I'm just using the default for now. However, this grid shouldn't run into memory issues. It can easily run on 64 processors (slowly at least), and this case is running on 2048.

[jayeshkrishna]

I am assuming that deallocating an unassociated ptr might have corrupted data structures.

[douglasjacobsen]

@jayeshkrishna It wasn't unassociated though. It was an actually alloacted pointer (meaning r2 % array).

[jayeshkrishna]

Ok, I was thinking in the lines of "r2%array not NULLified" => associated() returns true even though the ptr is unassociated => deallocate(r%array) corrupts memory. It might be worthwhile just trying to print couple of elements of the array to make sure it is what you expect it to be.

[jayeshkrishna]

• As I mentioned above the difference in ESMF_TimeMod is the cause of the warning messages in the cesm log (changing the fmt string got rid of the warning messages)
• With the DEBUG mode on, as you mentioned before, I am not able to get past the initialization

[singhbalwinder]

Not sure if it is relevant here but I got into a lot of unassociated
pointer errors and I ended up nullifying a lot of pointer in MPAS-O. My
changes to the code are on my local machine. If you think they may be
useful, I can push them to the branch.

On Wed, Jul 8, 2015 at 10:57 AM, jayeshkrishna notifications@github.com
wrote:

Ok, I was thinking in the lines of "r2%array not NULLified" =>
associated() returns true even though the ptr is unassociated =>
deallocate(r%array) corrupts memory. It might be worthwhile just trying to
print couple of elements of the array to make sure it is what you expect it
to be.

—
Reply to this email directly or view it on GitHub
#260 (comment).

[jayeshkrishna]

You could also try using the "-qinitauto" compiler option to see if that helps.

[douglasjacobsen]

@jayeshkrishna Yeah, the ESMF change needs to happen within ACME either way to prevent that message. I think the MPAS version has that change though, but within ACME the MPAS version doesn't get used so we don't need to modify it at all.

I tried adding a nullify(dptr % r2 % array) and it didn't help any, but I'll let you know about the -qinitauto

[douglasjacobsen]

@jayeshkrishna Adding -qinitauto actually made the model die during init. I'll have to look into why, but the error messages are not useful right now.

[jayeshkrishna]

I was able to get past the init failure with DEBUG turned on. However the run did not complete (crashed).
The logs and the patch is here - https://gist.github.com/759b2bbfeefb39794558

[douglasjacobsen]

Thanks @jayeshkrishna. I'll look into it with debug on now and see if it gets us any further.

[maltrud]

On Jul 9, 2015, at 8:28 AM, Doug Jacobsen <notifications@github.commailto:notifications@github.com> wrote:

Thanks @jayeshkrishnahttps://github.com/jayeshkrishna. I'll look into it with debug on now and see if it gets us any further.

Hi Guys-- i think you may just need to decrease the timestep to 10 minutes. or has it run on other platforms with 20 minute step? -mat

[douglasjacobsen]

@maltrud It runs fine with a 20 minute time step. The error on mira is some weird fortran + xlf issue that we're trying to figure out.

[rljacob]

Can this issue be closed?

[douglasjacobsen]

@rljacob I haven't tested it again yet. I need to get around to it, but the hope is that #412 should fix this issue.

[douglasjacobsen]

This is fixed now.