Restrict use of istanbul cpu target on Titan when using PGI

[worleyph]

Early on, it was determined that the PGI compiler required that
the CPU target be specified to be istanbul (instead of the actual
processor interlagos) in order for ACME (or CESM) to be
reproducible with respect to changes in process or thread counts.
Here this global specification of istanbul as the CPU target is
removed and only applied to the files that require it. Based
on experimentation, only CAM and MPASLI required
the modified CPU target, and we eventually identified a small
number of files within CAM for which this is required. We have not
yet examined these files to determine why these files require it.
We also have not yet looked at individual MPASLI files, and
instead continue to apply the istanbul cpu target for builds
of the entire GLC component.

Other changes include removing the dependence on the version of
pgi/17.5.0 installed in Dave Norton's directories, moving instead
to the now official OLCF installation. Also fixed a typo in the
module switch command for the PGI version used with PGIACC, and
changed the 'pin' flag to 'pinned' for PGIACC, as 'pin' is no
longer legal.

[Non-BFB] (on Titan when using PGI compiler)

Fixes #1620
Fixes #1610

(Note #1610 might need to be reopened, but this nominally addresses the primary issues.)

[worleyph]

Testing has been done fairly extensively using A_WCYCL2000 and ne30_oEC, but some sanity checks are probably called for, to determine that this change does preserve reproducibility in other instances. As this is not BFB, a little effort should be spent verifying that this is just roundoff differences. Note that all that has changed is compiler options, so if this is climate changing, then it is a compiler bug.

[rljacob]

@worleyph its not clear who is going to confirm that this PR is just roundoff differences.

[worleyph]

@rljacob , it has been along time since I have submitted a PR that was not BFB (or just a change in namelists). How is any PR confirmed to just be BFB? (Is this a RTFM?)

Probably my comment is just stupid and should be ignored? All that is changing here is compiler options, no code, so perhaps is the clearest case for a round-off-level change PR?

[minxu74]

How can we determine that the non-BFB here just a roundoff difference or climate changing? just based on where we make changes in the codes?

[worleyph]

I'm not finding anything in the development or integrator guide on this topic. Based on the type of change here (compiler option change, and still reproducible), guess that we can assume that it is roundoff difference only. Looking at the atm.log, it is the same for step 0 and differs by step 1:

(before)
nstep, te 1 0.33533586601617603E+10 0.33534149692914491E+10 0.31136967736702307E-02 0.98531035591945751E+05

(after)
nstep, te 1 0.33533586385805354E+10 0.33534149449283252E+10 0.31135429466769334E-02 0.98531035521281417E+05

so is consistent with being a roundoff level change.

[singhbalwinder]

How can we determine that the non-BFB here just a roundoff difference or climate changing? just based on where we make changes in the codes?

We are working on developing test methods to answer exactly this question. These methods are still under development but they will be soon available in test suites on experimental basis. Tagging @huiwanpnnl and @salilmahajan .

[singhbalwinder]

Looking at the atm.log, it is the same for step 0 and differs by step 1:
(before)
nstep, te 1 0.33533586601617603E+10 0.33534149692914491E+10 0.31136967736702307E-02 0.98531035591945751E+05
(after)
nstep, te 1 0.33533586385805354E+10 0.33534149449283252E+10 0.31135429466769334E-02 0.98531035521281417E+05
so is consistent with being a roundoff level change.

@worleyph : From your experience, what would be different for a "climate-changing" change? I am just trying to understand how you decided that this change looks like a roundoff level change.

For a "climate changing" scenario:

• Do you expect the values to differ at nstep=0 ?
  Or
• Do you expect the values at nstep=1 to be very different?

Thanks!

[worleyph]

@worleyph : From your experience, what would be different for a "climate-changing" change? I am just trying to understand how you decided that this change looks like a roundoff level change.

I don't have any experience with climate-changing change, so a climate-changing change could very look like this as well. This is simply also consistent with a round-off level change when there are multiple sources, in my experience. 8-10 digits of agreement for the large values and 4 digits of agreement for the smaller values don't raise any red flags for me. If this was a change of a single line of code, then I would expect a much smaller change, but this affects the whole code in who knows what ways.

Based on current practice when updating compiler versions (where we don't run experiments to verify that the climate has changed, we just update the baselines), I don't think that this is any worse. It would be nice to be more conservative, but we need the tools in current development.

[minxu74]

@worleyph @singhbalwinder Thanks a lot for your comments. We are looking forward to that tool.

[worleyph]

At timestep 1, there is also no input from the other components. Looking at timestep 97:

(old)
nstep, te 97 0.33506698271760306E+10 0.33506822919476566E+10 0.68924854905123140E-03 0.98532389263741556E+05

(new)
nstep, te 97 0.33506391949633703E+10 0.33506516729013834E+10 0.68998874601030344E-03 0.98530654088145704E+05

growth in these is not too bad. But a climate scientist would need to comment :-).

Also comparing intel with pgi:

(intel, unchanged, step 1)

nstep, te 1 0.33533586386590753E+10 0.33534149449900842E+10 0.31135420187541362E-02 0.98531035521301863E+05

the differences are very similar to those when changing the CPU target for PGI.

(pgi, original, step 1)

nstep, te 1 0.33533586601617603E+10 0.33534149692914491E+10 0.31136967736702307E-02 0.98531035591945751E+05

[worleyph]

Corrected previous comment as I had duplicated the intel data rather than comparing intel with pgi.

[worleyph]

Actually, the Intel compiler has never had the istanbul cpu target, and comparing Intel with PGI after the modifications:

(intel, unchanged, step 1)

nstep, te 1 0.33533586386590753E+10 0.33534149449900842E+10 0.31135420187541362E-02 0.98531035521301863E+05

(pgi, with restricted use of cpu target, step 1)

nstep, te 1 0.33533586385805354E+10 0.33534149449283252E+10 0.31135429466769334E-02 0.98531035521281417E+05

the results are more similar. Interesting ...

[singhbalwinder]

Yes, they seems to be diverging very slowly which corroborates your hypothesis that it looks like a round-off level change.

[minxu74]

It is interesting that the different compilers got more similar results than the same compiler with different options. What are these values? global average of some variables? or just outputs at a single random grid?

[rljacob]

global averages and yes that looks like roundoff.

[worleyph]

@singhbalwinder can hopefully provide a better answer - these are global averages output from check_energy.F90 in CAM physics:

 teinp_glob, teout_glob, heat_glob, psurf_glob

They are pretty sensitive, so is where I look to check for reproducibility during development.

[singhbalwinder]

Like @worleyph and @rljacob mentioned, they are the global average sums which capture model state. Any change in the model state will bring change to these numbers. So far I have been using it for checking reproducibility just like @worleyph. @wlin7 or @philrasch might have more insights into how they are computed and how they capture model state.

[minxu74]

@worleyph I ran the acme_developer test suite using your branch. There were two more failure tests compared with the results from cdash using next branch. The two tests are listed as follows and both of them have an active MPAS Land Ice model.
ERS.f09_g16_g.MPASLISIA.titan_pgi
SMS.T62_oQU120_ais20.MPAS_LISIO_TEST.titan_pgi

The errors are also very similar. There are a lot of log.landice.????.err files in the RUN directory. The error message is "CRITICAL ERROR: An error has occurred in mpas_core_run. Aborting..."

[worleyph]

@minxu74 , I never tested with active land ice. Sorry. I'll look at this now. There may be a simple fix for this PR (just re-add the istanbul cpu target for glc), and we can revisit why this is necessary next quarter. I'll give this a try first. Thanks.

[worleyph]

@minxu74 , I added the istanbul CPU target back for builds of GLC. With this change, the cases

 -compset MPASLISIA -res f09_g16_g

and

 -compset MPAS_LISIO_TEST -res T62_oQU120_ais20

completed successfully. (Without this change, I observed the same runtime error that you reported.) Please update your copy of this branch and rerun the acme_developer branch and see if this eliminates the failures.

[minxu74]

it looks good to me

[worleyph]

@mrnorman , this is still waiting on your review.

[worleyph]

@minxu74 , what do you want to do about this? You and @mrnorman were concerned about the glboal application for the istanbul target, and this addresses it. Do you care any more?

I have not yet looked at pgiacc, in fact have not even tried to repeat your success (from using more OpenMP and less MPI per node).

[minxu74]

@worleyph I have not any concerns about this PR. I just wonder if @mrnorman has any concerns and wait for his approval.

[mrnorman]

It looks good to me.

[mrnorman]

Sorry yall were waiting on my review so long. I checked my e-mail, and I didn't appear to get any. Maybe I set a bad setting somewhere on github that I need to fix?

[worleyph]

Thanks.

[jgfouca]

@mrnorman check your notifications. I've also had some epic struggles with GitHub not sending me important emails. GitHub has three email systems, one for notifications, one for at-mentions, and a third I can't remember. If one of those systems sends you an email and it bounces, that system will stop sending emails entirely until you contact GitHub and tell them to clear the bounce flag.

[rljacob]

@minxu74 please start merging this.

[rljacob]

@minxu74 Is your github email not working?