Add two new compiler options for cori-knl to make it easier for testing

[ndkeen]

For cori-knl only right now:
--compiler=intel18 to use the beta version of Intel v18
--compiler=intel-impi to use the Intel MPI (with the default Intel compiler)
[BFB]

[worleyph]

@ndkeen , would I be able to reproduce your seg. fault, reported elsewhere, by specifying intel-impi ? What compset and resolution should I use?

[ndkeen]

Yes. Would it make it easier if I pushed to master?
I do see failures with acme_dev tests with this using --compiler=intel-impi (and some with intel18). I'm working through them to see if they make sense, etc.

The results I reported before were with a different branch, but same idea -- probably would work the same, but I (or you) can try again now.

[worleyph]

Would it make it easier if I pushed to master?

My "master" has lots of local changes, for the ATM initialization study. Checking out your branch is just as convenient for me. I would like to know exactly what case you ran when you observed this. If you want to repeat it first yourself, to verify that it is still there, that would be great.

[ndkeen]

@amametjanov wants me to revert and rework using the -mpilib flag.

[worleyph]

@ndkeen , added code to check for NaNs and INFs in the data being passed to the reproducible sum routine. The case that is failing for you with the Intel MPI library is due to NaNs being passed to this routine. This of course does not address where the NaNs are coming from.

[rljacob]

@ndkeen how is this going?

[ndkeen]

I decided I will revert this change and create 2 new PR's: one to add --compiler=intel18 which I can do quickly, the other to add option for using Intel MPI using the --mpilib=impi option which may take some time. Related: I'm not sure how to revert my change which is on next. I know it's git revert, but not sure the args.

[rljacob]

Find the merge commit that placed it on next, which is d963c03, and then do
git revert -m 1 d963c03

[ndkeen]

Done differently here: #1710