Building with S2SWA fails on Orion from a clean environment

[WalterKolczynski-NOAA]

Description

When using a clean environment on Orion (i.e. after running module purge or module reset), the build of UFS using the tests/compile.sh script fails when building the S2SWA app (and probably all apps using GOCART).

To Reproduce:

LMOD_SYSTEM_DEFAULT_MODULES='contrib' module reset
tests/compile.sh orion '-DAPP=S2SWA -D32BIT=ON -DCCPP_SUITES=FV3_GFS_v17_p8,FV3_GFS_v17_coupled_p8' 0 intel YES NO

Additional context

The failure is occurring in a python script within the MAPL library. While cmake finds a sufficiently recent version of python, it does not set the path to python3 in the environment. The shebang of etc/MAPL_GridCompSpecs_ACG.py looks to the environment for python3 (the appropriate method for shebangs), causing the script and the build to fail:

head -1 /work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2_ncdf492/intel-2022.1.2/impi-2022.1.2/mapl/2.35.2-esmf-8.4.2/etc/MAPL_GridCompSpecs_ACG.py
#!/usr/bin/env python3

Loading a of the python3 modules prior to calling compile.sh works around the problem:

LMOD_SYSTEM_DEFAULT_MODULES='contrib' module reset
module load intelpython3/2022.1.2
tests/compile.sh orion '-DAPP=S2SWA -D32BIT=ON -DCCPP_SUITES=FV3_GFS_v17_p8,FV3_GFS_v17_coupled_p8' 0 intel YES NO

The failure was encountered while updating global-workflow from 4d05445 to 5aeeffd.

Output

Relevant bits of the failing build log:

Currently Loaded Modules:
  1) contrib/0.1     5) intel/2022.1.2       9) scotch/7.0.3   13) hdf5/1.14.0   17) fms/2023.01  21) g2tmpl/1.10.2  25) gftl-shared/v1.5.0
  2) noaatools/3.1   6) hpc-intel/2022.1.2  10) jasper/2.0.25  14) netcdf/4.9.2  18) bacio/2.4.1  22) ip/3.3.3       26) mapl/2.35.2-esmf-8.4.2
  3) cmake/3.22.1    7) impi/2022.1.2       11) zlib/1.2.11    15) pio/2.5.10    19) crtm/2.4.0   23) sp/2.3.3       27) ufs_common
  4) hpc/1.2.0       8) hpc-impi/2022.1.2   12) libpng/1.6.37  16) esmf/8.4.2    20) g2/3.4.5     24) w3emc/2.9.2    28) ufs_orion.intel

 

+ echo 'Compiling -DAPP=S2SWA -D32BIT=ON -DCCPP_SUITES=FV3_GFS_v17_p8,FV3_GFS_v17_coupled_p8 into fv3_0.exe on orion'
Compiling -DAPP=S2SWA -D32BIT=ON -DCCPP_SUITES=FV3_GFS_v17_p8,FV3_GFS_v17_coupled_p8 into fv3_0.exe on orion

...

-- Found Python: /apps/intel-2022.1.2/intel-2022.1.2/intelpython/latest/bin/python3.9 (found suitable version "3.9.7", minimum required is "3.6") found components: Interpreter 
Found Python: /apps/intel-2022.1.2/intel-2022.1.2/intelpython/latest/bin/python3.9

...

[  0%] Generating automatic code for CA2G_StateSpecs.rc
cd /work2/noaa/global/wkolczyn/save/global-workflow/reflectivity/sorc/ufs_model.fd/build_fv3_0/GOCART/ESMF/GOCART2G_GridComp/CA2G_GridComp && /work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2_ncdf492/intel-2022.1.2/impi-2022.1.2/mapl/2.35.2-esmf-8.4.2/etc/MAPL_GridCompSpecs_ACG.py /work2/noaa/global/wkolczyn/save/global-workflow/reflectivity/sorc/ufs_model.fd/GOCART/ESMF/GOCART2G_GridComp/CA2G_GridComp/CA2G_StateSpecs.rc -i -x -p -g -d
[  0%] Generating automatic code for GOCART2G_StateSpecs.rc
cd /work2/noaa/global/wkolczyn/save/global-workflow/reflectivity/sorc/ufs_model.fd/build_fv3_0/GOCART/ESMF/GOCART2G_GridComp && /work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2_ncdf492/intel-2022.1.2/impi-2022.1.2/mapl/2.35.2-esmf-8.4.2/etc/MAPL_GridCompSpecs_ACG.py /work2/noaa/global/wkolczyn/save/global-workflow/reflectivity/sorc/ufs_model.fd/GOCART/ESMF/GOCART2G_GridComp/GOCART2G_StateSpecs.rc -i -x -p -g -d
Scanning dependencies of target Process_Library
make[2]: Leaving directory `/work2/noaa/global/wkolczyn/save/global-workflow/reflectivity/sorc/ufs_model.fd/build_fv3_0'
make  -f GOCART/Process_Library/CMakeFiles/Process_Library.dir/build.make GOCART/Process_Library/CMakeFiles/Process_Library.dir/build
[  0%] Generating automatic code for DU2G_StateSpecs.rc
cd /work2/noaa/global/wkolczyn/save/global-workflow/reflectivity/sorc/ufs_model.fd/build_fv3_0/GOCART/ESMF/GOCART2G_GridComp/DU2G_GridComp && /work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2_ncdf492/intel-2022.1.2/impi-2022.1.2/mapl/2.35.2-esmf-8.4.2/etc/MAPL_GridCompSpecs_ACG.py /work2/noaa/global/wkolczyn/save/global-workflow/reflectivity/sorc/ufs_model.fd/GOCART/ESMF/GOCART2G_GridComp/DU2G_GridComp/DU2G_StateSpecs.rc -i -x -p -g -d
[  0%] Generating automatic code for Aerosol_StateSpecs.rc
cd /work2/noaa/global/wkolczyn/save/global-workflow/reflectivity/sorc/ufs_model.fd/build_fv3_0/GOCART/ESMF/Aerosol_GridComp && /work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2_ncdf492/intel-2022.1.2/impi-2022.1.2/mapl/2.35.2-esmf-8.4.2/etc/MAPL_GridCompSpecs_ACG.py /work2/noaa/global/wkolczyn/save/global-workflow/reflectivity/sorc/ufs_model.fd/GOCART/ESMF/Aerosol_GridComp/Aerosol_StateSpecs.rc -i -x -p -g -d
make[2]: Entering directory `/work2/noaa/global/wkolczyn/save/global-workflow/reflectivity/sorc/ufs_model.fd/build_fv3_0'
[  0%] Generating automatic code for NI2G_StateSpecs.rc
cd /work2/noaa/global/wkolczyn/save/global-workflow/reflectivity/sorc/ufs_model.fd/build_fv3_0/GOCART/ESMF/GOCART2G_GridComp/NI2G_GridComp && /work/noaa/epic-ps/role-epic-ps/hpc-stack/libs/intel-2022.1.2_ncdf492/intel-2022.1.2/impi-2022.1.2/mapl/2.35.2-esmf-8.4.2/etc/MAPL_GridCompSpecs_ACG.py /work2/noaa/global/wkolczyn/save/global-workflow/reflectivity/sorc/ufs_model.fd/GOCART/ESMF/GOCART2G_GridComp/NI2G_GridComp/NI2G_StateSpecs.rc -i -x -p -g -d
/usr/bin/env: /usr/bin/env: python3python3: No such file or directory: No such file or directory

make[2]: *** [GOCART/ESMF/Aerosol_GridComp/Aerosol_Import___.h] Error 127
/usr/bin/env: python3make[2]: *** [GOCART/ESMF/GOCART2G_GridComp/CA2G_GridComp/CA2G_Import___.h] Error 127

[laurenfrederick]

Hi, @WalterKolczynski-NOAA , @natalie-perlin is going to take a look at this and will be in touch with you.

[natalie-perlin]

Hi,
Thanks for reporting the issue. This is to be addressed in the WM repo, as the Orion stack (EPIC-maintained) needs to use different location due to a mandatory transition to a new EPIC role account and disk space.

FYI - @jkbk2004 - #1745 did not have the updated Orion stack location. The PR-1846 addresses that, but it now has to be updated according to the recent changes in a develop branch following PR-1745.

[natalie-perlin]

Updates on testing: #1846 (comment)

PR-1846 resolves the issue when merged.

(FYI - @jkbk2004 )