Changes to move SRW app towards NCO complaince #337

danielabdi-noaa · 2022-08-14T02:37:37Z

DESCRIPTION OF CHANGES:

Some changes needed for NCO standard compliance.

option to build directly to an exec directory instead of bin
option to build each of the components of SRW app (currently five) separately or in combination.

TARGETS
   default = builds the default list of apps (also not passing any target does the same)
   all = builds all apps
   Or any combinations of (ufs, ufs_utils, upp, gsi, rrfs_utils)

I don't think a separate script for each of them is needed especially because you can build two of them at a time
with the latter approach which the former can't do.

./devbuild.sh -p=hera           # build default list
./devbuild.sh -p=hera default   # same as default
./devbuild.sh -p=hera all       # build everything
./devbuild.sh -p=hera ufs       # build only ufs weather model
./devbuild.sh -p=hera ufs_utils # build only ufs_utils
./devbuild.sh -p=hera upp       # build only upp
./devbuild.sh -p=hera gsi       # build only gsi
./devbuild.sh -p=hera gsi rrfs_utils   # build only gsi and rrfs_utils 
                                       # rrfs_utils needs gsi to be built first

renamed src to sorc along with necessary changes in Externals.cfg
Added options --clean, --build and --install to do separate build and install

TESTS CONDUCTED:

Run one test locally on hera, will run automated test

DEPENDENCIES:

Currently none, but will need to modify regional_workflow when default bin directly is changed back to exec

DOCUMENTATION:

Currently none, but changes in devbuild.sh and directory structure need to be documented later.

venitahagerty · 2022-08-14T11:11:21Z

Machine: hera
Compiler: intel
Job: WE
Repo location: /scratch1/BMC/zrtrr/rrfs_ci/autoci/pr/1025708072/20220814105016/ufs-srweather-app
Build was Successful
Rocoto jobs started
Long term tracking will be done on 9 experiments
If test failed, please make changes and add the following label back:
ci-hera-intel-WE
Experiment Succeeded on hera: grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2
Experiment Succeeded on hera: grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR
Experiment Succeeded on hera: grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
Experiment Succeeded on hera: grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_HRRR
Experiment Succeeded on hera: grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2
Experiment Succeeded on hera: nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR
Experiment Succeeded on hera: grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
Experiment Succeeded on hera: grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_RRFS_v1beta
Experiment Succeeded on hera: grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR
All experiments completed

christinaholtNOAA

@danielabdi-noaa I think this is a nice, clean implementation of the NCO standards that we were looking for. I left just a few comments/questions below.

christinaholtNOAA · 2022-08-16T03:01:37Z

CMakeLists.txt

@@ -133,4 +132,7 @@ configure_file(

 FILE(COPY "${CMAKE_CURRENT_BINARY_DIR}/ufs_srweather_app_meta.h" DESTINATION include)

-add_subdirectory(src)
+add_custom_target(mbuild DEPENDS sorc/pbuild)
+add_custom_target(minstall DEPENDS sorc/pinstall)


Is this the magic that separates the build and install steps?

Are the names of the targets arbitrary? I suppose I'm curious why mbuild and minstall instead of just build and install.

Yes, but the cause of the problem was the external projects add. You would expect a make all would just build the binaries and not install but default behavior of ExternalProject_Add is non-intiutive -- it builds and installs on a make all. So I added STEP_TARGETS build install to separate them and then connect to a custom target at the top level. I think future CMake version could potentially simplify this. I used mbuild and minstall because I couldn't use "install" as target since it is a reserved keyword. "build" was fine though but changed it to be consistent.

I have removed the custom install target because there was already one anyway, so we have a build target instead of mbuild now. Still couldn't find a solution to prevent make all from installing the project...

christinaholtNOAA · 2022-08-16T03:07:32Z

devbuild.sh

  --build-dir=BUILD_DIR
      build directory
  --install-dir=INSTALL_DIR
      installation prefix
+  --bin-dir=BIN_DIR
+      installation binary directory name ("bin" or "exec")


Are these the only two options? Could this be set arbitrarily?

Unfortunately no. UPP and UFS_UTILS do not use CMAKE_INSTALL_BINDIR as the destination target, instead they set to either "bin" or "exec" where the latter is used if an "EMC_EXEC_DIR" variable is turned on. But once these are fixed, we should be able to install to any bin directory name.

christinaholtNOAA · 2022-08-16T03:16:02Z

test/build.sh

@@ -139,7 +139,7 @@ declare -a executables_rrfs_utl_created=( adjust_soiltq.exe \
    EXEC_DIR=${BIN_DIR}/bin
    if [ $build_it -eq 0 ] ; then
      ./devbuild.sh --platform=${machine} --compiler=${compiler} --build-dir=${BUILD_DIR} --install-dir=${BIN_DIR} \
-        --clean --rrfs || fail "Build ${machine} ${compiler} FAILED"
+        --clean all || fail "Build ${machine} ${compiler} FAILED"


Does this --clean behave the same as it did before? Should we have both --clean and --remove?

Thanks for catching that bug! --clean does a make clean on the target you selected. I am not sure if --remove that does rm -rf build is needed now but I kept it just in case.

chan-hoo · 2022-08-24T16:08:35Z

@danielabdi-noaa, "./devbuild.sh -p=hera --bin-dir=exec upp" doesn't work correctly. The executable "upp.x" is not created in the "exec" directory but in "bin" directory.

danielabdi-noaa · 2022-08-24T16:43:24Z

@chan-hoo Hmm that is weird. It does put the upp.x executable in the exec directory for me. Maybe removing the build directory first can help?

$ ./devbuild.sh -p=hera --bin-dir=exec upp
....
[ 89%] Built target upp
[100%] Built target upp.x
Install the project...
-- Install configuration: "Release"
-- Installing: /scratch2/BMC/gsd-hpcs/Daniel.Abdi/ufs-srweather-app/exec/upp.x
-- Up-to-date: /scratch2/BMC/gsd-hpcs/Daniel.Abdi/ufs-srweather-app/include
$ ls exec
ufs_srweather_app.settings  upp.x

chan-hoo · 2022-08-24T18:11:32Z

@danielabdi-noaa, I don't know what happened. I removed the 'build' directory before I tested. What I got:
"CMake Warning:
Manually-specified variables were not used by the project:
EMC_EXEC_DIR"

[Chan-hoo.Jeon@hfe09 exec]$ ls
ufs_srweather_app.settings

[Chan-hoo.Jeon@hfe09 ufs-srweather-app]$ cd bin
[Chan-hoo.Jeon@hfe09 bin]$ ls
upp.x

danielabdi-noaa · 2022-08-24T18:35:53Z

@chan-hoo It looks like a recent PR has removed "EMC_EXEC_DIR" option which is what is causing the problem.
NOAA-EMC/UPP@8767929#diff-1e7de1ae2d059d21e1dd75d5812d5a34b0222cef273b7c3a2af62eb747f9d20a
I had a couple of days older UPP which was why I was not seeing it.
This is a good thing because now we can now build not only to "bin" and "exec" but any arbitrary exe directory.
UFS_UTILS is the only one that uses this hack but it have been removed too so I will check ...

danielabdi-noaa · 2022-08-24T18:56:50Z

@chan-hoo Could you please try again? I have updated for the UPP change regarding removal of EMC_EXEC_DIR=on/off.
Now it uses "exec_dir" to specifiy the excutable directory which is unnecssary in my opnion. There is alread a CMAKE_INSTALL_BINDIR for that purpose which GSI and others use. UFS_UTILS still uses "EMC_EXEC_DIR" so that still needs to be removed for us to build to any executable directory name.

chan-hoo · 2022-08-24T19:45:39Z

@danielabdi-noaa, It works well now. Thank you for the quick fix!

danielabdi-noaa · 2022-08-24T20:40:45Z

@chan-hoo Thanks for testing! I have created a PR to modify UFS_UTILS installation directory to be anything aside from bin and exec ufs-community/UFS_UTILS#685

panll

Tested on Hera. Works fine. Thanks!

danielabdi-noaa added 4 commits August 14, 2022 02:10

Build to exec directory instead of bin.

7c07edc

Rename src to sorc

ac1d194

Add separate building of rrfs components.

643437f

Temporarily change bin directory from "exec" to "bin"

2bed9be

danielabdi-noaa changed the title ~~Feature/nco standard~~ Changes to move SRW app towards NCO complaince Aug 14, 2022

danielabdi-noaa added the ci-hera-intel-WE Kicks off automated workflow test on hera with intel label Aug 14, 2022