Merge pull request #1892 from negin513/subset_mesh_dask

Regional CTSM Simulations and Capability of Creating Mesh Files

.git-blame-ignore-revs

@@ -2,6 +2,7 @@
 b429b63824e09f82e95d2982f14311cbbd8e4a37
 d229b5c6689efc4c2a6cef077515c4ccd5c18ff6
 4cd83cb3ee6d85eb909403487abf5eeaf4d98911
+d229b5c6689efc4c2a6cef077515c4ccd5c18ff6
 0aa2957c1f8603c63fa30b11295c06cfddff44a5
 2cdb380febb274478e84cd90945aee93f29fa2e6
 e44dc469439e02e9ee582dab274d890ebdfab104

cime_config/usermods_dirs/f09_37x288pt_PanBoreal/shell_commands

@@ -0,0 +1,8 @@
+
+# Change below line if you move the subset data directory
+
+./xmlchange CLM_USRDAT_DIR='$DIN_LOC_ROOT/lnd/clm2/regional_datasets/f09_38x288pt_PanBoreal'
+
+./xmlchange ATM_DOMAIN_MESH='$CLM_USRDAT_DIR/domain.lnd.fv0.9x1.25_gx1v7_f09_38x288pt_PanBoreal_c230524_ESMF_UNSTRUCTURED_MESH.nc'
+./xmlchange LND_DOMAIN_MESH='$CLM_USRDAT_DIR/domain.lnd.fv0.9x1.25_gx1v7_f09_38x288pt_PanBoreal_c230524_ESMF_UNSTRUCTURED_MESH.nc'
+./xmlchange MASK_MESH='$CLM_USRDAT_DIR/domain.lnd.fv0.9x1.25_gx1v7_f09_38x288pt_PanBoreal_c230524_ESMF_UNSTRUCTURED_MESH.nc'

cime_config/usermods_dirs/f09_37x288pt_PanBoreal/user_nl_clm

@@ -0,0 +1,22 @@
+!----------------------------------------------------------------------------------
+! Users should add all user specific namelist changes below in the form of 
+! namelist_var = new_namelist_value 
+!
+! EXCEPTIONS: 
+! Set use_cndv           by the compset you use and the CLM_BLDNML_OPTS -dynamic_vegetation setting
+! Set use_vichydro       by the compset you use and the CLM_BLDNML_OPTS -vichydro           setting
+! Set use_cn             by the compset you use and CLM_BLDNML_OPTS -bgc  setting
+! Set use_crop           by the compset you use and CLM_BLDNML_OPTS -crop setting
+! Set spinup_state       by the CLM_BLDNML_OPTS -bgc_spinup      setting
+! Set co2_ppmv           with CCSM_CO2_PPMV                      option
+! Set fatmlndfrc         with LND_DOMAIN_PATH/LND_DOMAIN_FILE    options
+! Set finidat            with RUN_REFCASE/RUN_REFDATE/RUN_REFTOD options for hybrid or branch cases
+!                        (includes $inst_string for multi-ensemble cases)
+!                        or with CLM_FORCE_COLDSTART to do a cold start
+!                        or set it with an explicit filename here.
+! Set maxpatch_glc       with GLC_NEC                            option
+! Set glc_do_dynglacier  with GLC_TWO_WAY_COUPLING               env variable
+!----------------------------------------------------------------------------------
+
+
+fsurdat = '$CLM_USRDAT_DIR/surfdata_f09_38x288pt_PanBoreal_hist_16pfts_Irrig_CMIP6_simyr2000_c230523.nc'

doc/design/python_script_user_interface.rst

@@ -81,3 +81,10 @@ More verbose output should go in ``logger.info`` or ``logger.debug`` statements
       * e.g. You might want to output a ``logging.debug`` statement for every variable in a file you are editing.
 
 Near the top of each python module where logging is used, there should be a line, ``logger = logging.getLogger(__name__)``. Then logging statements should be done using statements like ``logger.info(...)``, *not* ``logging.info(...)``: this allows more contextual information in logging output.
+
+====================================================
+ Considerations on inclusion of python packages
+====================================================
+
+Since, this is somewhat an implementation detail the discussion for this is in ``../../python/README.python_pkgs.rst``. The python
+packages used is somewhat both an important part of the user interface, the tool design, and an implementation detail.

doc/source/conf.py

@@ -45,8 +45,8 @@
 # The suffix(es) of source filenames.
 # You can specify multiple suffix as a list of string:
 #
-# source_suffix = ['.rst', '.md']
-source_suffix = '.rst'
+source_suffix = ['.rst', '.md']
+#source_suffix = '.rst'
 
 # The master toctree document.
 master_doc = 'index'

doc/source/how-to-make-mesh.md

@@ -0,0 +1,98 @@
+# Creating an ESMF mesh file from a netCDF file
+
+This gist includes instructions for creating and visualizing a mesh file from a netcdf file with valid 1D or 2D lats and lons coordinates.
+
+* **ESMF Mesh file** aka **Unstructured Grid File Format** is a netcdf file (format) that includes the information about the grids coordinates and their connectivity to each other. 
+
+Additional information about ESMF mesh files are available [here](https://earthsystemmodeling.org/docs/release/ESMF_8_0_1/ESMF_refdoc/node3.html#SECTION03028200000000000000).
+
+------
+
+In this example, we will use `./mesh_maker.py` which uses `mesh_type.py` to create a mesh file and visualize it.
+
+1- First clone my fork and branch that includes these capabilities:
+``` Shell
+git clone https://github.com/negin513/ctsm.git ctsm_mesh
+cd ctsm_mesh
+
+git checkout subset_mesh_dask
+```
+
+2- Next run mesh_maker.py for a netcdf file:
+
+```
+cd tools/site_and_regional
+```
+Check all the avaialble options:
+
+```
+./mesh_maker.py --help
+```
+
+The output shows all available options for this script:
+```
+|------------------------------------------------------------------|
+|---------------------  Instructions  -----------------------------|
+|------------------------------------------------------------------|
+This script creates ESMF unstructured GRID (mesh file) from a netcdf
+file with valid lats and lons. Provided lats and lons can be 1D or 2D.
+
+For example for running WRF-CTSM cases, the user can create a mesh
+file for their domain :
+    ./mesh_maker.py --input wrfinput_d01 --output my_region
+        --lat XLAT --lon XLONG --verbose
+
+optional arguments:
+  -h, --help        show this help message and exit
+  --input INPUT     Netcdf input file for creating ESMF mesh.
+  --output OUTPUT   Name of the ESMF mesh created.
+  --outdir OUT_DIR  Output directory (only if name of output mesh is not
+                    defined)
+  --lat LAT_NAME    Name of latitude varibale on netcdf input file. If none
+                    given, looks to find variables that include 'lat'.
+  --lon LON_NAME    Name of latitude varibale on netcdf input file. If none
+                    given, looks to find variables that include 'lon'.
+  --mask MASK_NAME  Name of mask varibale on netcdf input file. If none given,
+                    create a fake mask with values of 1.
+  --area AREA_NAME  Name of area variable on netcdf input file. If none given,
+                    ESMF calculates element areas automatically.
+  --overwrite       If meshfile exists, overwrite the meshfile.
+  -v, --verbose     Increase output verbosity
+  
+  ```
+
+Let's create a mesh file from a netcdf file with 1D lats and lons. On the sample files provided 1D lat and long coordinates are saved on `lsmlat` and `lsmlon` variables. 
+
+```
+./mesh_maker.py --input /glade/scratch/negins/example_files/surfdata_4x5_hist_78pfts_CMIP6_simyr1850_275.0-330.0_-40-15_c220705.nc --output test_mesh_1d.nc --lat lsmlat --lon lsmlon --overwrite
+```
+`--verbose` option also provide additional information for debugging. 
+
+This script will create regional and global mesh plots. For example for the above files, the plos are:
+test_mesh_1d_regional.png
+![image](https://user-images.githubusercontent.com/17344536/200441736-972a8136-5c05-4bc9-9bca-b498d972914a.png)
+
+
+test_mesh_1d_global.png
+
+![image](https://user-images.githubusercontent.com/17344536/200441753-d06e95d1-d85b-4216-9c23-d11ba89a31e4.png)
+
+
+
+------
+  ## Creating Mesh files for a WRF domain:
+For running WRF-CTSM cases, we need to create ESMF mesh files for the WRF domain. We can create mesh file from wrfinput (wrf initial condition files). wrfinput has 2D coordinate information on `XLAT` and `XLONG` variable.
+
+For example, let's create a mesh file from a WRF input file for WRF-CTSM run.  
+ ```
+./mesh_maker.py --input /glade/scratch/negins/example_files/wrfinput_d01  --output test_mesh_wrf.nc --lat XLAT --lon XLONG --overwrite
+```
+
+This produce mesh files for running for our WRF domain. 
+
+Here is how the regional plot looks like for this mesh file:
+
+ ![image](https://user-images.githubusercontent.com/17344536/200442002-1ee5595c-9252-4934-a07c-2f6ad86aff1b.png)
+
+
+

py_env_create

@@ -20,6 +20,7 @@ if [ $error != 0 ]; then
    echo "For notes on installing on a user system see: https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html"
    echo "Error code was $error"
    cat condahelp.txt
+   rm condahelp.txt
    exit -1
 fi
 rm condahelp.txt
@@ -44,7 +45,7 @@ usage() {
   echo "[-v|--verbose]  "
   echo "     Run with verbose mode for the install so you see the progress bar"
   echo "[-f|--file <file>]  "
-  echo "     Conda environment file to use (can be a text format or YAML format)"
+  echo "     Conda environment requirements text file to use (text format) in addition to the others"
   echo "     Assumed to be under the directory: $condadir"
   echo "     Default is: $condafile"
   echo "[--option <option>]  "
@@ -80,6 +81,9 @@ while [ $# -gt 0 ]; do
    shift
 done
 
+#
+# Error checking options and setup
+#
 if [ ! -f $condadir/$condafile ]; then
    echo "$condadir/$condafile does NOT exist"
    echo "Use the --file option with a valid filename"
@@ -89,11 +93,13 @@ fi
 echo "Use conda to install the python environment needed to run the CTSM python tools in the conda environment: $ctsm_python"
 echo "Using the file: $condadir/$condafile"
 
+#
 # Check if the environment already exists, if it does continue, if not create it
+#
 conda list -n $ctsm_python >& /dev/null
 if [ $? != 0 ]; then
    echo "Create $ctsm_python"
-   cmd="conda create --force -n $ctsm_python -q"
+   cmd="conda create --force --name $ctsm_python --quiet"
    echo "$cmd"
    $cmd
    if [ $? != 0 ]; then
@@ -103,12 +109,20 @@ if [ $? != 0 ]; then
 else
    echo "$ctsm_python environment already exists"
 fi
-echo "Install $ctsm_python this can take a long time, be patient...."
-verbosity="-q"
+#
+# Install the environemnt
+#
+echo "Install $ctsm_python this can take a long time (12 to 20 minutes is expected), be patient...."
+echo "    ...."
+echo "    ...."
+echo "    ...."
+verbosity="--quiet"
+loglevel="ERROR"
 if [ "$verbose" == "Yes" ]; then
-  verbosity="-v"
+  verbosity="--verbose"
+  loglevel="INFO"
 fi
-cmd="conda install --yes $verbosity -c conda-forge -n $ctsm_python --file $condadir/$condafile $option"
+cmd="conda install --yes $verbosity --channel conda-forge --name $ctsm_python $condadir/$condafile $option"
 echo "$cmd"
 $cmd
 if [ $? != 0 ]; then
@@ -117,6 +131,9 @@ if [ $? != 0 ]; then
    echo "Change the file and try again"
    exit -2
 fi
+#
+# Report on success
+#
 echo "Successfully installed the $ctsm_python python environment"
 echo
 echo "activate the environment by doing the following..."

python/README.python_pkgs.rst

@@ -0,0 +1,33 @@
+.. sectnum::
+
+.. contents::
+
+=====================================================================================
+ Requirements to consider for python packages to be included for CTSM python tools
+=====================================================================================
+
+Requirements for CTSM python tools:
+
+Any added dependencies should be discussed and approved by the CTSM-software team. Criteria for evaluation of third-party packages
+include:
+- How much value does this package provide to us beyond what we could get without it?
+- How difficult it is to install this package (including: how much does this complicate / slow down the creation of the conda
+environment)?
+- How stable is this package?
+- How well maintained is this package?
+- Are there other packages that are more stable or better maintained that would provide nearly the same level of value?
+- Tools that require extra packages should be done in a "contrib" type area out of the main part of tools (this would apply for
+advanced plotting capability for example)
+- We need to be able to reproduce working conda environments minimally on our test machines (currently cheyenne and izumi), but also
+on any machine that we run CTSM on. If there is a machine that we can run CTSM on that we can't build the conda environments or run
+the tools on -- that needs to be fixed.
+- We need to tell the user how long to expect the conda environment to load, and give them options if the conda load is taking too
+long
+- Conda environments need to build robustly even for users who don't have ctsm_pylib loaded in their conda environment
+- Currently dask will NOT be something we require for any of the main CTSM tools
+- Currently we won't use conda-lock
+- We specify the black version exactly so that black will function identically for all users
+- We specify the pylint version exactly because pylint is finicky with version and we need it to work identically for all developers
+- We might remove the need for python packages that aren't providing enough utility
+Packages where specific versions seem to be required will have the version requirements in a least the >= form if not an exact
+version

python/conda_env_ctsm_py.txt

@@ -22,3 +22,5 @@ xesmf
 numba=0.55.2  # Avoid 0.56 until numpy>=1.20.  This is the minimum for xesmf
 pylint=2.8.3
 black=22.3.0  # NOTE: The version here needs to be coordinated with the black github action under ../.github/workflows
+cartopy=0.20.2
+matplotlib=3.3.2

python/conda_env_ctsm_py_latest.txt

@@ -12,3 +12,5 @@ xarray>=2022.3.0
 pylint>=2.8.3,<2.9.0  # Once, you move off of 2.8.3, make lint shows 2 errors, at 2.11.1 there are 2 more errors beyond that
                       # By, 2.14.4 there are 75 errors
 black>=22.6.0       # NOTE: The version here needs to be coordinated with the black github action under ../.github/workflows
+cartopy=0.20.2
+matplotlib=3.3.2

python/ctsm/.pylintrc

@@ -433,6 +433,12 @@ good-names=i,
            k,
            ex,
            Run,
+           ds,
+           m,
+           n,
+           l1,
+           l2,
+           ax,
            _,
 # --- default list is above here, our own list is below here ---
 # Allow logger as a global name in each module, because this seems to follow general recommended convention: