Regional CTSM Simulations and Capability of Creating Mesh Files

[negin513]

Description of changes

Quick Summary:

This PR includes:
1 - Script that enables our users to easily set up and run regional CTSM simulations.
2- Python class and scripts for creating a mesh file from any netCDF file and visualize it.

• Detailed instructions for running CTSM regional cases are available in this link.

• Instructions on how to use ./mesh_maker.py for different domains including one example WRF domain are available here.

Additional Details:

In this PR, we first introduced mesh_type python class and its new capabilities.
This new mesh object enables our users to create a mesh file from any netCDF files with either 1D or 2D latitude and longitude coordinates.

• mesh_type is implemented in ./subset_data to create mesh file for surface dataset for easily running regional cases. Next, we added additional capabilities to automatically set up usermods necessary to run a regional case. This enable our users to create and run a regional case with 2 commands.
• For other purposes, mesh_type is implemented in a script called mesh_maker.py to create a mesh file from any netCDF file.

Please note that the method that is used for creating a mesh file is different from #1735, where we subset a global mesh file for a region.

Specific notes

Contributors other than yourself, if any: @swensosc

CTSM Issues Fixed :
Fixes #1513: Create ESMF mesh file for regional grids
Fixes #1773: High Resolution Regional Simulations
Fixes #1397: Generating ESMFMesh files for WRF-CTSM cases --> discussions #1832

Are answers expected to change? No

Any User Interface Changes?
1- Changes in subset_data user interface with addition of --create-mesh
2- Adding mesh_maker.py with the following user interface:

usage: mesh_maker.py [-h] --input INPUT [--output OUTPUT] [--outdir OUT_DIR]
                     --lat LAT_NAME --lon LON_NAME [--mask MASK_NAME]
                     [--area AREA_NAME] [--overwrite] [-v]

|------------------------------------------------------------------|
|---------------------  Instructions  -----------------------------|
|------------------------------------------------------------------|
This script creates ESMF unstructured GRID (mesh file) from a netcdf
file with valid lats and lons. Provided lats and lons can be 1D or 2D.

For example for running WRF-CTSM cases, the user can create a mesh
file for their domain :
    ./mesh_maker.py --input wrfinput_d01 --output my_region
        --lat XLAT --lon XLONG --verbose

optional arguments:
  -h, --help        show this help message and exit
  --input INPUT     Netcdf input file for creating ESMF mesh.
  --output OUTPUT   Name of the ESMF mesh created.
  --outdir OUT_DIR  Output directory (only if name of output mesh is not
                    defined)
  --lat LAT_NAME    Name of latitude varibale on netcdf input file. If none
                    given, looks to find variables that include 'lat'.
  --lon LON_NAME    Name of latitude varibale on netcdf input file. If none
                    given, looks to find variables that include 'lon'.
  --mask MASK_NAME  Name of mask varibale on netcdf input file. If none given,
                    create a fake mask with values of 1.
  --area AREA_NAME  Name of area variable on netcdf input file. If none given,
                    ESMF calculates element areas automatically.
  --overwrite       If meshfile exists, overwrite the meshfile.
  -v, --verbose     Increase output verbosity

Testing performed, if any: regional simulations for different regions.

[adrifoster]

@negin513 I am working on bringing this PR in, after removing some of the conda and git workflow-related things @ekluzek was working on. This branch is here.

How would you like me to work from here? Shall I just push to your branch/PR? If so, I think I need collaborator access on your CTSM fork. Otherwise I can just make another PR to CTSM, which would include all of the code mods you made (and still be linked to you).

I'm happy to do whatever you'd like!

[negin513]

Hello @adrifoster , Thanks! I added you to the collaborators on the fork.

[negin513]

should we remove the --create-domain feature of this work since mct is no longer being supported for CESM3?

No @wwieder and @samsrabin : for WRF-CTSM runs domain files are needed. So I would suggest keeping --create-domain to create domain files for WRF-CTSM

[negin513]

@wwieder as I recall we still need the domain files. I think they are to create the mesh files, but don't quite remember.

Yes @ekluzek , domain files are needed for WRF-CTSM (if the procedure is still the same). ;-)

[negin513]

Hello. Everyone,
Comparing the code with my last commit (3588342) , I noticed the dask arrays are converted to numpy arrays, which means it does not benefit from the lazy loading offered by dask.

I suggest reverting these back to use dask array instead, which should make subsetting for high resolution mesh faster.

For example:

git diff 543e424 python/ctsm/site_and_regional/mesh_type.py

-            self.center_lat2d = da.from_array(self.center_lats)
-            self.center_lon2d = da.from_array(self.center_lons)
+            # self.center_lat2d = da.from_array(np.array(self.center_lats))
+            # self.center_lon2d = da.from_array(np.array(self.center_lons))
+            self.center_lat2d = np.array(self.center_lats)
+            self.center_lon2d = np.array(self.center_lons)
....

I can start from [3588342](https://github.com/ESCOMP/CTSM/pull/1892/commits/358834295a2377bcd8edf03200f10a91ebff63fd) and check the changes made here. 

[adrifoster]

All system and python tests are passing. @ekluzek I think this just needs another review before it is ready to be merged.

[ekluzek]

Awesome, this is great to see. I resolved the conversations I had after checking.

I do have one change to do from the last conversation. Go ahead fix that and resolve and merge this in.

[samsrabin]

I think it would be best to get these instructions (once updated, if necessary) moved into the CTSM User's Guide as part of this PR. I can help if needed.

• Detailed instructions for running CTSM regional cases are available in this link.
• Instructions on how to use ./mesh_maker.py for different domains including one example WRF domain are available here.

Just wanted to re-raise this. It'd probably be best to get these brought in to the documentation with this PR, rather than letting it wait until users ask us questions about it. I can do this and submit a PR to negin513:subset_mesh_dask if you prefer, @adrifoster, although I'd need you to review to make sure the instructions are all correct.

[negin513]

Thanks Sam. I am going to add them later today. Do you parse md in you docs build? If not i can change them to rst.

…

On Thu, Apr 25, 2024 at 7:28 AM Sam Rabin ***@***.***> wrote: I think it would be best to get these instructions (once updated, if necessary) moved into the CTSM User's Guide as part of this PR. I can help if needed. - *Detailed instructions for running CTSM regional cases are available in this link <https://gist.github.com/negin513/d71268419f808b91a82cde5530d390b1#file-regional_ctsm-md>.* - *Instructions on how to use ./mesh_maker.py for different domains including one example WRF domain are available here <https://gist.github.com/negin513/dde385a28145928786e2bc498e2b5137>.* Just wanted to re-raise this. It'd probably be best to get these brought in to the documentation with this PR, rather than letting it wait until users ask us questions about it. I can do this and submit a PR to negin513:subset_mesh_dask <https://github.com/negin513/ctsm/tree/subset_mesh_dask> if you prefer, @adrifoster <https://github.com/adrifoster>. — Reply to this email directly, view it on GitHub <#1892 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEEKQGAGBPYR565CRL2K7DLY7EAJNAVCNFSM6AAAAAARZUT4NKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZXGE4DGMBZGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[negin513]

Okay reading through conf.py, it seems like you don’t current parse markdown file. If interested, I can add that capability. LMK!

…

On Thu, Apr 25, 2024 at 7:35 AM Negin Sobhani ***@***.***> wrote: Thanks Sam. I am going to add them later today. Do you parse md in you docs build? If not i can change them to rst. On Thu, Apr 25, 2024 at 7:28 AM Sam Rabin ***@***.***> wrote: > I think it would be best to get these instructions (once updated, if > necessary) moved into the CTSM User's Guide as part of this PR. I can help > if needed. > > > - *Detailed instructions for running CTSM regional cases are > available in this link > <https://gist.github.com/negin513/d71268419f808b91a82cde5530d390b1#file-regional_ctsm-md>.* > - *Instructions on how to use ./mesh_maker.py for different domains > including one example WRF domain are available here > <https://gist.github.com/negin513/dde385a28145928786e2bc498e2b5137>.* > > Just wanted to re-raise this. It'd probably be best to get these brought > in to the documentation with this PR, rather than letting it wait until > users ask us questions about it. I can do this and submit a PR to > negin513:subset_mesh_dask > <https://github.com/negin513/ctsm/tree/subset_mesh_dask> if you prefer, > @adrifoster <https://github.com/adrifoster>. > > — > Reply to this email directly, view it on GitHub > <#1892 (comment)>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AEEKQGAGBPYR565CRL2K7DLY7EAJNAVCNFSM6AAAAAARZUT4NKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZXGE4DGMBZGY> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[samsrabin]

@negin513 If it's not too much trouble, that'd be great! As long as it can do things like linking to other notes/sections in notes, being included in chapter lists, etc.