Merge release/public-v3 (#509)

* run_upp script got the following updates:
-updated directory name
-updated executable path
-added error message for binarynemsiompiio option
-added netcdfpara option

* Further refinement from Tracy's comment.

* Minor fix.

* More minor change from Tracy.

* Link update from Tracy.

* Documentation updates for release

* minor updates based on comments

* minor revisions to documentation

* Increment version for public release

* update informat - only netcdfpara

* Removed netcdf option and rename GFS input file names.

* Minor change.

Co-authored-by: kayee <kayee.wong@noaa.gov>
Co-authored-by: Tracy <hertneky@ucar.edu>
Co-authored-by: Kate Fossell <you@example.com>

VERSION

@@ -1 +1 @@
-10.2.0
+10.2.0

docs/Acknowledgments.rst

@@ -18,4 +18,4 @@ acknowledge the Developmental Testbed Center UPP Team.
 
 For referencing this document please use:
 
-UPP Users Guide V9.0.0, 24 pp.
+UPP Users Guide V10.1.0, 24 pp.

docs/AddNewVariable.rst

@@ -93,8 +93,7 @@ with examples in the sections below.
 
     The variable is declared in one of these modules defining files depending on its dimension.
 
-5.  Read model output if necessary: INITPOST_GFS_NETCDF_PARA.f (current operational netcdf output with GFS V16),
-    INITPOST_NETCDF.f (LAM FV3 netcdf)
+5.  Read model output if necessary: INITPOST_NETCDF.f
 
     Check first to see if all variables needed to derive your new variable are already available in UPP. If not,
     you will need to use one of these files for reading the model output files. The appropriate one will need to
@@ -254,8 +253,8 @@ with examples in the sections below.
 
      tg3(:,:)
 
-5. Read the field from the GFS model output file by adding the new variable into INITPOST_GFS_NETCDF_PARA.f.
-   This file is used for reading the GFS model FV3 output files in netcdf format.
+5. Read the field from the GFS model output file by adding the new variable into INITPOST_NETCDF.f.
+   This file is used for reading the GFS model FV3 output files in parallel netcdf format.
 
    User procedure
     - Add to top section of the routine in ‘use vrbls2d’ to initiate the new variable as:
@@ -317,12 +316,12 @@ with examples in the sections below.
 
 7. Build or rebuild the code for changes to take effect before running your UPP run script.
 
-   User procedure IF you already have the code built. Otherwise, see the User's Guide for instructions on building.
+   User procedure for building on pre-configured machines. Otherwise, see the User's Guide for instructions on building.
 
     ::
 
-    >> cd UPP/build
-    >> make install
+    >> cd UPP/tests
+    >> ./compile_upp.sh
 
    Assuming the modified code built successfully and you were able to produce Grib2 output, you can check the Grib2
    file for your new variable.
@@ -343,4 +342,3 @@ with examples in the sections below.
          number of latitudes between pole-equator=96 #points=73728
          lat 89.284225 to -89.284225
          lon 0.000000 to 359.062500 by 0.937500
-

docs/CodeOverview.rst

@@ -2,19 +2,32 @@
 Code Overview
 *************
 
-The UPP can be used to post-process WRF-ARW, WRF-NMM, NMMB, GFS, CFS, and FV3 forecasts with current
-support within UFS applications available for FV3 only. It can ingest FV3 write component files in
-netCDF and binarynemsiompiio format.
+The UPP is used to post-process model forecasts and provides the capability to compute a variety of
+diagnostic fields and interpolate to pressure levels or other vertical coordinates.
 
-UPP Functionalities:
+The UPP also incorporates the Joint Center for Satellite Data Assimilation (JCSDA) Community Radiative
+Transfer Model (CRTM) to compute model derived brightness temperature (TB) for various instruments and
+channels. This additional feature enables the generation of a number of simulated satellite products
+including GOES products.
 
-   -  Interpolates the forecasts from the models native vertical coordinate to NWS standard output
-      levels (e.g., pressure, height) and computes mean sea level pressure. If the requested parameter
-      is on a models native level, then no vertical interpolation is performed.
+Output from the UPP is in National Weather Service (NWS) and World Meteorological Organization (WMO)
+`GRIB2 <https://www.nco.ncep.noaa.gov/pmb/docs/grib2/>`_ format and can be used directly by
+visualization, plotting, or verification packages, or for further downstream post-processing, e.g.
+statistical post-processing techniques.
 
-   -  Computes diagnostic output quantities (e.g., convective available potential energy, helicity,
-      relative humidity). A full list of fields that can be generated by the UPP is provided in
-      :doc:`UPP_GRIB2_Table`.
+Examples of UPP products include:
 
-   -  Outputs the results in NWS and WMO standard GRIB2 format (see
-      `Grib documentation <http://www.nco.ncep.noaa.gov/pmb/docs/>`_).
+- T, Z, humidity, wind, cloud water, cloud ice, rain, and snow on pressure levels
+- SLP, shelter level T, humidity, and wind fields
+- Precipitation-related fields
+- PBL-related fields
+- Severe weather products (e.g. CAPE, Vorticity, Wind shear)
+- Radiative/Surface fluxes
+- Cloud related fields
+- Aviation products
+- Radar reflectivity products
+- Satellite look-alike products
+
+A full list of fields that can be generated by the UPP is provided in :doc:`UPP_GRIB2_Table`.
+
+Support for the community UPP is provided through the `UFS Forum <https://forums.ufscommunity.org/>`_.

docs/InputsOutputs.rst

@@ -14,10 +14,24 @@ Input files
 ===========
 
 The UPP requires the following input files:
+ - The model forecast file
  - The itag namelist file
  - The GRIB2 control file
  - Additional data files (e.g. lookup tables, coefficient files for satellite)
 
+--------------
+Model Forecast
+--------------
+
+The UPP ingests FV3 write component files in parallel netCDF format.
+
+The table below is a list of the unified model variables available from the FV3 model core. Whether a
+specific variable is able to be read by UPP relies on dependencies such as physics options and model.
+This table does not include variables that are diagnosed when running the UPP.
+
+UFS Unified Model Variables
+ - :doc:`UFS_unified_variables_table`
+
 ----
 ITAG
 ----
@@ -28,9 +42,9 @@ generated automatically within the UFS application workflow or stand-alone run s
 user-defined options. It should not be necessary to edit this. For description purposes, the namelist
 &model_inputs (:bolditalic:`itag` file) contains 7 lines for FV3:
 
-#. Name of the FV3 (pressure level) output file to be posted.
+#. Name of the FV3 (pressure level) output file to be post-processed.
 
-#. Format of FV3 model output (netcdf, binarynemsiompiio).
+#. Format of FV3 model output (netcdfpara).
 
 #. Format of UPP output (GRIB2)
 
@@ -54,22 +68,12 @@ which fields and levels to process.
 A default control file, :bolditalic:`postxconfig-NT.txt`, is provided and read by the UPP. For users
 wishing to customize the control file to add or remove fields and/or levels, they may do so by
 modifying the :bolditalic:`postcntrl.xml` and then remaking the text file as described in the later section
-:ref:`Creating the Flat Text File`.
+:ref:`create_txt_file`.
 
 .. Note::
    The control file names :bolditalic:`postxconfig-NT.txt` and :bolditalic:`postcntrl.xml` are generic
-   names and are different depending on the application used.
-
-The tables below list all fields that are included in the control files for the various UFS
-applications. All fields in the tables may not be present in your output depending on whether the field
-dependencies are available in your model output.
-
-UFS MRW Table (GFS model)
- - :doc:`MRW_GFSPRS_table`
-
-UFS SRW Tables (LAM - Limited Area Model)
- - :doc:`SRW_BGDAWP_table`
- - :doc:`SRW_BGRD3D_table`
+   names and are different depending on the application used. Control files for various operational
+   models are located in the :bolditalic:`UPP/parm` directory.
 
 Controlling which variables the UPP outputs
 -------------------------------------------
@@ -108,9 +112,11 @@ levels are currently available for output:
 - For PBL layer averages, the levels correspond to 6 layers with a thickness of 30 hPa each.
 - For flight level, the levels are 30 m, 50 m, 80 m, 100 m, 305 m, 457 m, 610 m, 914 m, 1524 m, 1829 m,
   2134 m, 2743 m, 3658 m, 4572 m, 6000 m, 7010 m.
-- For AGL radar reflectivity, the levels are 4000 and 1000 m (see Appendix A for details).
+- For AGL radar reflectivity, the levels are 4000 and 1000 m.
 - For surface or shelter-level output, the <level> is not necessary.
 
+.. _create_txt_file:
+
 Creating the Flat Text File
 ---------------------------
 
@@ -131,14 +137,15 @@ run-time failures with UPP. To run the validation:
     xmllint --noout --schema EMC_POST_CTRL_Schema.xsd postcntrl.xml
     xmllint --noout --schema EMC_POST_Avblflds_Schema.xsd post_avblflds.xml
 
-Once the xmls are validated, the user will need to generate the flat file. The makefile will call the
-perl program :bolditalic:`parm/POSTXMLPreprocessor.pl` to regenerate any post flat files
-:bolditalic:`postxconfig-NT.txt` where modifications were made since it was last run. Generate the flat
+Once the xmls are validated, the user will need to generate the flat file. The below command will run the
+perl program :bolditalic:`parm/POSTXMLPreprocessor.pl` to generate the post flat file. Generate the flat
 file:
 
 .. code-block:: console
 
-    make
+    /usr/bin/perl POSTXMLPreprocessor.pl your_user_defined_xml post_avblflds.xml your_user_defined_flat
+
+where *your_user_defined_xml* is your modified xml and *your_user_defined_flat* is the output text file.
 
 ============
 Output Files

docs/Installation.rst

@@ -7,89 +7,78 @@
 Building Stand-Alone
 ********************
 
+The UPP uses a CMake-based build system to integrate all the required components for building the UPP.
+Once built, the UPP can be run stand-alone (outside the UFS Applications) to post-process model output.
+
 =====================
 Software Requirements
 =====================
 
-Before installing the UPP code, it is necessary to ensure that you have the required libraries
-available on your system. These libraries include:
-
-  - The external NCEP libraries
-    https://github.com/NOAA-EMC/NCEPLIBS-external
-
-  - The NCEP libraries
-    https://github.com/NOAA-EMC/NCEPLIBS
-
-An introduction of each can be found in their respective top level :bolditalic:`README.md` files.
-Detailed instructions for building the libraries on various platforms can be found in the
-**NCEPLIBS-external/doc** directory.
+The UPP is tested on a variety of research platforms, including NOAA HPC systems (e.g. Hera, Orion) and
+the NCAR HPC Cheyenne. These supported platforms are pre-configured for building and running the UPP and already
+have the required libraries available via `HPC-Stack <https://github.com/NOAA-EMC/hpc-stack>`_ in a centralized
+location. The HPC-Stack is a script-based build system that builds the software stack required by UFS components.
 
-Certain machines do have the NCEP libraries in a pre-installed location for use to build UPP. Paths to
-these pre-installed libraries are available on the
-`UFS-SRW wiki <https://github.com/ufs-community/ufs-srweather-app/wiki/Supported-Platforms-and-Compilers>`_
-and include platform name and compiler version.
+Users working on unsupported platforms will need to install the HPC-Stack on their system and can do so following
+the instructions in the `HPC-Stack User's Guide <https://hpc-stack.readthedocs.io/en/latest/>`_.
 
 ============================
 Obtaining and Installing UPP
 ============================
 
-Building and running UPP V9.0.0 has been tested on the following platforms using pre-configured libraries.
+Building and running UPP V10.1.0 has been tested and is supported on the following pre-configured platforms.
 
 +---------------+----------------------+
 | System        | Compiler and Version |
 +===============+======================+
-| NCAR Cheyenne | Intel 19.1.1         |
+| NCAR Cheyenne | Intel 2021.2         |
 |               +----------------------+
-|               | GNU 9.1.0            |
 |               | GNU 10.1.0           |
 +---------------+----------------------+
 | NOAA Hera     | Intel 18.0.5.274     |
 +---------------+----------------------+
+| NOAA Orion    | Intel 2018.4         |
++---------------+----------------------+
 
-Move to the directory where you want to clone and build UPP and clone the repository into the directory
-UPP.
+Move to the directory where you want to install UPP and clone the repository.
 
 .. code-block:: console
 
     git clone -b branch-or-tag-name https://github.com/NOAA-EMC/UPP
 
 where, ``branch-or-tag-name`` is the release branch or tag you wish to clone.
 
-Move into the top level UPP directory and create and move into the build directory. Then build the UPP code
-using the cmake utility.
-The path ``INSTALL_PREFIX`` should point to the location of the pre-installed NCEP libraries.
+Move to the directory with the build script and build the UPP.
 
 .. code-block:: console
 
-    cd UPP
-    mkdir build && cd build
+    cd UPP/tests
 
-    cmake .. -DCMAKE_INSTALL_PREFIX=.. -DCMAKE_PREFIX_PATH=${INSTALL_PREFIX}
-    make install
+    ./compile_upp.sh
 
 .. note::
-   To build in debug mode, you can add :bolditalic:`-DCMAKE_BUILD_TYPE=Debug` to the cmake command.
+   To build in debug mode, you can add :bolditalic:`-DCMAKE_BUILD_TYPE=Debug` to the *cmake_opts*
+   parameter in the :bolditalic:`compile_upp.sh` script.
    This removes compiler optimization flags and adds -g to the fortran compilation. You can also use
    :bolditalic:`-DCMAKE_BUILD_TYPE=RELWITHDEBINFO`, which gives the -g, but keeps the -O2 optimization
    for the fortran compilation.
 
 Move back to the top level UPP directory and create a directory for the CRTM fix files to be unpacked
 in. Download the fix files from the Github `release page
-<https://github.com/NOAA-EMC/UPP/releases/tag/upp_v9.0.0>`_ or use the wget command. Unpack the
+<https://github.com/NOAA-EMC/UPP/releases/tag/upp_v10.1.0>`_ or use the wget command. Unpack the
 tar file.
 
 .. code-block:: console
 
     cd ../
     mkdir crtm && cd crtm
-    wget https://github.com/NOAA-EMC/UPP/releases/download/upp_v9.0.0/fix.tar.gz
+    wget https://github.com/NOAA-EMC/UPP/releases/download/upp_v10.1.0/fix.tar.gz
     tar -xzf fix.tar.gz
 
 .. note::
-   To make a clean build, simply remove both the **/build** directory and the
-   :bolditalic:`bin/upp.x` executable and then re-create the build from step #2. This is recommended if a
-   mistake is made during the installation process. If a simple change is made to the code, you can simply
-   type :bolditalic:`make install` again in the pre-existing build directory.
+   To make a clean build, simply remove both the **tests/build** and **tests/install** directories and the
+   :bolditalic:`exec/upp.x` executable and then rerun the :bolditalic:`compile_upp.sh script. This is
+   recommended if a mistake is made during the installation process.
    
 =======================
 UPP Directory Structure
@@ -98,13 +87,10 @@ UPP Directory Structure
 Under the main directory **UPP** reside the following relevant subdirectories (The * indicates a
 directory that exists only after the build is complete):
 
-     | **bin***: Contains the :bolditalic:`upp.x` executable after successful compilation
-
-     | **build**: Contains the UPP build
+     | **exec***: Contains the :bolditalic:`upp.x` executable after successful compilation
 
-     | **include***: Contains include modules built/used during compilation of UPP
-
-     | **lib***: Libraries built/used by UPP that are separate from NCEPlibs
+     | **modulefiles**: Contains modulefiles for specific platforms and compilers for building on
+       pre-configured machines.
 
      | **parm**: Contains parameter files, which can be modified by the user to control how the post
        processing is performed.
@@ -114,3 +100,7 @@ directory that exists only after the build is complete):
 
      | **sorc**: Contains source codes for:
      |   - **ncep_post.fd**: Source code for the UPP
+
+     | **tests**: Contains the scripts used to install UPP
+     |   - **build***: Contains the UPP build
+     |   - **install***: Contains the installed executable (bin/upp.x), modules, and libraries

docs/Introduction.rst

@@ -2,32 +2,16 @@
 Introduction
 ************
 
-The Unified Post Processor (UPP) software package is a software package designed to generate useful
-products from raw model output. The UPP is currently used in operations with the Global Forecast
-System (GFS), GFS Ensemble Forecast System (GEFS), North American Mesoscale (NAM), Rapid Refresh (RAP),
-High Resolution Rapid Refresh (HRRR), Short Range Ensemble Forecast (SREF), Hurricane WRF (HWRF)
-applications, and is also used in Unified Forecasting System (UFS) applications. The UPP provides the
-capability to compute a variety of diagnostic fields and interpolate to pressure levels or other
-vertical coordinates. UPP also incorporates the Joint Center for Satellite Data Assimilation (JCSDA)
-Community Radiative Transfer Model (CRTM) to compute model derived brightness temperature (TB) for
-various instruments and channels. This additional feature enables the generation of a number of
-simulated satellite products including GOES products. Output from the UPP is in National Weather
-Service (NWS) and World Meteorological Organization (WMO) GRIB2 format and can be used directly by
-visualization, plotting, or verification packages, or for further downstream post-processing, e.g.
-statistical post-processing techniques.
+The Unified Post Processor (UPP) is a software package designed to generate useful
+products from raw model output.
 
-Examples of UPP products include:
+The UPP is currently used in operations with the Global Forecast System (GFS), GFS Ensemble Forecast
+System (GEFS), North American Mesoscale (NAM), Rapid Refresh (RAP), High Resolution Rapid Refresh
+(HRRR), Short Range Ensemble Forecast (SREF), and Hurricane WRF (HWRF) applications. It is also used
+in the Unified Forecasting System (UFS), including the Rapid Refresh Forecast System (RRFS), Hurricane
+Application Forecasting System (HAFS), and the Medium Range Weather (MRW) and Short Range Weather (SRW)
+Applications.
 
-- T, Z, humidity, wind, cloud water, cloud ice, rain, and snow on pressure levels
-- SLP, shelter level T, humidity, and wind fields
-- Precipitation-related fields
-- PBL-related fields
-- Severe weather products (e.g. CAPE, Vorticity, Wind shear)
-- Radiative/Surface fluxes
-- Cloud related fields
-- Aviation products
-- Radar reflectivity products
-- Satellite look-alike products
-
-Support for the UFS UPP is provided through the UFS Forum by the Developmental Testbed Center (DTC) for
-FV3-based applications.
+This software package can be run inline, built as a library to be used by the model, and offline,
+built stand-alone and run separate from the model. This documentation largely details the offline
+procedures.

docs/Running.rst

@@ -64,7 +64,7 @@ Run Script Overview
 
        | **TOP_DIR**: Top level directory for building and running UPP
        | **DOMAINPATH**: Working directory for this run
-       | **UNIPOST_HOME**: Location of the **UPP** directory
+       | **UPP_HOME**: Location of the **UPP** directory
        | **POSTEXEC**: Location of the **UPP** executable
        | **modelDataPath**: Location of the model output data files to be processed
        | **txtCntrlFile**: Name and location of the flat text file that lists desired fields for
@@ -82,7 +82,7 @@ Run Script Overview
 
 3. Specify the format for the input model files and output UPP files
 
-       | **inFormat**: Format of the model data ("binarynemsiompiio": GFS only or "netcdf": GFS/LAM)
+       | **inFormat**: Format of the model data ("netcdfpara")
        | **outFormat**: Format of output from UPP ("grib2")
 
 4. Specify the forecast cycles to be post-processed