diff --git a/docs/UsersGuide/source/ConfigWorkflow.rst b/docs/UsersGuide/source/ConfigWorkflow.rst index ea7b4b5479..e5299a534c 100644 --- a/docs/UsersGuide/source/ConfigWorkflow.rst +++ b/docs/UsersGuide/source/ConfigWorkflow.rst @@ -410,7 +410,7 @@ A standard set of environment variables has been established for *nco* mode to s Only *community* mode is fully supported for this release. *nco* mode is used by those at the Environmental Modeling Center (EMC) and Global Systems Laboratory (GSL) who are working on pre-implementation operational testing. Other users should run the SRW App in *community* mode. ``envir, NET, model_ver, RUN``: - Standard environment variables defined in the NCEP Central Operations WCOSS Implementation Standards document. These variables are used in forming the path to various directories containing input, output, and workflow files. The variables are defined in the `WCOSS Implementation Standards `__ document (pp. 4-5) as follows: + Standard environment variables defined in the NCEP Central Operations WCOSS Implementation Standards document. These variables are used in forming the path to various directories containing input, output, and workflow files. The variables are defined in the `WCOSS Implementation Standards `__ document (pp. 4-5) as follows: ``envir``: (Default: "para") Set to "test" during the initial testing phase, "para" when running in parallel (on a schedule), and "prod" in production. @@ -427,6 +427,8 @@ A standard set of environment variables has been established for *nco* mode to s ``OPSROOT``: (Default: "") The operations root directory in *nco* mode. +.. _workflow-switches: + WORKFLOW SWITCHES Configuration Parameters ============================================= @@ -689,6 +691,8 @@ Non-default parameters for the ``make_sfc_climo`` task are set in the ``task_mak ``SFC_CLIMO_DIR``: (Default: "") The directory containing pre-generated surface climatology files to use when ``MAKE_SFC_CLIMO_TN`` is set to false. +.. _task_get_extrn_ics: + GET_EXTRN_ICS Configuration Parameters ========================================= @@ -770,6 +774,7 @@ Set parameters associated with NOMADS online data. ``NOMADS_file_type``: (Default: "nemsio") Flag controlling the format of the data. Valid values: ``"GRIB2"`` | ``"grib2"`` | ``"NEMSIO"`` | ``"nemsio"`` +.. _task_get_extrn_lbcs: GET_EXTRN_LBCS Configuration Parameters ========================================== @@ -1768,17 +1773,20 @@ Additional Parameters Typically, the following parameters must be set explicitly by the user in the configuration file (``config.yaml``) when executing the plotting tasks. ``COMOUT_REF``: (Default: "") - The directory where the GRIB2 files from post-processing are located. In *community* mode (i.e., when ``RUN_ENVIR: "community"``), this directory will correspond to the location in the experiment directory where the post-processed output can be found (e.g., ``$EXPTDIR/$DATE_FIRST_CYCL/postprd``). In *nco* mode, this directory should be set to the location of the COMOUT directory and end with ``$PDY/$cyc``. + The directory where the GRIB2 files from post-processing are located. In *community* mode (i.e., when ``RUN_ENVIR: "community"``), this directory will correspond to the location in the experiment directory where the post-processed output can be found (e.g., ``$EXPTDIR/$DATE_FIRST_CYCL/postprd``). In *nco* mode, this directory should be set to the location of the ``COMOUT`` directory and end with ``$PDY/$cyc``. For more detail on *nco* standards and directory naming conventions, see `WCOSS Implementation Standards `__ (particularly pp. 4-5). ``PLOT_FCST_START``: (Default: 0) The starting forecast hour for the plotting task. For example, if a forecast starts at 18h/18z, this is considered the 0th forecast hour, so "starting forecast hour" should be 0, not 18. If a forecast starts at 18h/18z, but the user only wants plots from the 6th forecast hour on, "starting forecast hour" should be 6. ``PLOT_FCST_INC``: (Default: 3) - Forecast hour increment for the plotting task. This may be the same as ``INCR_CYCL_FREQ``, or it may be a multiple of ``INCR_CYCL_FREQ``. For example, if ``INCR_CYCL_FREQ`` is set to 3, there will be forecast output every three hours for the duration of the forecast. If the user wants plots of all of this output, they should set ``PLOT_FCST_INC: 3``. If the user only wants plots for some of the output (e.g., every 6 hours), they should set ``PLOT_FCST_INC: 6``. However, there must be forecast output available at the designated increments to produce the plots. In this example, setting ``PLOT_FCST_INC: 7`` would produce an error because there is only forecast output available for hours 3, 6, 9, ..., etc. + Forecast hour increment for the plotting task. For example, if the user wants plots for each forecast hour, they should set ``PLOT_FCST_INC: 1``. If the user only wants plots for some of the output (e.g., every 6 hours), they should set ``PLOT_FCST_INC: 6``. ``PLOT_FCST_END``: (Default: "") The last forecast hour for the plotting task. For example, if a forecast run for 24 hours, and the user wants plots for each available hour of forecast output, they should set ``PLOT_FCST_END: 24``. If the user only wants plots from the first 12 hours of the forecast, the "last forecast hour" should be 12. +``PLOT_DOMAINS``: (Default: ["conus"]) + Domains to plot. Currently supported options are ["conus"], ["regional"], or both (i.e., ["conus", "regional"]). + Global Configuration Parameters =================================== diff --git a/docs/UsersGuide/source/Glossary.rst b/docs/UsersGuide/source/Glossary.rst index 2386d83177..2c0dbd9590 100644 --- a/docs/UsersGuide/source/Glossary.rst +++ b/docs/UsersGuide/source/Glossary.rst @@ -67,6 +67,9 @@ Glossary cycle-independent Describes a workflow task that only needs to be run once per experiment, regardless of the number of cycles in the experiment. + data assimilation + One of the major sources of error in weather and climate forecasts is uncertainty related to the initial conditions that are used to generate future predictions. Even the most precise instruments have a small range of unavoidable measurement error, which means that tiny measurement errors (e.g., related to atmospheric conditions and instrument location) can compound over time. These small differences result in very similar forecasts in the short term (i.e., minutes, hours), but they cause widely divergent forecasts in the long term. Data assimilation systems seek to mitigate this problem by combining the most timely observational data with other sources of data, such as historical data, to provide an analysis of possible atmospheric states and the probabilities of each. Errors in weather and climate forecasts can also arise because models are imperfect representations of reality. Data assimilation systems can use techniques including stochastic physics, which applies randomized perturbations to the physical tendencies or the physical parameters of a model, to compensate for model uncertainty. + dycore dynamical core Global atmospheric model based on fluid dynamics principles, including Euler's equations of motion. @@ -188,6 +191,9 @@ Glossary NWP Numerical Weather Prediction (NWP) takes current observations of weather and processes them with computer models to forecast the future state of the weather. + NWS + The `National Weather Service `__ (NWS) is an agency of the United States government that is tasked with providing weather forecasts, warnings of hazardous weather, and other weather-related products to organizations and the public for the purposes of protection, safety, and general information. It is a part of the National Oceanic and Atmospheric Administration (NOAA) branch of the Department of Commerce. + Orography The branch of physical geography dealing with mountains. @@ -229,6 +235,9 @@ Glossary Umbrella repository A repository that houses external code, or "externals," from additional repositories. + Updraft helicity + Helicity measures the rotation in a storm's updraft (rising) air. Significant rotation increases the probability that the storm will become a supercell thunderstorm or a tornado. See http://ww2010.atmos.uiuc.edu/(Gh)/guides/mtr/svr/modl/fcst/params/hel.rxml for more details on updraft helicity. + UPP The `Unified Post Processor `__ is software developed at :term:`NCEP` and used operationally to post-process raw output from a variety of :term:`NCEP`'s :term:`NWP` models, including the :term:`FV3`. diff --git a/docs/UsersGuide/source/InputOutputFiles.rst b/docs/UsersGuide/source/InputOutputFiles.rst index 8456049c93..f89a67dbfa 100644 --- a/docs/UsersGuide/source/InputOutputFiles.rst +++ b/docs/UsersGuide/source/InputOutputFiles.rst @@ -306,7 +306,7 @@ where: For example, a forecast using FV3GFS GRIB2 data that starts at 18h00 UTC would have a {cycle} value of 18, which is the 000th forecast hour. The LBCS file for 21h00 UTC would be named ``gfs.t18z.pgrb2.0p25.f003``. -In some cases, it may be necessary to specify values for ``EXTRN_MDL_FILES_*CS``variables. This is often the case with HRRR and RAP data. An example ``config.yaml`` excerpt using HRRR and RAP data appears below: +In some cases, it may be necessary to specify values for ``EXTRN_MDL_FILES_*CS`` variables. This is often the case with HRRR and RAP data. An example ``config.yaml`` excerpt using HRRR and RAP data appears below: .. code-block:: console diff --git a/docs/UsersGuide/source/RunSRW.rst b/docs/UsersGuide/source/RunSRW.rst index 1d74541837..0592af8470 100644 --- a/docs/UsersGuide/source/RunSRW.rst +++ b/docs/UsersGuide/source/RunSRW.rst @@ -537,8 +537,8 @@ To configure an experiment and python environment for a general Linux or Mac sys .. _PlotOutput: -Plot the Output ------------------ +Plotting Configuration (optional) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ An optional Python plotting task (PLOT_ALLVARS) can be activated in the workflow to generate plots for the :term:`FV3`-:term:`LAM` post-processed :term:`GRIB2` output over the :term:`CONUS`. It generates graphics plots for a number of variables, including: @@ -561,12 +561,12 @@ the same cycle starting date/time and forecast hours. Other parameters may diffe .. _Cartopy: Cartopy Shapefiles -^^^^^^^^^^^^^^^^^^^^^ +````````````````````` The Python plotting tasks require a path to the directory where the Cartopy Natural Earth shapefiles are located. The medium scale (1:50m) cultural and physical shapefiles are used to create coastlines and other geopolitical borders on the map. On `Level 1 `__ systems, this path is already set in the system's machine file using the variable ``FIXshp``. Users on other systems will need to download the shapefiles and update the path of ``$FIXshp`` in the machine file they are using (e.g., ``$SRW/ush/machine/macos.yaml`` for a generic MacOS system, where ``$SRW`` is the path to the ``ufs-srweather-app`` directory). The subset of shapefiles required for the plotting task can be obtained from the `SRW Data Bucket `__. The full set of medium-scale (1:50m) Cartopy shapefiles can be downloaded `here `__. Task Configuration -^^^^^^^^^^^^^^^^^^^^^ +````````````````````` Users will need to add or modify certain variables in ``config.yaml`` to run the plotting task(s). At a minimum, users must set ``RUN_TASK_PLOT_ALLVARS`` to true in the ``workflow_switches:`` section: @@ -593,16 +593,16 @@ When plotting output from a single experiment, no further adjustments are necess corresponds to the cycle date and hour in YYYYMMDDHH format (e.g., ``2019061518``). Plotting the Difference Between Two Experiments -``````````````````````````````````````````````````` +"""""""""""""""""""""""""""""""""""""""""""""""""" -When plotting the difference between two experiments, users must set the baseline experiment directory using the ``COMOUT_REF`` template variable. For example, in *community* mode, users can set: +When plotting the difference between two experiments (``expt1`` and ``expt2``), users must set the ``COMOUT_REF`` template variable in ``expt2``'s ``config.yaml`` file to point at forecast output from the ``expt1`` directory. For example, in *community* mode, users can set ``COMOUT_REF`` as follows in the ``expt2`` configuration file: .. code-block:: console task_plot_allvars: - COMOUT_REF: '${EXPT_BASEDIR}/${EXPT_SUBDIR}/${PDY}${cyc}/postprd' + COMOUT_REF: '${EXPT_BASEDIR}/expt1/${PDY}${cyc}/postprd' -In *community* mode, using default directory names and settings, ``$COMOUT_REF`` will resemble ``/path/to/expt_dirs/test_community/2019061518/postprd``. Additional details on the plotting variables are provided in :numref:`Section %s `. +This will ensure that ``expt2`` can produce a difference plot comparing ``expt1`` and ``expt2``. In *community* mode, using default directory names and settings, ``$COMOUT_REF`` will resemble ``/path/to/expt_dirs/test_community/2019061518/postprd``. Additional details on the plotting variables are provided in :numref:`Section %s `. The output files (in ``.png`` format) will be located in the ``postprd`` directory for the experiment. diff --git a/docs/UsersGuide/source/TemplateVars.rst b/docs/UsersGuide/source/TemplateVars.rst index 8950de80c8..2677888f5a 100644 --- a/docs/UsersGuide/source/TemplateVars.rst +++ b/docs/UsersGuide/source/TemplateVars.rst @@ -1,8 +1,8 @@ .. _TemplateVars: -=============================================================== -Using Template Variables in the Experiment Configuration Files -=============================================================== +====================== +Template Variables +====================== The SRW App's experiment configuration system supports the use of template variables in ``config_defaults.yaml`` and ``config.yaml``. A template variable --- or "template" --- is an experiment configuration variable that contains references to values of other variables. @@ -54,17 +54,22 @@ To generate difference plots, users must use the template variable ``COMOUT_REF` to indicate where the :term:`GRIB2` files from post-processing are located. In *community* mode (i.e., when ``RUN_ENVIR: "community"``), this directory will -take the form ``/path/to/exptdir/$PDY$cyc/postprd``, where ``$PDY`` refers to the -cycle date in YYYYMMDD format, and ``$cyc`` refers to the starting hour of the cycle. +take the form ``/path/to/expt_dirs/expt_name/$PDY$cyc/postprd``, where ``$PDY`` refers +to the cycle date in YYYYMMDD format, and ``$cyc`` refers to the starting hour of the cycle. (These variables are set in previous tasks based on the value of ``DATE_FIRST_CYCL``.) -Concretely, users can set ``COMOUT_REF`` as follows: +Given two experiments, ``expt1`` and ``expt2``, users can generate difference plots by +setting ``COMOUT_REF`` in the ``expt2`` configuration file (``config.yaml``) as follows: .. code-block:: console - COMOUT_REF: '${EXPT_BASEDIR}/${EXPT_SUBDIR}/${PDY}${cyc}/postprd' + COMOUT_REF: '${EXPT_BASEDIR}/expt1/${PDY}${cyc}/postprd' -In *nco* mode, this directory should be set to the location of the ``COMOUT`` directory -(``${COMOUT}`` in the example below) and end with ``${PDY}/${cyc}``. For example: +The ``expt2`` workflow already knows where to find its own post-processed output, so +``COMOUT_REF`` should point to post-processed output for the other experiment (``expt1``). + +In *nco* mode, this directory should be set to the location of the first experiment's +``COMOUT`` directory (``${COMOUT}`` in the example below) and end with ``${PDY}/${cyc}``. +For example: .. code-block:: console diff --git a/docs/UsersGuide/source/Tutorial.rst b/docs/UsersGuide/source/Tutorial.rst new file mode 100644 index 0000000000..169a5f2109 --- /dev/null +++ b/docs/UsersGuide/source/Tutorial.rst @@ -0,0 +1,589 @@ +.. _Tutorial: + +============= +Tutorials +============= + +This chapter walks users through experiment configuration options for various severe weather events. It assumes that users have already (1) :ref:`built the SRW App ` successfully and (2) run the out-of-the-box case contained in the ``config.community.yaml`` (and copied to ``config.yaml`` in :numref:`Step %s ` or :numref:`Step %s `) to completion. + +Users can run through the entire set of tutorials or jump to the one that interests them most. The five tutorials address different skills: + + #. :ref:`Severe Weather Over Indianapolis `: Change physics suites and compare graphics plots. + #. :ref:`Cold Air Damming `: Coming soon! + #. :ref:`Southern Plains Winter Weather Event `: Coming soon! + #. :ref:`Halloween Storm `: Coming soon! + #. :ref:`Hurricane Barry `: Coming soon! + +Each section provides a summary of the weather event and instructions for configuring an experiment. + +.. _fcst1: + +Sample Forecast #1: Severe Weather Over Indianapolis +======================================================= + +**Objective:** Modify physics options and compare forecast outputs for similar experiments using the graphics plotting task. + +Weather Summary +-------------------- + +A surface boundary associated with a vorticity maximum over the northern Great Plains moved into an unstable environment over Indianapolis, which led to the development of isolated severe thunderstorms before it congealed into a convective line. The moist air remained over the southern half of the area on the following day. The combination of moist air with daily surface heating resulted in isolated thunderstorms that produced small hail. + +**Weather Phenomena:** Numerous tornado and wind reports (6/15) and hail reports (6/16) + + * `Storm Prediction Center (SPC) Storm Report for 20190615 `__ + * `Storm Prediction Center (SPC) Storm Report for 20190616 `__ + +.. figure:: _static/IndySevereWeather18z.gif + :alt: Radar animation of severe weather over Indianapolis on June 15, 2019 starting at 18z. The animation shows areas of heavy rain and tornado reports moving from west to east over Indianapolis. + + *Severe Weather Over Indianapolis Starting at 18z* + +Data +------- + +On `Level 1 `__ systems, users can find data for the Indianapolis Severe Weather Forecast in the usual input model data locations (see :numref:`Section %s ` for a list). The data can also be downloaded from the `UFS SRW Application Data Bucket `__. + + * FV3GFS data for the first forecast (``control``) is located at: + + * https://noaa-ufs-srw-pds.s3.amazonaws.com/index.html#input_model_data/FV3GFS/grib2/2019061518/ + + * HRRR and RAP data for the second forecast (``test_expt``) is located at: + + * https://noaa-ufs-srw-pds.s3.amazonaws.com/index.html#input_model_data/HRRR/2019061518/ + * https://noaa-ufs-srw-pds.s3.amazonaws.com/index.html#input_model_data/RAP/2019061518/ + +Load the Regional Workflow +------------------------------- + +To load the regional workflow environment, source the lmod-setup file. Then load the workflow conda environment. From the ``ufs-srweather-app`` directory, run: + +.. code-block:: console + + source etc/lmod-setup.sh # OR: source etc/lmod-setup.csh when running in a csh/tcsh shell + module use /path/to/ufs-srweather-app/modulefiles + module load wflow_ + +where ```` is a valid, lowercased machine name (see ``MACHINE`` in :numref:`Section %s ` for valid values). + +After loading the workflow, users should follow the instructions printed to the console. Usually, the instructions will tell the user to run ``conda activate regional_workflow``. For example, a user on Hera with permissions on the ``nems`` project may issue the following commands to load the regional workflow (replacing ``User.Name`` with their actual username): +etai +.. code-block:: console + + source /scratch1/NCEPDEV/nems/User.Name/ufs-srweather-app/etc/lmod-setup.sh hera + module use /scratch1/NCEPDEV/nems/User.Name/ufs-srweather-app/modulefiles + module load wflow_hera + conda activate regional_workflow + +Configuration +------------------------- + +Navigate to the ``ufs-srweather-app/ush`` directory. The default (or "control") configuration for this experiment is based on the ``config.community.yaml`` file in that directory. Users can copy this file into ``config.yaml`` if they have not already done so: + +.. code-block:: console + + cd /path/to/ufs-srweather-app/ush + cp config.community.yaml config.yaml + +Users can save the location of the ``ush`` directory in an environment variable (``$USH``). This makes it easier to navigate between directories later. For example: + +.. code-block:: console + + export USH=/path/to/ufs-srweather-app/ush + +Users should substitute ``/path/to/ufs-srweather-app/ush`` with the actual path on their system. As long as a user remains logged into their system, they can run ``cd $USH``, and it will take them to the ``ush`` directory. The variable will need to be reset for each login session. + +Experiment 1: Control +^^^^^^^^^^^^^^^^^^^^^^^^ + +Edit the configuration file (``config.yaml``) to include the variables and values in the sample configuration excerpts below. + +.. Hint:: + + To open the configuration file in the command line, users may run the command: + + .. code-block:: console + + vi config.yaml + + To modify the file, hit the ``i`` key and then make any changes required. To close and save, hit the ``esc`` key and type ``:wq`` to write the changes to the file and exit/quit the file. Users may opt to use their preferred code editor instead. + +Start in the ``user:`` section and change the ``MACHINE`` and ``ACCOUNT`` variables. For example, when running on a personal MacOS device, users might set: + +.. code-block:: console + + user: + RUN_ENVIR: community + MACHINE: macos + ACCOUNT: none + +For a detailed description of these variables, see :numref:`Section %s `. + +Users do not need to change the ``platform:`` section of the configuration file for this tutorial. The default parameters in the ``platform:`` section pertain to METplus verification, which is not addressed here. For more information on verification, see :numref:`Chapter %s `. + +In the ``workflow:`` section of ``config.yaml``, update ``EXPT_SUBDIR`` and ``PREDEF_GRID_NAME``. + +.. code-block:: console + + workflow: + USE_CRON_TO_RELAUNCH: false + EXPT_SUBDIR: control + CCPP_PHYS_SUITE: FV3_GFS_v16 + PREDEF_GRID_NAME: SUBCONUS_Ind_3km + DATE_FIRST_CYCL: '2019061518' + DATE_LAST_CYCL: '2019061518' + FCST_LEN_HRS: 12 + PREEXISTING_DIR_METHOD: rename + VERBOSE: true + COMPILER: intel + +.. _CronNote: + +.. note:: + + Users may also want to set ``USE_CRON_TO_RELAUNCH: true`` and add ``CRON_RELAUNCH_INTVL_MNTS: 3``. This will automate submission of workflow tasks when running the experiment. However, not all systems have :term:`cron`. + +``EXPT_SUBDIR:`` This variable can be changed to any name the user wants. This tutorial uses ``control`` to establish a baseline, or "control", forecast. Users can choose any name they want, from "gfsv16_physics_fcst" to "forecast1" to "a;skdfj". However, the best names will indicate useful information about the experiment. For example, this tutorial helps users to compare the output from two different forecasts: one that uses the FV3_GFS_v16 physics suite and one that uses the FV3_RRFS_v1beta physics suite. Therefore, "gfsv16_physics_fcst" could be a good alternative directory name. + +``PREDEF_GRID_NAME:`` This experiment uses the SUBCONUS_Ind_3km grid, rather than the default RRFS_CONUS_25km grid. The SUBCONUS_Ind_3km grid is a high-resolution grid (with grid cell size of approximately 3km) that covers a small area of the U.S. centered over Indianapolis, IN. For more information on this grid, see :numref:`Section %s `. + +For a detailed description of other ``workflow:`` variables, see :numref:`Section %s `. + +In the ``workflow_switches:`` section, turn on the plotting task by setting ``RUN_TASK_PLOT_ALLVARS`` to true. All other variables should remain as they are. + +.. code-block:: console + + workflow_switches: + RUN_TASK_MAKE_GRID: true + RUN_TASK_MAKE_OROG: true + RUN_TASK_MAKE_SFC_CLIMO: true + RUN_TASK_GET_OBS_CCPA: false + RUN_TASK_GET_OBS_MRMS: false + RUN_TASK_GET_OBS_NDAS: false + RUN_TASK_VX_GRIDSTAT: false + RUN_TASK_VX_POINTSTAT: false + RUN_TASK_VX_ENSGRID: false + RUN_TASK_VX_ENSPOINT: false + RUN_TASK_PLOT_ALLVARS: true + +For a detailed description of the ``workflow-switches:`` variables, see :numref:`Section %s `. + +In the ``task_get_extrn_ics:`` section, add ``USE_USER_STAGED_EXTRN_FILES`` and ``EXTRN_MDL_SOURCE_BASEDIR_ICS``. Users will need to adjust the file path to reflect the location of data on their system (see :numref:`Section %s ` for locations on `Level 1 `__ systems). + +.. code-block:: console + + task_get_extrn_ics: + EXTRN_MDL_NAME_ICS: FV3GFS + FV3GFS_FILE_FMT_ICS: grib2 + USE_USER_STAGED_EXTRN_FILES: true + EXTRN_MDL_SOURCE_BASEDIR_ICS: /path/to/UFS_SRW_App/develop/input_model_data/FV3GFS/grib2/${yyyymmddhh} + +For a detailed description of the ``task_get_extrn_ics:`` variables, see :numref:`Section %s `. + +Similarly, in the ``task_get_extrn_lbcs:`` section, add ``USE_USER_STAGED_EXTRN_FILES`` and ``EXTRN_MDL_SOURCE_BASEDIR_LBCS``. Users will need to adjust the file path to reflect the location of data on their system (see :numref:`Section %s ` for locations on Level 1 systems). + +.. code-block:: console + + task_get_extrn_lbcs: + EXTRN_MDL_NAME_LBCS: FV3GFS + LBC_SPEC_INTVL_HRS: 6 + FV3GFS_FILE_FMT_LBCS: grib2 + USE_USER_STAGED_EXTRN_FILES: true + EXTRN_MDL_SOURCE_BASEDIR_LBCS: /path/to/UFS_SRW_App/develop/input_model_data/FV3GFS/grib2/${yyyymmddhh} + +For a detailed description of the ``task_get_extrn_lbcs:`` variables, see :numref:`Section %s `. + +Users do not need to modify the ``task_run_fcst:`` section for this tutorial. + +Lastly, in the ``task_plot_allvars:`` section, add ``PLOT_FCST_INC: 6`` and ``PLOT_DOMAINS: ["regional"]``. Users may also want to add ``PLOT_FCST_START: 0`` and ``PLOT_FCST_END: 12`` explicitly, but these can be omitted since the default values are the same as the forecast start and end time respectively. + +.. code-block:: console + + task_plot_allvars: + COMOUT_REF: "" + PLOT_FCST_INC: 6 + PLOT_DOMAINS: ["regional"] + +``PLOT_FCST_INC:`` This variable indicates the forecast hour increment for the plotting task. By setting the value to ``6``, the task will generate a ``.png`` file for every 6th forecast hour starting from 18z on June 15, 2019 (the 0th forecast hour) through the 12th forecast hour (June 16, 2019 at 06z). + +``PLOT_DOMAINS:`` The plotting scripts are designed to generate plots over the entire CONUS by default, but by setting this variable to ["regional"], the experiment will generate plots for the smaller SUBCONUS_Ind_3km regional domain instead. + +After configuring the forecast, users can generate the forecast by running: + +.. code-block:: console + + ./generate_FV3LAM_wflow.py + +To see experiment progress, users should navigate to their experiment directory. Then, use the ``rocotorun`` command to launch new workflow tasks and ``rocotostat`` to check on experiment progress. + +.. code-block:: console + + cd /path/to/expt_dirs/control + rocotorun -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 + rocotostat -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 + +Users will need to rerun the ``rocotorun`` and ``rocotostat`` commands above regularly and repeatedly to continue submitting workflow tasks and receiving progress updates. + +.. note:: + + When using cron to automate the workflow submission (as described :ref:`above `), users can omit the ``rocotorun`` command and simply use ``rocotostat`` to check on progress periodically. + +Users can save the location of the ``control`` directory in an environment variable (``$CONTROL``). This makes it easier to navigate between directories later. For example: + +.. code-block:: console + + export CONTROL=/path/to/expt_dirs/control + +Users should substitute ``/path/to/expt_dirs/control`` with the actual path on their system. As long as a user remains logged into their system, they can run ``cd $CONTROL``, and it will take them to the ``control`` experiment directory. The variable will need to be reset for each login session. + +Experiment 2: Test +^^^^^^^^^^^^^^^^^^^^^^ + +Once the control case is running, users can return to the ``config.yaml`` file (in ``$USH``) and adjust the parameters for a new forecast. Most of the variables will remain the same. However, users will need to adjust ``EXPT_SUBDIR`` and ``CCPP_PHYS_SUITE`` in the ``workflow:`` section as follows: + +.. code-block:: console + + workflow: + EXPT_SUBDIR: test_expt + CCPP_PHYS_SUITE: FV3_RRFS_v1beta + +``EXPT_SUBDIR:`` This name must be different than the ``EXPT_SUBDIR`` name used in the previous forecast experiment. Otherwise, the first forecast experiment will be overwritten. ``test_expt`` is used here, but the user may select a different name if desired. + +``CCPP_PHYS_SUITE:`` The FV3_RRFS_v1beta physics suite was specifically created for convection-allowing scales and is the precursor to the operational physics suite that will be used in the Rapid Refresh Forecast System (:term:`RRFS`). + +.. hint:: + + Later, users may want to conduct additional experiments using the FV3_HRRR and FV3_WoFS_v0 physics suites. Like FV3_RRFS_v1beta, these physics suites were designed for use with high-resolution grids for storm-scale predictions. + +Next, users will need to modify the data parameters in ``task_get_extrn_ics:`` and ``task_get_extrn_lbcs:`` to use HRRR and RAP data rather than FV3GFS data. Users will need to change the following lines in each section: + +.. code-block:: console + + task_get_extrn_ics: + EXTRN_MDL_NAME_ICS: HRRR + EXTRN_MDL_SOURCE_BASEDIR_ICS: /path/to/UFS_SRW_App/develop/input_model_data/HRRR/${yyyymmddhh} + task_get_extrn_lbcs: + EXTRN_MDL_NAME_LBCS: RAP + EXTRN_MDL_SOURCE_BASEDIR_LBCS: /path/to/UFS_SRW_App/develop/input_model_data/RAP/${yyyymmddhh} + EXTRN_MDL_LBCS_OFFSET_HRS: '-0' + +HRRR and RAP data are better than FV3GFS data for use with the FV3_RRFS_v1beta physics scheme because these datasets use the same physics :term:`parameterizations` that are in the FV3_RRFS_v1beta suite. They focus on small-scale weather phenomena involved in storm development, so forecasts tend to be more accurate when HRRR/RAP data are paired with FV3_RRFS_v1beta and a high-resolution (e.g., 3-km) grid. Using HRRR/RAP data with FV3_RRFS_v1beta also limits the "spin-up adjustment" that takes place when initializing with model data coming from different physics. + +``EXTRN_MDL_LBCS_OFFSET_HRS:`` This variable allows users to use lateral boundary conditions (LBCs) from a previous forecast run that was started earlier than the start time of the current forecast configured in this experiment. This variable is set to 0 by default except when using RAP data; with RAP data, the default value is 3, so the forecast will look for LBCs from a forecast started 3 hours earlier (i.e., at 2019061515 --- 15z --- instead of 2019061518). To avoid this, users must set ``EXTRN_MDL_LBCS_OFFSET_HRS`` explicitly. + +Add a section to ``config.yaml`` to increase the maximum wall time (``WTIME_RUN_POST``) for the postprocessing tasks. The wall time is the maximum length of time a task is allowed to run. On some systems, the default of 15 minutes may be enough, but on others (e.g., NOAA Cloud), the post-processing time exceeds 15 minutes, so the tasks fail. + +.. code-block:: console + + task_run_post: + WTIME_RUN_POST: 00:20:00 + +Lastly, users must set the ``COMOUT_REF`` variable in the ``task_plot_allvars:`` section to create difference plots that compare output from the two experiments. ``COMOUT_REF`` is a template variable, so it references other workflow variables within it (see :numref:`Section %s ` for details on template variables). ``COMOUT_REF`` should provide the path to the ``control`` experiment forecast output using single quotes as shown below: + +.. code-block:: console + + task_plot_allvars: + COMOUT_REF: '${EXPT_BASEDIR}/control/${PDY}${cyc}/postprd' + +Here, ``$EXPT_BASEDIR`` is the path to the main experiment directory (named ``expt_dirs`` by default). ``$PDY`` refers to the cycle date in YYYYMMDD format, and ``$cyc`` refers to the starting hour of the cycle. ``postprd`` contains the post-processed data from the experiment. Therefore, ``COMOUT_REF`` will refer to ``control/2019061518/postprd`` and compare those plots against the ones in ``test_expt/2019061518/postprd``. + +After configuring the forecast, users can generate the second forecast by running: + +.. code-block:: console + + ./generate_FV3LAM_wflow.py + +To see experiment progress, users should navigate to their experiment directory. As in the first forecast, the following commands allow users to launch new workflow tasks and check on experiment progress. + +.. code-block:: console + + cd /path/to/expt_dirs/test_expt + rocotorun -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 + rocotostat -w FV3LAM_wflow.xml -d FV3LAM_wflow.db -v 10 + +.. note:: + + When using cron to automate the workflow submission (as described :ref:`above `), users can omit the ``rocotorun`` command and simply use ``rocotostat`` to check on progress periodically. + +.. note:: + + If users have not automated their workflow using cron, they will need to ensure that they continue issuing ``rocotorun`` commands to launch all of the tasks in each experiment. While switching between experiment directories to run ``rocotorun`` and ``rocotostat`` commands in both is possible, it may be easier to finish the ``control`` experiment's tasks before starting on ``test_expt``. + + +Compare and Analyze Results +----------------------------- + +Navigate to ``test_expt/2019061518/postprd``. This directory contains the post-processed data generated by the :term:`UPP` from the forecast. After the ``plot_allvars`` task completes, this directory will contain ``.png`` images for several forecast variables including 2-m temperature, 2-m dew point temperature, 10-m winds, accumulated precipitation, composite reflectivity, and surface-based CAPE/CIN. Plots with a ``_diff`` label in the file name are plots that compare the ``control`` forecast and the ``test_expt`` forecast. + +Copy ``.png`` Files onto Local System +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Users who are working on the cloud or on an HPC cluster may want to copy the ``.png`` files onto their local system to view in their preferred image viewer. Detailed instructions can are available in the :ref:`Introduction to SSH & Data Transfer `. + +In summary, users can run the ``scp`` command in a new terminal/command prompt window to securely copy files from a remote system to their local system if an SSH tunnel is already established between the local system and the remote system. Users can adjust one of the following commands for their system: + +.. code-block:: console + + scp username@your-IP-address:/path/to/source_file_or_directory /path/to/destination_file_or_directory + # OR + scp -P 12345 username@localhost:/path/to/source_file_or_directory path/to/destination_file_or_directory + +Users would need to modify ``username``, ``your-IP-address``, ``-P 12345``, and the file paths to reflect their systems' information. See the :ref:`Introduction to SSH & Data Transfer ` for example commands. + +.. _ComparePlots: + +Compare Images +^^^^^^^^^^^^^^^^^^ + +The plots generated by the experiment cover a variety of variables. After downloading the ``.png`` plots, users can open and view the plots on their local system in their preferred image viewer. :numref:`Table %s ` lists the available plots (``hhh`` corresponds to the three-digit forecast hour): + +.. _DiffPlots: + +.. table:: Sample Indianapolis Forecast Plots + + +-----------------------------------------+-----------------------------------+ + | Field | File Name | + +=========================================+===================================+ + | 2-meter dew point temperature | 2mdew_diff_regional_fhhh.png | + +-----------------------------------------+-----------------------------------+ + | 2-meter temperature | 2mt_diff_regional_fhhh.png | + +-----------------------------------------+-----------------------------------+ + | 10-meter winds | 10mwind_diff_regional_fhhh.png | + +-----------------------------------------+-----------------------------------+ + | 250-hPa winds | 250wind_diff_regional_fhhh.png | + +-----------------------------------------+-----------------------------------+ + | Accumulated precipitation | qpf_diff_regional_fhhh.png | + +-----------------------------------------+-----------------------------------+ + | Composite reflectivity | refc_diff_regional_fhhh.png | + +-----------------------------------------+-----------------------------------+ + | Surface-based CAPE/CIN | sfcape_diff_regional_fhhh.png | + +-----------------------------------------+-----------------------------------+ + | Sea level pressure | slp_diff_regional_fhhh.png | + +-----------------------------------------+-----------------------------------+ + | Max/Min 2 - 5 km updraft helicity | uh25_diff_regional_fhhh.png | + +-----------------------------------------+-----------------------------------+ + +Each difference plotting ``.png`` file contains three subplots. The plot for the second experiment (``test_expt``) appears in the top left corner, and the plot for the first experiment (``control``) appears in the top right corner. The difference plot that compares both experiments appear at the bottom. Areas of white signify no difference between the plots. Therefore, if the forecast output from both experiments is exactly the same, the difference plot will show a white square (see :ref:`Sea Level Pressure ` for an example). If the forecast output from both experiments is extremely different, the plot will show lots of color. + +In general, it is expected that the results for ``test_expt`` (using FV3_RRFS_v1beta physics and HRRR/RAP data) will be more accurate than the results for ``control`` (using FV3_GFS_v16 physics and FV3GFS data) because the physics in ``test_expt`` is designed for high-resolution, storm-scale prediction over a short period of time. The ``control`` experiment physics is better for predicting the evolution of larger scale weather phenomena, like jet stream movement and cyclone development, since the cumulus physics in the FV3_GFS_v16 suite is not configured to run at 3-km resolution. + +Analysis +^^^^^^^^^^^ + +.. _fcst1_slp: + +Sea Level Pressure +````````````````````` +In the Sea Level Pressure (SLP) plots, the ``control`` and ``test_expt`` plots are nearly identical at forecast hour f000, so the difference plot is entirely white. + +.. figure:: _static/plots/slp_diff_regional_f000.png + :align: center + + *Difference Plot for Sea Level Pressure at f000* + +As the forecast continues, the results begin to diverge, as evidenced by the spattering of light blue dispersed across the f006 SLP difference plot. + +.. figure:: _static/plots/slp_diff_regional_f006.png + :align: center + + *Difference Plot for Sea Level Pressure at f006* + +The predictions diverge further by f012, where a solid section of light blue in the top left corner of the difference plot indicates that to the northwest of Indianapolis, the SLP predictions for the ``control`` forecast were slightly lower than the predictions for the ``test_expt`` forecast. + +.. figure:: _static/plots/slp_diff_regional_f012.png + :align: center + + *Difference Plot for Sea Level Pressure at f012* + +.. _fcst1_refc: + +Composite Reflectivity +`````````````````````````` + +Reflectivity images visually represent the weather based on the energy (measured in decibels [dBZ]) reflected back from radar. Composite reflectivity generates an image based on reflectivity scans at multiple elevation angles, or "tilts", of the antenna. See https://www.weather.gov/jetstream/refl for a more detailed explanation of composite reflectivity. + +At f000, the ``test_expt`` plot (top left) is showing more severe weather than the ``control`` plot (top right). The ``test_expt`` plot shows a vast swathe of the Indianapolis region covered in yellow with spots of orange, corresponding to composite reflectivity values of 35+ dBZ. The ``control`` plot radar image covers a smaller area of the grid, and with the exception of a few yellow spots, composite reflectivity values are <35 dBZ. The difference plot (bottom) shows areas where the ``test_expt`` plot (red) and the ``control`` plot (blue) have reflectivity values greater than 20 dBZ. The ``test_expt`` plot has significantly more areas with high composite reflectivity values. + +.. figure:: _static/plots/refc_diff_regional_f000.png + :align: center + + *Composite Reflectivity at f000* + +As the forecast progresses, the radar images resemble each other more (see :numref:`Figure %s `). Both the ``test_expt`` and ``control`` plots show the storm gaining energy (with more orange and red areas), rotating counterclockwise, and moving east. Both forecasts do a good job of picking up on the convection. However, the ``test_expt`` forecast still indicates a higher-energy storm with more areas of dark red. It appears that the ``test_expt`` case was able to resolve more discrete storms over northwest Indiana and in the squall line. The ``control`` plot has less definition and depicts widespread storms concentrated together over the center of the state. + +.. _refc006: + +.. figure:: _static/plots/refc_diff_regional_f006.png + :align: center + + *Composite reflectivity at f006 shows storm gathering strength* + +At forecast hour 12, the plots for each forecast show a similar evolution of the storm with both resolving a squall line. The ``test_expt`` plot shows a more intense squall line with discrete cells (areas of high composite reflectivity in dark red), which could lead to severe weather. The ``control`` plot shows an overall decrease in composite reflectivity values compared to f006. It also orients the squall line more northward with less intensity, possibly due to convection from the previous forecast runs cooling the atmosphere. In short, ``test_expt`` suggests that the storm will still be going strong at 06z on June 15, 2019, whereas the ``control`` suggests that the storm will begin to let up. + +.. figure:: _static/plots/refc_diff_regional_f012.png + :align: center + + *Composite Reflectivity at f012* + +.. _fcst1_sfcape: + +Surface-Based CAPE/CIN +`````````````````````````` + +Background +"""""""""""" + +The National Weather Service (:term:`NWS`) defines Surface-Based Convective Available Potential Energy (CAPE) as "the amount of fuel available to a developing thunderstorm." According to NWS, CAPE "describes the instabilily of the atmosphere and provides an approximation of updraft strength within a thunderstorm. A higher value of CAPE means the atmosphere is more unstable and would therefore produce a stronger updraft" (see `NWS, What is CAPE? `__ for further explanation). + +According to the NWS `Storm Prediction Center `__, Convective Inhibition (CIN) "represents the 'negative' area on a sounding that must be overcome for storm initiation." In effect, it measures negative buoyancy (-B) --- the opposite of CAPE, which measures positive buoyancy (B or B+) of an air parcel. + +.. + More CAPE/CIN info: https://www.e-education.psu.edu/files/meteo361/image/Section4/cape_primer0301.html + +Interpreting the Plots +"""""""""""""""""""""""" + +CAPE measures are represented on the plots using color. They range in value from 100-5000 Joules per kilogram (J/kg). Lower values are represented by cool colors and higher values are represented by warm colors. In general, values of approximately 1000+ J/kg can lead to severe thunderstorms, although this is also dependent on season and location. + +CIN measures are displayed on the plots using hatch marks: + + * ``*`` means CIN <= -500 J/kg + * ``+`` means -500 < CIN <= -250 J/kg + * ``/`` means -250 < CIN <= -100 J/kg + * ``.`` means -100 < CIN <= -25 J/kg + +In general, the higher the CIN values are (i.e., the closer they are to zero), the lower the convective inhibition and the greater the likelihood that a storm will develop. Low CIN values (corresponding to high convective inhibition) make it unlikely that a storm will develop even in the presence of high CAPE. + +At the 0th forecast hour, the ``test_expt`` plot (below, left) shows lower values of CAPE and higher values of CIN than in the ``control`` plot (below, right). This means that ``test_expt`` is projecting lower potential energy available for a storm but also lower inhibition, which means that less energy would be required for a storm to develop. The difference between the two plots is particularly evident in the southwest corner of the difference plot, which shows a 1000+ J/kg difference between the two plots. + +.. figure:: _static/plots/sfcape_diff_regional_f000.png + :width: 1200 + :align: center + + *CAPE/CIN Difference Plot at f000* + +At the 6th forecast hour, both ``test_expt`` and ``control`` plots are forecasting higher CAPE values overall. Both plots also predict higher CAPE values to the southwest of Indianapolis than to the northeast. This makes sense because the storm was passing from west to east. However, the difference plot shows that the ``control`` forecast is predicting higher CAPE values primarily to the southwest of Indianapolis, whereas ``test_expt`` is projecting a rise in CAPE values throughout the region. The blue region of the difference plot indicates where ``test_expt`` predictions are higher than the ``control`` predictions; the red/orange region shows places where ``control`` predicts significantly higher CAPE values than ``test_expt`` does. + +.. figure:: _static/plots/sfcape_diff_regional_f006.png + :width: 1200 + :align: center + + *CAPE/CIN Difference Plot at f006* + +At the 12th forecast hour, the ``control`` plot indicates that CAPE may be decreasing overall. ``test_expt``, however, shows that areas of high CAPE remain and continue to grow, particularly to the east. The blue areas of the difference plot indicate that ``test_expt`` is predicting higher CAPE than ``control`` everywhere but in the center of the plot. + +.. figure:: _static/plots/sfcape_diff_regional_f012.png + :width: 1200 + :align: center + + *CAPE/CIN Difference Plot at f012* + +Try It! +---------- + +Option 1: Adjust frequency of forecast plots. +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For a simple extension of this tutorial, users can adjust ``PLOT_FCST_INC`` to output plots more frequently. For example, users can set ``PLOT_FCST_INC: 1`` to produce plots for every hour of the forecast. This would allow users to conduct a more fine-grained visual comparison of how each forecast evolved. + +Option 2: Compare output from additional physics suites. +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Users are encouraged to conduct additional experiments using the FV3_HRRR and FV3_WoFS_v0 physics suites. Like FV3_RRFS_v1beta, these physics suites were designed for use with high-resolution grids for storm-scale predictions. Compare them to each other or to the control! + +Users may find the difference plots for :term:`updraft helicity` particularly informative. The FV3_GFS_v16 physics suite does not contain updraft helicity output in its ``diag_table`` files, so the difference plot generated in this tutorial is empty. However, high updraft helicity values increase the probability that a storm will become a supercell thunderstorm or a tornado. Comparing the results from two physics suites that measure this parameter can therefore prove insightful. + +.. _fcst2: + +Sample Forecast #2: Cold Air Damming +======================================== + +Weather Summary +----------------- + +Cold air damming occurs when cold dense air is topographically trapped along the leeward (downwind) side of a mountain. Starting on February 3, 2020, weather conditions leading to cold air damming began to develop east of the Appalachian mountains. By February 6-7, 2020, this cold air damming caused high winds, flash flood advisories, and wintery conditions. + +**Weather Phenomena:** Cold air damming + + * `Storm Prediction Center (SPC) Storm Report for 20200205 `__ + * `Storm Prediction Center (SPC) Storm Report for 20200206 `__ + * `Storm Prediction Center (SPC) Storm Report for 20200207 `__ + +.. figure:: _static/ColdAirDamming.gif + :alt: Radar animation of precipitation resulting from cold air damming in the southern Appalachian mountains. + + *Precipitation Resulting from Cold Air Damming East of the Appalachian Mountains* + +Tutorial Content +------------------- + +Coming Soon! + +.. _fcst3: + +Sample Forecast #3: Southern Plains Winter Weather Event +=========================================================== + +Weather Summary +-------------------- + +A polar vortex brought arctic air to much of the U.S. and Mexico. A series of cold fronts and vorticity disturbances helped keep this cold air in place for an extended period of time resulting in record-breaking cold temperatures for many southern states and Mexico. This particular case captures two winter weather disturbances between February 14, 2021 at 06z and February 17, 2021 at 06z that brought several inches of snow to Oklahoma City. A lull on February 16, 2021 resulted in record daily low temperatures. + +**Weather Phenomena:** Snow and record-breaking cold temperatures + +.. figure:: _static/SouthernPlainsWinterWeather.gif + :alt: Radar animation of the Southern Plains Winter Weather Event centered over Oklahoma City. Animation starts on February 14, 2021 at 6h00 UTC and ends on February 17, 2021 at 6h00 UTC. + + *Southern Plains Winter Weather Event Over Oklahoma City* + +Tutorial Content +------------------- + +Coming Soon! + +.. _fcst4: + +Sample Forecast #4: Halloween Storm +======================================= + +Weather Summary +-------------------- + +A line of severe storms brought strong winds, flash flooding, and tornadoes to the eastern half of the US. + +**Weather Phenomena:** Flooding and high winds + * `Storm Prediction Center (SPC) Storm Report for 20191031 `__ + +.. figure:: _static/HalloweenStorm.gif + :alt: Radar animation of the Halloween Storm that swept across the Eastern United States in 2019. + + *Halloween Storm 2019* + +Tutorial Content +------------------- + +Coming Soon! + +.. _fcst5: + +Sample Forecast #5: Hurricane Barry +======================================= + +Weather Summary +-------------------- + +Hurricane Barry made landfall in Louisiana on July 11, 2019 as a Category 1 hurricane. It produced widespread flooding in the region and had a peak wind speed of 72 mph and a minimum pressure of 992 hPa. + +**Weather Phenomena:** Flooding, wind, and tornado reports + + * `Storm Prediction Center (SPC) Storm Report for 20190713 `__ + * `Storm Prediction Center (SPC) Storm Report for 20190714 `__ + +.. figure:: _static/HurricaneBarry_Making_Landfall.gif + :alt: Radar animation of Hurricane Barry making landfall. + + *Hurricane Barry Making Landfall* + +Tutorial Content +------------------- + +Coming Soon! diff --git a/docs/UsersGuide/source/VXCases.rst b/docs/UsersGuide/source/VXCases.rst index 09903fdaa4..2198ce10c4 100644 --- a/docs/UsersGuide/source/VXCases.rst +++ b/docs/UsersGuide/source/VXCases.rst @@ -201,7 +201,7 @@ Compare Once the experiment has completed (i.e., all tasks have "SUCCEEDED" and the end of the ``log.launch_FV3LAM_wflow`` file lists "Workflow status: SUCCESS"), users can compare their forecast results against the forecast results provided in the ``Indy-Severe-Weather`` directory downloaded in :numref:`Section %s `. This directory contains the forecast output and plots from NOAA developers under the ``postprd`` directory and METplus verification files under the ``metprd`` directory. -Qualitative Comparision of the Plots +Qualitative Comparison of the Plots ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Comparing the plots is relatively straightforward since they are in ``.png`` format, and most computers can render them in their default image viewer. :numref:`Table %s ` lists plots that are available every 6 hours of the forecast (where ``hhh`` is replaced by the three-digit forecast hour): diff --git a/docs/UsersGuide/source/index.rst b/docs/UsersGuide/source/index.rst index 9fc9c9acbb..651881337e 100644 --- a/docs/UsersGuide/source/index.rst +++ b/docs/UsersGuide/source/index.rst @@ -16,6 +16,7 @@ UFS Short-Range Weather App Users Guide ContainerQuickstart BuildSRW RunSRW + Tutorial Components InputOutputFiles LAMGrids diff --git a/docs/UsersGuide/source/tables/SSHIntro.rst b/docs/UsersGuide/source/tables/SSHIntro.rst new file mode 100644 index 0000000000..8089d2b8a9 --- /dev/null +++ b/docs/UsersGuide/source/tables/SSHIntro.rst @@ -0,0 +1,172 @@ +:orphan: + +.. _SSHIntro: + +====================================== +Introduction to SSH & Data Transfer +====================================== + +.. attention:: + + Note that all port numbers, IP addresses, and SSH keys included in this chapter are placeholders and do not refer to known systems. They are used purely for illustrative purposes, and users should modify the commands to correspond to their actual systems. + +A Secure SHell (SSH) tunnel creates an encrypted connection between two computer systems. This secure connection allows users to access and use a remote system via the command line on their local machine. SSH connections can also be used to transfer data securely between two systems. Many HPC platforms, including NOAA `Level 1 systems `__, are accessed via SSH from the user's own computer via SSH. + +.. attention:: + + Note that the instructions on this page assume that users are working on a UNIX-like system (i.e., Linux or MacOS). They may not work as-is on Windows systems, but users can adapt them for Windows or use a tool such as Cygwin, which enables the use of UNIX-like commands on Windows. Users may also consider installing a virtual machine such as VirtualBox. + +.. _CreateSSH: + +Creating an SSH Tunnel +============================ + +Create an SSH Key +-------------------- + +To generate an SSH key, open a terminal window and run: + +.. code-block:: console + + ssh-keygen -t rsa + +Hit enter three times to accept defaults, or if customization is desired: + + * Enter the file in which to save the key (for example: ``~/.ssh/id_rsa``) + * Enter passphrase (empty for no passphrase) + * Enter same passphrase again + +To see the SSH public key contents, run: + +.. code-block:: console + + cat id_rsa.pub + +SSH into a Remote Machine +---------------------------- + +This process differs somewhat from system to system. However, this section provides general guidance. + +Create/Edit an SSH Configuration File (``~/.ssh/config``) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If it exists, the SSH ``config`` file is located in the hidden ``.ssh`` directory. If it does not exist, opening it will create the file. In a terminal window, run: + +.. code-block:: console + + vi ~/.ssh/config + +Press ``i`` to edit the file, and add an entry in the following format: + +.. code-block:: console + + Host + Hostname + User + IdentityFile ~/.ssh/ + +When finished, hit the ``esc`` key and type ``:wq`` to write the data to the file and quit the file editor. + +.. note:: + + The ``IdentityFile`` line is not required unless the user has multiple SSH keys. However, there is no harm in adding it. + +Concretely, a user logging into an AWS cluster might enter something similar to the following. + +.. code-block:: console + + Host aws + Hostname 50.60.700.80 + User Jane.Doe + IdentityFile ~/.ssh/id_rsa + +Users attempting to authenticate via SSH on GitHub might create the following code block instead: + +.. code-block:: console + + Host github + Hostname github.com + User git + IdentityFile ~/.ssh/id_ed25519 + +SSH into the Remote System +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To SSH into the remote system, simply run: + +.. code-block:: console + + ssh + +where ```` is the "name_of_your_choice" that was added to the ``config`` file. For example, a user logging into the AWS cluster above would type: + +.. code-block:: console + + ssh aws + +This will create an SSH tunnel between the user's local system and the AWS cluster. The user will be able to work on the AWS cluster by running commands in the terminal window. + +In some cases, the user may be asked if they want to connect: + +.. code-block:: console + + The authenticity of host '50.60.700.80 (50.60.700.80)' can't be established. + ECDSA key fingerprint is SHA256:a0ABbC4cdeDEfFghi+j3kGHlO5mnIJKLMop7NOqPrQR. + Are you sure you want to continue connecting (yes/no/[fingerprint])? + +Enter ``yes`` to continue connecting. The user is responsible for verifying that they are connecting to the correct system. + +.. _SSHDataTransfer: + +Data Transfer via SSH +============================ + +Introduction +--------------- + +Users who are working on a remote cloud or HPC system may want to copy files (e.g., graphics plots) to or from their local system. Users can run the ``scp`` command in a new terminal/command prompt window to securely copy these files from their remote system to their local system or vice versa. The structure of the command is: + +.. code-block:: console + + scp [OPTION] [user@]SRC_HOST:]file1 [user@]DEST_HOST:]file2 + +Here, ``SRC_HOST`` refers to the system where the files are currently located. ``DEST_HOST`` refers to the system that the files will be moved to. ``file1`` is the path to the file or directory to copy, and ``file2`` is the location that the file or directory should be copied to on the ``DEST_HOST`` system. + +.. _SSHDownload: + +Download the Data from a Remote System to a Local System +----------------------------------------------------------- + +.. note:: + + Users should transfer data to or from non-`Level 1 `__ platforms using the recommended approach for that platform. This section outlines some basic guidance, but users may need to supplement with research of their own. On Level 1 systems, users may find it helpful to refer to the `RDHPCS CommonDocs Wiki `__. + +To download data using ``scp``, users can typically adjust one of the following commands for use on their system: + +.. code-block:: console + + scp username@your-IP-address:/path/to/file_or_directory_1 /path/to/file_or_directory_2 + # OR + scp -P 12345 username@localhost:/path/to/file_or_directory_1 path/to/file_or_directory_2 + +To copy an entire directory, use ``scp -r`` instead of ``scp``. + +Users who know the IP address of their remote system can use the first command. For example: + +.. code-block:: console + + scp Jane.Doe@10.20.300.40:/contrib/Jane.Doe/expt_dirs/test_community/2019061518/postprd/*.png /Users/janedoe/plots + +This command will copy all files ending in ``.png`` from the remote ``test_community/2019061518/postprd/`` experiment subdirectory into Jane Doe's local ``plots`` directory. + +Users who know their ``localhost`` port number should use the second command and, if requested, enter the password to the remote system. For example: + +.. code-block:: console + + scp -P 3355 Jane.Doe@localhost:/lustre/Jane.Doe/expt_dirs/test_community/2019061518/postprd/*.png . + +This command will copy all files ending in ``.png`` from the ``test_community/2019061518/postprd/`` experiment subdirectory on a remote HPC system into Jane Doe's present working directory (``.``). + +.. attention:: + + Note that all port numbers, IP addresses, and SSH keys included in this chapter are placeholders and do not refer to known systems. They are used purely for illustrative purposes, and users should modify the commands to correspond to their actual systems. \ No newline at end of file