Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/rtodling/geo sadas 5 29 3 config5 #130

Merged
merged 5 commits into from
Nov 16, 2021
Merged

Feature/rtodling/geo sadas 5 29 3 config5 #130

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b40b40f
adjustment for C720 and higher case - enable o-server
rtodling Nov 16, 2021
aa78bc3
adjust batch time to more reasonable value
rtodling Nov 16, 2021
fa30fd6
support for GEOS-IT
rtodling Nov 16, 2021
041ff80
agreement to leave these data out of GEOS-IT
rtodling Nov 16, 2021
0c68715
fix for CrIS-NPP on/off/on 2021; support for GEOS-IT(zero-diff)
rtodling Nov 16, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions components.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ FMS:
GEOSana_GridComp:
local: ./src/Components/@GEOSana_GridComp
remote: ../GEOSana_GridComp.git
tag: v1.5.0
tag: v1.5.1
develop: develop

GEOSgcm_GridComp:
Expand Down Expand Up @@ -121,7 +121,7 @@ mom6:
GEOSgcm_App:
local: ./src/Applications/@GEOSgcm_App
remote: ../GEOSgcm_App.git
tag: rt1_5_4_mom6
tag: rt_v1_5_6_geosit_0
develop: develop

UMD_Etc:
Expand Down
2 changes: 1 addition & 1 deletion src/Applications/GEOSdas_App/Create_anasa_script.pm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ sub anasa_script {
#SBATCH --ntasks=$ncpus_gsi
#SBATCH --ntasks-per-node=24
#SBATCH --constraint=$nodeflg
#SBATCH --time=${fcswallclk}:00
#SBATCH --time=1:30:00
#PBS -N anasa
#PBS -o anasa.log.o%j
#PBS -l ncpus=$ncpus_gsi
Expand Down
2 changes: 1 addition & 1 deletion src/Applications/GEOSdas_App/Create_asens_script.pm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ sub asens_script {
#SBATCH --ntasks=$ncpus_gsi
#SBATCH --ntasks-per-node=24
#SBATCH --constraint=$nodeflg
#SBATCH --time=${fcswallclk}:00
#SBATCH --time=2:00:00
#PBS -N asens
#PBS -o asens.log.o%j.txt
#PBS -l ncpus=$ncpus_gsi
Expand Down
8 changes: 6 additions & 2 deletions src/Applications/GEOSdas_App/edhist.pl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1359,10 +1359,14 @@ sub add_silo_mstorage_traits {
my (@anaID, @chemID, @diagID, @progID, $name, $storage);

@anaID = qw( vtx .eta .sfc .prs );
@chemID = qw( _adg_ _aer_ _chm_ _gas_ _nav_ _tag_ );
@chemID = qw( _adg_ _aer_ _chm_ _gas_ _nav_ _tag_
adg_ aer_ chm_ gas_ nav_ tag_ );
@diagID = qw( _asm_ _chm_ _cld_ _csp_ _dyn_ _ext_ _flx_ _glc_
_hwl_ _int_ _lfo_ _lnd_ _lsf_ _met_ _mst_ _ocn_
_odt_ _qdt_ _rad_ _slv_ _tdt_ _tmp_ _trb_ _udt_ _wnd_ );
_odt_ _qdt_ _rad_ _slv_ _tdt_ _tmp_ _trb_ _udt_ _wnd_
asm_ cld_ csp_ dyn_ ext_ flx_ glc_
hwl_ int_ lfo_ lnd_ lsf_ met_ mst_ ocn_
odt_ qdt_ rad_ slv_ tdt_ tmp_ trb_ udt_ wnd_ );
@progID = qw( prog traj ptrj );

# add silo trait if not present
Expand Down
87 changes: 73 additions & 14 deletions src/Applications/GEOSdas_App/fvsetup
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -386,11 +386,13 @@ my ($acqloc);
my ($fcstimes,$fcswait_hrs,$asnwait_hrs);
my ($landbcs);
my ($coupled, $ores, $mometc);
my ($o_servers);

my ($sysfile, $nodeflg);
my (@rmTilde);
my ($merra2, $acftbias, $doRcorr, $nrt, $rstype);
my ($hyb_ens, $do4diau, $newradbc, $siglevs, $vres, $aensupa);
my ($geosit, $r21c);

system clear;
get_runtime_values();
Expand Down Expand Up @@ -478,7 +480,7 @@ $ENV{"PATH"} = $FindBin::Bin .":$ENV{PATH}";
# --------------------------------
welcome();
while3 ( \&get_rem_acct_info );
while3 ( \&get_reanalysis_info );
while3 ( \&get_specific_info );
get_dimsg5gcm();

# User defined options
Expand Down Expand Up @@ -1209,13 +1211,19 @@ sub defaults {
}

#=======================================================================
sub get_reanalysis_info {
sub get_specific_info {

$geosit = 0;
$merra2 = 0;
$ans = query("\n Is this a MERRA2 experiment (y/n)?", "n");
$merra2 = 1 if yes($ans);
$r21c = 0;

$ans = query(" Is this a MERRA2 (1), GEOSIT (2), R21C (3) experiment?", "0");
return 0 unless ( $ans );

if ($ans == 1) {$merra2 = 1};
if ($ans == 2) {$geosit = 1};
if ($ans == 3) {$r21c = 1};

return 0;
}

#=======================================================================
Expand Down Expand Up @@ -3498,6 +3506,11 @@ EOF
$berror_file = "\$FVHOME/fvInput/gsi/etc/berror_gmao/gmao24Jun2011_fp+oz_fix/MERRA2/Final";
$berror_env = "setenv BERROR $berror_file";
}
if ( $geosit ) {
# $berror_file = "\$FVHOME/fvInput/gsi/etc/berror_gmao/gmao24Jun2011_fp+oz_fix/MERRA2/Final";
# $berror_env = "setenv BERROR $berror_file";
$berror_env = "# Likely need to point to new BERROR for GEOSIT (adjust fvsetup)";
}
}
if ( $siglevs == 72 || $siglevs == 91 || $siglevs == 132 || $siglevs == 137 || $siglevs == 181 ) { $hybrid = ".true." };

Expand Down Expand Up @@ -4305,10 +4318,15 @@ sub get_history {
# query for HISTORY.rc.tmpl file
#-------------------------------
@HISTORY = ( <$fvetc/HISTORY*.rc.tmpl*> );
$g5hist_rc = "HISTORY.rc.tmpl";
if ( $merra2 ) {
$g5hist_rc = "HISTORY_MERRA2.rc.tmpl";
} else {
$g5hist_rc = "HISTORY.rc.tmpl";
}
if ( $geosit ) {
$g5hist_rc = "HISTORY_GEOSIT.rc.tmpl";
}
if ( $r21c ) {
$g5hist_rc = "HISTORY_R21C.rc.tmpl";
}
$g5hist_rc = basename $HISTORY[0] unless -e "$fvetc/$g5hist_rc";

Expand Down Expand Up @@ -4682,6 +4700,7 @@ EOF
$nx_pert = 1;
$ny_pert = 6 * $nx_pert;
$use_shmem = 0;
$o_servers = 0;
$ios_nds = 1;
$cldmicro = "1MOMENT";
if ( "$res" eq "c" && "$vres" eq "55" ) {
Expand Down Expand Up @@ -5022,6 +5041,7 @@ EOF
$ana_jm_ens = 181;
} elsif ( "$res" eq "C720" ) { # Cubed-sphere
$cubed = 1;
$o_servers = 8;
# $ios_nds = 3;
$specres = "254";
$jcap = "254";
Expand Down Expand Up @@ -5056,6 +5076,7 @@ EOF
$ana_jm_ens = 361;
} elsif ( "$res" eq "C1440" ) { # Cubed-sphere
$cubed = 1;
$o_servers = 8;
# $ios_nds = 4;
$specres = "254";
$jcap = "254";
Expand Down Expand Up @@ -7478,6 +7499,8 @@ if ( $fvchem && $lm > 2 ) {
}
$pi = $i; # p for PSAS

my $xncpus = $ncpus + $o_servers * $ncpus_per_node;

# Turn off some parallelization on discover
#------------------------------------------
if ( ($siteID eq "nccs") || ($siteID eq "nas") ) {
Expand Down Expand Up @@ -7516,7 +7539,7 @@ if ( $siteID eq "nccs" ) {
print SCRIPT <<"EOF";
#SBATCH --job-name=$jobn
#SBATCH --output=$jobn.log.o%j.txt
#SBATCH --ntasks=$ncpus
#SBATCH --ntasks=$xncpus
#SBATCH --ntasks-per-node=$ncpus_per_node
#SBATCH --constraint=$nodeflg
#SBATCH --time=${daswallclk}:00
Expand Down Expand Up @@ -7591,11 +7614,13 @@ print SCRIPT <<"EOF";
setenv ARCH `uname -s`
setenv HOST `uname -n`
setenv NCPUS $ncpus # Number of CPUs to run GCM
setenv NCPUSX $xncpus # Number of CPUs plus IO-server requirement
setenv NCPUS_IDF $ncpus_idf # Number of CPUs to run IDF
setenv NCPUS_IAU $ncpus_iau # Number of CPUs to run IAU
setenv NCPUS_GSI $ncpus_gsi # Number of CPUs to run GSI
setenv NCPUS_GPERT $ncpus_gpert # Number of CPUs to run gcmPERT
setenv NCPUS_AOD $ncpus_aod # Number of CPUs to run PSAS-AOD
setenv O_SERVERS $o_servers # Number of IO servers
setenv GAAS_RUN_SLURM 1 # launch AOD analysis as separate batch job
setenv AODBLOCKJOB 1
EOF
Expand Down Expand Up @@ -7821,9 +7846,10 @@ print SCRIPT <<"EOF";
setenv PSM2_MEMORY large
setenv I_MPI_ADJUST_GATHERV 3
setenv I_MPI_ADJUST_ALLREDUCE 12
# setenv I_MPI_EXTRA_FILESYSTEM 1
# setenv I_MPI_EXTRA_FILESYSTEM_LIST gpfs
# setenv ROMIO_FSTYPE_FORCE "gpfs:"
setenv I_MPI_EXTRA_FILESYSTEM 1
setenv I_MPI_EXTRA_FILESYSTEM_LIST gpfs
setenv ROMIO_FSTYPE_FORCE "gpfs:"
setenv I_MPI_FABRICS shm:ofi
# setenv I_MPI_FABRICS shm:dapl
# setenv I_MPI_FABRICS_LIST "dapl,ofa"
# setenv I_MPI_FALLBACK "enable"
Expand Down Expand Up @@ -7985,7 +8011,11 @@ print SCRIPT <<"EOF";
endif
setenv RUN_OPT_BLEND "esma_mpirun -np \$NCPUS \$HDF2RSX"
if (\$G5GCM ) then
setenv RUN_OPT_BEGIN "esma_mpirun -np \$NCPUS \$GCMX"
if ( \$O_SERVERS > 0 ) then
setenv RUN_OPT_BEGIN "mpirun -np \$NCPUSX \$GCMX --npes_model \$NCUPS --nodes_output_server \$O_SERVERS --oserver_type multigroup --npes_backend_pernode \$O_SERVERS"
else
setenv RUN_OPT_BEGIN "esma_mpirun -np \$NCPUS \$GCMX"
endif
setenv ADMRUN_OPT_BEGIN "esma_mpirun -np \$NCPUS_GPERT \$GCMPTX"
else
setenv RUN_OPT_BEGIN "esma_mpirun -np \$AGCM_NUM_MPI \$GCMX"
Expand Down Expand Up @@ -8672,6 +8702,8 @@ sub create_fscript {
open(SCRIPT,">$fvhome/fcst/$jobf.j") or
die ">>> ERROR <<< cannot write $fvhome/fcst/$jobf.j";

my $xncpus = $ncpus + $o_servers * $ncpus_per_node;

print SCRIPT <<"EOF";
#!/bin/csh -fx
# ------------------------------------------
Expand All @@ -8690,7 +8722,7 @@ if ( $siteID eq "nccs" ) {
print SCRIPT <<"EOF";
#SBATCH --job-name=fcst
#SBATCH --output=fcst.log.o%j.txt
#SBATCH --ntasks=$ncpus
#SBATCH --ntasks=$xncpus
#SBATCH --ntasks-per-node=$ncpus_per_node
#SBATCH --constraint=$nodeflg
#SBATCH --time=${fcswallclk}:00
Expand Down Expand Up @@ -8760,10 +8792,12 @@ print SCRIPT <<"EOF";
setenv ARCH `uname -s`
setenv HOST `uname -n`
setenv NCPUS $ncpus # number of CPUS
setenv NCPUSX $xncpus # NCPUS plus IO-server requirements
setenv NCPUS_IDF $ncpus_idf # Numbers of CPUs to run IDF
setenv NCPUS_IAU $ncpus_iau # Numbers of CPUs to run IAU
setenv NCPUS_GSI $ncpus_gsi # Numbers of CPUs to run GSI
setenv NCPUS_GPERT $ncpus_gpert # Numbers of CPUs to run GSI
setenv O_SERVERS $o_servers # Number of IO servers
setenv N_CPU \$NCPUS
setenv EXPID $expid # experiment ID
setenv CASE \$EXPID # experiment ID (for LSM's sake)
Expand Down Expand Up @@ -8875,6 +8909,7 @@ print SCRIPT <<"EOF";
# -----------------
if (\$?I_MPI_ROOT) then
# setenv I_MPI_USE_DYNAMIC_CONNECTIONS 0
setenv I_MPI_FABRICS shm:ofi
# setenv I_MPI_FABRICS shm:dapl
# setenv I_MPI_FABRICS_LIST "dapl,ofa"
# setenv I_MPI_FALLBACK "enable"
Expand Down Expand Up @@ -9160,7 +9195,11 @@ print SCRIPT <<"EOF";
setenv MPIRUN "esma_mpirun -np \$PSAS_NUM_MPI "
setenv RUN_OPT_BLEND "esma_mpirun -np \$NCPUS \$HDF2RSX"
if (\$G5GCM ) then
setenv RUN_OPT_BEGIN "esma_mpirun -np \$NCPUS \$GCMX"
if ( \$O_SERVERS > 0 ) then
setenv RUN_OPT_BEGIN "mpirun -np \$NCPUSX \$GCMX --npes_model \$NCUPS --nodes_output_server \$O_SERVERS --oserver_type multigroup --npes_backend_pernode \$O_SERVERS"
else
setenv RUN_OPT_BEGIN "esma_mpirun -np \$NCPUS \$GCMX"
endif
setenv ADMRUN_OPT_BEGIN "esma_mpirun -np \$NCPUS_GPERT \$GCMPTX"
else
setenv RUN_OPT_BEGIN "esma_mpirun -np \$AGCM_NUM_MPI \$GCMX"
Expand Down Expand Up @@ -10080,6 +10119,26 @@ sub copy_resources {
if (-e "$pyradmon/scripts/radmon.defaults.rc") {
cp("$pyradmon/scripts/radmon.defaults.rc", "$fvhome/radmon");
}

# When applicable, overwrite w/ pre-set RC files for GSI
# ------------------------------------------------------
if ( $geosit ) {
if ( -d "$fvetc/gsi/GEOSIT" ) {
my @files = glob("$fvetc/gsi/GEOSIT" . "/*");
foreach my $fn ( @files ) {
cp("$fn", "$fvhome/run");
}
}
}
if ( $r21c ) {
if ( -d "$fvetc/gsi/R21C" ) {
my @files = glob("$fvetc/gsi/R21C" . "/*");
foreach my $fn ( @files ) {
cp("$fn", "$fvhome/run");
}
}
}

}

#=======================================================================
Expand Down
2 changes: 1 addition & 1 deletion src/Applications/GEOSdas_App/testsuites/C180RPY.input
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ tag: GEOSadas-5_29_3
Remote account for Intranet plots? [rtodling@train]
>

Is this a MERRA2 experiment (y/n)? [n]
Is this a MERRA2 (1), GEOSIT (2), R21C (3) experiment? [0]
>

AGCM Horizontal Resolution? [C48]
Expand Down
2 changes: 1 addition & 1 deletion src/Applications/GEOSdas_App/testsuites/C180T14RPY.input
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ tag: GEOSadas-5_29_3
Remote account for Intranet plots? [rtodling@train]
>

Is this a MERRA2 experiment (y/n)? [n]
Is this a MERRA2 (1), GEOSIT (2), R21C (3) experiment? [0]
>

AGCM Horizontal Resolution? [C48]
Expand Down
2 changes: 1 addition & 1 deletion src/Applications/GEOSdas_App/testsuites/C360L181_replay.input
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ tag: GEOSadas-5_29_3
Remote account for Intranet plots? [dao_it@train]
>

Is this a MERRA2 experiment (y/n)? [n]
Is this a MERRA2 (1), GEOSIT (2), R21C (3) experiment? [0]
>

AGCM Horizontal Resolution? [C48]
Expand Down
2 changes: 1 addition & 1 deletion src/Applications/GEOSdas_App/testsuites/C360L91_replay.input
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ tag: GEOSadas-5_29_3
Remote account for Intranet plots? [dao_it@train]
>

Is this a MERRA2 experiment (y/n)? [n]
Is this a MERRA2 (1), GEOSIT (2), R21C (3) experiment? [0]
>

AGCM Horizontal Resolution? [C48]
Expand Down
2 changes: 1 addition & 1 deletion src/Applications/GEOSdas_App/testsuites/C48f.input
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ fvsetupflags: -sensdeg 1
Remote account for Intranet plots? [jstassi@train]
>

Is this a MERRA2 experiment (y/n)? [n]
Is this a MERRA2 (1), GEOSIT (2), R21C (3) experiment? [0]
>

AGCM Horizontal Resolution? [C48]
Expand Down
2 changes: 1 addition & 1 deletion src/Applications/GEOSdas_App/testsuites/C90C.input
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ tag: GEOSadas-5_29_3
Remote account for Intranet plots? [jstassi@train]
>

Is this a MERRA2 experiment (y/n)? [n]
Is this a MERRA2 (1), GEOSIT (2), R21C (3) experiment? [0]
>

AGCM Horizontal Resolution? [C48]
Expand Down
2 changes: 1 addition & 1 deletion src/Applications/GEOSdas_App/testsuites/C90C_ens.input
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ tag: GEOSadas-5_29_3
Remote account for Intranet plots? [jstassi@train]
>

Is this a MERRA2 experiment (y/n)? [n]
Is this a MERRA2 (1), GEOSIT (2), R21C (3) experiment? [0]
>

AGCM Horizontal Resolution? [C48]
Expand Down
2 changes: 1 addition & 1 deletion src/Applications/GEOSdas_App/testsuites/C90C_replay.input
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ tag: 86f27c6
Remote account for Intranet plots? [rtodling@train]
>

Is this a MERRA2 experiment (y/n)? [n]
Is this a MERRA2 (1), GEOSIT (2), R21C (3) experiment? [0]
>

AGCM Horizontal Resolution? [C48]
Expand Down
2 changes: 1 addition & 1 deletion src/Applications/GEOSdas_App/testsuites/fpp.input
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ tag: GEOSadas-5_29_3
Remote account for Intranet plots? [dao_ops@train]
>

Is this a MERRA2 experiment (y/n)? [n]
Is this a MERRA2 (1), GEOSIT (2), R21C (3) experiment? [0]
>

AGCM Horizontal Resolution? [C48]
Expand Down
Loading