Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DO NOT MERGE] Weekly CI Tests Friday Jan 19, 2024 #2240

Closed
wants to merge 1 commit into from
Closed

[DO NOT MERGE] Weekly CI Tests Friday Jan 19, 2024 #2240

wants to merge 1 commit into from

Conversation

emcbot
Copy link

@emcbot emcbot commented Jan 19, 2024

[DO NOT MERGE] Weekly CI Tests Friday Jan 19, 2024

@emcbot emcbot marked this pull request as draft January 19, 2024 22:47
@emcbot emcbot added CI/CD Issue related to CI/CD CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion and removed CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion labels Jan 19, 2024
@emcbot
Copy link
Author

emcbot commented Jan 19, 2024 

CI Update on Orion at 01/19/24 04:48:08 PM
============================================
Cloning and Building global-workflow PR: 2240
with PID: 58721 on host: Orion-login-1

@emcbot emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Jan 19, 2024
@emcbot
Copy link
Author

emcbot commented Jan 19, 2024 

CI Update on Hera at 01/19/24 10:48:11 PM
============================================
Cloning and Building global-workflow PR: 2240
with PID: 2097 on host: hfe05

@emcbot emcbot added CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Jan 19, 2024
@emcbot
Copy link
Author

emcbot commented Jan 19, 2024

Automated global-workflow Testing Results:

Machine: Hera
Start: Fri Jan 19 23:02:08 UTC 2024 on hfe05
---------------------------------------------------
Build: Completed at 01/19/24 11:46:48 PM
Case setup: Completed for experiment C384C192_hybatmda_b0ee609e
Case setup: Completed for experiment C384_S2SWA_b0ee609e
Case setup: Completed for experiment C384_atm3DVar_b0ee609e

@emcbot emcbot added CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress and removed CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Jan 19, 2024
@emcbot
Copy link
Author

emcbot commented Jan 19, 2024

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Jan 19 16:51:49 CST 2024 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build: Completed at 01/19/24 05:50:04 PM
Case setup: Completed for experiment C384_atm3DVar_b0ee609e
Case setup: Completed for experiment C384C192_hybatmda_b0ee609e
Case setup: Completed for experiment C384_S2SWA_b0ee609e

@emcbot emcbot added CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Jan 20, 2024
@emcbot
Copy link
Author

emcbot commented Jan 20, 2024 

Experiment C384_S2SWA_b0ee609e  *** FAILED *** on Orion
Experiment C384_S2SWA_b0ee609e  with 1 tasks failed at 01/19/24 06:20:32 PM
Error logs:
/work2/noaa/stmp/GFS_CI_ROOT/ORION/PR/2240/RUNTESTS/COMROOT/C384_S2SWA_b0ee609e/logs/2016070100/gfsfcst.log

@emcbot emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Jan 20, 2024
@emcbot
Copy link
Author

emcbot commented Jan 20, 2024 

Experiment C384_S2SWA_b0ee609e  *** FAILED *** on Hera
Experiment C384_S2SWA_b0ee609e  with 1 tasks failed at 01/20/24 12:32:22 AM
Error logs:
/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/2240/RUNTESTS/COMROOT/C384_S2SWA_b0ee609e/logs/2016070100/gfsfcst.log

@DavidHuber-NOAA
Copy link
Contributor

The C384_S2SWA GFS forecast is failing on both Orion and Hera with MAPL errors starting in ExtDataGridCompMod.F90:

   0:  in radiation_clouds_prop=           8 F           4 F F           2           1
   0:  in radiation_clouds_prop=           8 F           4 F F           2           1
   0:  in radiation_clouds_prop=           8 F           4 F F           2           1
   0:  in radiation_clouds_prop=           8 F           4 F F           2           1
   0:  in radiation_clouds_prop=           8 F           4 F F           2           1
 762: pe=00762 FAIL at line=02588    ExtDataGridCompMod.F90                   <unknown error>
 762: pe=00762 FAIL at line=01323    ExtDataGridCompMod.F90                   <status=-1>
 762: pe=00762 FAIL at line=01901    MAPL_Generic.F90                         <status=-1>
 762: pe=00762 FAIL at line=01243    MAPL_CapGridComp.F90                     <status=-1>
 762: pe=00762 FAIL at line=01206    MAPL_CapGridComp.F90                     <status=-1>
 762: pe=00762 FAIL at line=01166    MAPL_CapGridComp.F90                     <status=-1>
 762: pe=00762 FAIL at line=00834    MAPL_CapGridComp.F90                     <status=-1>
 762: pe=00762 FAIL at line=00974    MAPL_CapGridComp.F90                     <status=-1>
...

@WalterKolczynski-NOAA
Copy link
Contributor

May be related to the changes last week in #2201, though the low-res tests passed. @bbakernoaa can you take a look?

@bbakernoaa
Copy link
Contributor

Yea I'll take a look first thing tomorrow

@bbakernoaa
Copy link
Contributor

@WalterKolczynski-NOAA @DavidHuber-NOAA

Found the issue

We needed to switch the flag for climatology in the biogenic files. Do you want me to setup a separate PR or combine it with this one?

The change is very easy. The lines that needs changed are here: https://github.com/NOAA-EMC/global-workflow/blob/develop/parm/ufs/gocart/ExtData.other#L57-L60

specifically

OC_ISOPRENE NA  N Y %y4-%m2-%d2t12:00:00 none none isoprene ExtData/nexus/MEGAN_OFFLINE_BVOC/v2021-12/MEGAN_OFFLINE_CLIMO_2000_2022_%m2.nc
OC_LIMO     NA  N Y %y4-%m2-%d2t12:00:00 none none limo     ExtData/nexus/MEGAN_OFFLINE_BVOC/v2021-12/MEGAN_OFFLINE_CLIMO_2000_2022_%m2.nc
OC_MTPA     NA  N Y %y4-%m2-%d2t12:00:00 none none mtpa     ExtData/nexus/MEGAN_OFFLINE_BVOC/v2021-12/MEGAN_OFFLINE_CLIMO_2000_2022_%m2.nc
OC_MTPO     NA  N Y %y4-%m2-%d2t12:00:00 none none mtpo     ExtData/nexus/MEGAN_OFFLINE_BVOC/v2021-12/MEGAN_OFFLINE_CLIMO_2000_2022_%m2.nc

needs to be changed to Y in the 3rd column. Like so....

OC_ISOPRENE NA  Y Y %y4-%m2-%d2t12:00:00 none none isoprene ExtData/nexus/MEGAN_OFFLINE_BVOC/v2021-12/MEGAN_OFFLINE_CLIMO_2000_2022_%m2.nc
OC_LIMO     NA  Y Y %y4-%m2-%d2t12:00:00 none none limo     ExtData/nexus/MEGAN_OFFLINE_BVOC/v2021-12/MEGAN_OFFLINE_CLIMO_2000_2022_%m2.nc
OC_MTPA     NA  Y Y %y4-%m2-%d2t12:00:00 none none mtpa     ExtData/nexus/MEGAN_OFFLINE_BVOC/v2021-12/MEGAN_OFFLINE_CLIMO_2000_2022_%m2.nc
OC_MTPO     NA  Y Y %y4-%m2-%d2t12:00:00 none none mtpo     ExtData/nexus/MEGAN_OFFLINE_BVOC/v2021-12/MEGAN_OFFLINE_CLIMO_2000_2022_%m2.nc

@WalterKolczynski-NOAA
Copy link
Contributor

@bbakernoaa Great job. Separate PR please, this "PR" isn't a real PR, it is a kludge for the CI system to run weekly testing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA Awaiting requested review from TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA will be requested when the pull request is marked ready for review TerrenceMcGuinness-NOAA is a code owner

@WalterKolczynski-NOAA WalterKolczynski-NOAA Awaiting requested review from WalterKolczynski-NOAA WalterKolczynski-NOAA will be requested when the pull request is marked ready for review WalterKolczynski-NOAA is a code owner

Assignees
No one assigned
Labels
CI/CD Issue related to CI/CD CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@emcbot @DavidHuber-NOAA @WalterKolczynski-NOAA @bbakernoaa @TerrenceMcGuinness-NOAA