Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Newly added gfsfbwinds job fails to run #2160

Closed
GwenChen-NOAA opened this issue Dec 14, 2023 · 5 comments · Fixed by #2161 or #2479
Closed

Newly added gfsfbwinds job fails to run #2160

GwenChen-NOAA opened this issue Dec 14, 2023 · 5 comments · Fixed by #2161 or #2479
Assignees
Labels
bug Something isn't working

Comments

@GwenChen-NOAA
Copy link
Contributor

GwenChen-NOAA commented Dec 14, 2023

What is wrong?

Newly added gfsfbwinds job fails to run and create a bulletin for Pacific region.

What should have happened?

gfsfbwinds job should run successfully and create a bulletin for Pacific region.

What machines are impacted?

WCOSS2

Steps to reproduce

Run the following experiment on WCOSS2:

./setup_expt.py gfs forecast-only --app ATM --resdet 384 --start cold --comrot $COMROT --expdir $EXPDIR --idate 2016070100 --edate 2016070100 --pslot test

with DO_AWIPS set to "YES" in config.base

Additional information

Per Rahul's request, this issue is opened for each of the jobs in Issue #2151 with relevant details.

Error messages in the log file indicate that output directory paths are not set up correctly:

  • JGFS_ATMOS_FBWIND[38]: mkdir -m 775 -p /lfs/h1/ops/prod/com/gfs/v16.3/gfs.20160701/00/atmos /lfs/h1/ops/prod/com/gfs/v16.3/gfs.20160701/00/atmos/wmo
    mkdir: cannot create directory â /lfs/h1/ops/prod/com/gfs/v16.3/gfs.20160701â : Permission denied
    mkdir: cannot create directory â /lfs/h1/ops/prod/com/gfs/v16.3/gfs.20160701â : Permission denied
  • JGFS_ATMOS_FBWIND[1]: postamble JGFS_ATMOS_FBWIND 1702585814 1
  • preamble.sh[68]: set +x
    End JGFS_ATMOS_FBWIND at 20:30:16 with error code 1 (time elapsed: 00:00:02)

gfsfbwinds.log

Do you have a proposed solution?

No response

@GwenChen-NOAA GwenChen-NOAA added bug Something isn't working triage Issues that are triage labels Dec 14, 2023
@JessicaMeixner-NOAA
Copy link
Contributor

@GwenChen-NOAA this seems like a simple typo that https://github.com/NOAA-EMC/global-workflow/blob/develop/jobs/rocoto/fbwinds.sh#L14 should change from:
"${HOMEgfs}/jobs/JGFS_ATMOS_FBWINDS"
to
"${HOMEgfs}/jobs/JGFS_ATMOS_FBWIND"

Or https://github.com/NOAA-EMC/global-workflow/blob/develop/jobs/JGFS_ATMOS_FBWIND should be renamed to JGFS_ATMOS_FBWINDS

@aerorahul aerorahul mentioned this issue Dec 14, 2023
7 tasks
@WalterKolczynski-NOAA WalterKolczynski-NOAA removed the triage Issues that are triage label Dec 14, 2023
@GwenChen-NOAA
Copy link
Contributor Author

Good catch, @JessicaMeixner-NOAA! I changed "${HOMEgfs}/jobs/JGFS_ATMOS_FBWINDS" to "${HOMEgfs}/jobs/JGFS_ATMOS_FBWIND" in fbwinds.sh, and rerun the task. It runs but not successful with the error messages below:

  • JGFS_ATMOS_FBWIND[38]: mkdir -m 775 -p /lfs/h1/ops/prod/com/gfs/v16.3/gfs.20160701/00/atmos /lfs/h1/ops/prod/com/gfs/v16.3/gfs.20160701/00/atmos/wmomkdir: cannot create directory â/lfs/h1/ops/prod/com/gfs/v16.3/gfs.20160701â: Permission deniedmkdir: cannot create directory â/lfs/h1/ops/prod/com/gfs/v16.3/gfs.20160701â: Permission denied+ JGFS_ATMOS_FBWIND[1]: postamble JGFS_ATMOS_FBWIND 1702585814 1
  • preamble.sh[68]: set +xEnd JGFS_ATMOS_FBWIND at 20:30:16 with error code 1 (time elapsed: 00:00:02)

Same error as the gfsgempakncdcupapgif job (#2157 (comment)). The output directory should be redirected to $COMROT.

@aerorahul
Copy link
Contributor

@GwenChen-NOAA
Again, it is not surprising that the job is trying to create and write to operational space. This job and its scripts (like the others) need to be examined, and updated to be able to be run in development mode.

@GwenChen-NOAA
Copy link
Contributor Author

@aerorahul, do you want to reopen this issue since the job still not running? Or should I open a new issue for it?

@aerorahul
Copy link
Contributor

I can reopen, but please update the issue to bring to the front the problem.

@aerorahul aerorahul reopened this Dec 15, 2023
@WalterKolczynski-NOAA WalterKolczynski-NOAA self-assigned this Jan 2, 2024
WalterKolczynski-NOAA added a commit to WalterKolczynski-NOAA/global-workflow that referenced this issue Apr 12, 2024
Updates fbwind job for the COM refactor and some other cleanup.

fbwind IS STILL NON-FUNCTIONAL after this commit. The exectuable is
failing to read the grib1 index file. The grib1 data file looks fine,
so this appears to me a problem either with the GRBINDEX executable or
with GETGB.

Refs: NOAA-EMC#289
Refs: NOAA-EMC#2160
WalterKolczynski-NOAA added a commit to WalterKolczynski-NOAA/global-workflow that referenced this issue Apr 13, 2024
Updates fbwind job for the COM refactor and some other cleanup.

fbwind IS STILL NON-FUNCTIONAL after this commit. The exectuable is
failing to read the grib1 index file. The grib1 data file looks fine,
so this appears to me a problem either with the GRBINDEX executable or
with GETGB.

Refs: NOAA-EMC#289
Refs: NOAA-EMC#2160
WalterKolczynski-NOAA added a commit that referenced this issue Apr 15, 2024
Updates fbwind job for the COM refactor and some other cleanup.

This works on WCOSS but not on Orion. There seems to be a problem with
either `grbindex` or `GETGB()` on Orion that causes the executable to be
unable to read the grib1 index file. The grib1 data file produced there
seems fine. Haven't checked Hera yet, maybe there is a problem with the
spack-stack build of `grbindex`.

Resolves: #2160
Refs: #289
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
4 participants