ecFlow Suite Generator Application #777
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently working with the ecflow base configurations. It loads in the tasks, families, and various other nodes, allowing for different suite hours to be created.
After coordinating with Rahul, Kate, Walter, and Lin, the result was a slimmed down version of the GFSApp for forecast only definition file. Committed here for use with unittests and reference
After the meeting on 2/18, the generator was reworked to use directory traversal to build the suites. This content was presented on 3/1 but ultimately determined that the family structure should be within the YAML so this is a working version but will be redone to meet the new requirements.
This is the new setup and definitions setup using the new ecflow build structure vetted by the team. Still requires inclusion of triggers and events.
Completion of the work by 3/11/22. Given the new requirements last week, this refactors the ecflow_setup script to handle the configuration file setup and then all the defintion pieces are handled in the ecflow_definitions. This also manages the looping structures much better and adds in some exceptions.
There were files originally included with the first release that are no longer needed. This removes them.
Added in the list capabilities for the inventory for edits.
Updated the logic for the tasks and the events to loop through lists but the triggers logic is still broken if the events are looped.
The event loops previously errored out due to the int being cast as strings. This commit fixes that up.
This generates a date repeat logic and basic time but doesn't work properly due to time being limited to a 24 hour time stamp.
Now uses cron for abnormal times over a day, a time if the start and end is within the same day and date if it is looping through days
This commit was done after the demonstration on 3/29 to the team.
The next steps are to change out the env. for ${} syntax and then
also start trying to put together a suite definition from current
production to identify any possible errors.
Additional tasks are:
- Documentation
- Identify a way to use previously defined variables within the
manifest with $()
Added in a readme file for use of the workflow object.
Currently working with the ecflow base configurations. It loads in the tasks, families, and various other nodes, allowing for different suite hours to be created.
After coordinating with Rahul, Kate, Walter, and Lin, the result was a slimmed down version of the GFSApp for forecast only definition file. Committed here for use with unittests and reference
After the meeting on 2/18, the generator was reworked to use directory traversal to build the suites. This content was presented on 3/1 but ultimately determined that the family structure should be within the YAML so this is a working version but will be redone to meet the new requirements.
This is the new setup and definitions setup using the new ecflow build structure vetted by the team. Still requires inclusion of triggers and events.
Completion of the work by 3/11/22. Given the new requirements last week, this refactors the ecflow_setup script to handle the configuration file setup and then all the defintion pieces are handled in the ecflow_definitions. This also manages the looping structures much better and adds in some exceptions.
There were files originally included with the first release that are no longer needed. This removes them.
Added in the list capabilities for the inventory for edits.
Updated the logic for the tasks and the events to loop through lists but the triggers logic is still broken if the events are looped.
The event loops previously errored out due to the int being cast as strings. This commit fixes that up.
This generates a date repeat logic and basic time but doesn't work properly due to time being limited to a 24 hour time stamp.
Now uses cron for abnormal times over a day, a time if the start and end is within the same day and date if it is looping through days
This commit was done after the demonstration on 3/29 to the team.
The next steps are to change out the env. for ${} syntax and then
also start trying to put together a suite definition from current
production to identify any possible errors.
Additional tasks are:
- Documentation
- Identify a way to use previously defined variables within the
manifest with $()
Added in a readme file for use of the workflow object.
…ns-NOAA/global-workflow into feature/ecflow_suite_generator
This change adds in the prod.yml file which is the start of the production definition file and the creation of multiple suites. In addition there have been fixes like the addition of the defstatus.
Added in the ability for the triggers to have an "or" addition and also updated the production YAML further.
Added additional sections to the README as well as updated in line code for the main script.
Added documentation to the setup_workflow and ecflow_setup modules and also made some adjustments based on flake8's recommendations.
Added in more code documentation and PEP 8 compliance.
Added the documentation for the various nodes along with notes about the ranges and lists to the readme. Also added in docs to the code following PEP8 style.
Based on the PR review, this commit adds in a lot of the RETURN and PARAMETER sections even if the value of them is none. Some of the sections were missed during the documentation phase. Also, some of the camelcase was fixed to use snake case. Also, there was a bug fix for the events range function. It was using a bad variable and so it didn't actually add the event.
WalterKolczynski-NOAA
approved these changes
May 11, 2022
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR is to merge in the workflow generator application that uses a YAML file to create suite definition files for ecflow.
Type of change
How Has This Been Tested?
To test:
cd global-workflow/workflow_generatorpython3 setup_workflow.py --expdir ../parm/config** Remaining items **