Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance updates #897

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Conversation

JoshCu
Copy link

@JoshCu JoshCu commented Feb 19, 2025

Tidies up and adds several performance enhancements:
It should be possible to cherry-pick any of the commits to pick and choose what changes to keep.

full details of original testing: ngen init speedup troute speedup parallel processing speedup

  • Adds new optional realization parameter write_catchment_outputs which allows you to disable the individual catchment csv files, useful for calibration and reducing the number of files produced running over large areas
  • Replaced the mpi barriers with manual wait loops that poll every 100ms to reduce cpu useage, allowing troute to use the full cpu
  • Adds an option to disable netcdf forcing cache, speedup for running in parallel
  • Adds script to create partitions with no remotes (no mpi communication), speedup for running in parallel
  • Refactors the forcing file finding to speed up the ngen init portion of the ngen run

Tested in ngiab with additional the script added for the alternative partitioning, url here.

Note

The partitioning script is currently just a python script that only works with geopackage input. When I get it properly incorporated into the existing partition generator I will update this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant