Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Viz EC2 Services to Lambda - Part 1 - High Water Probability #543

Merged
merged 14 commits into from
Sep 28, 2023

Conversation

TylerSchrag-NOAA
Copy link
Contributor

@TylerSchrag-NOAA TylerSchrag-NOAA commented Sep 20, 2023

This PR migrates the ensemble-based High Water Probability products from the Viz EC2 to the Viz Max Values lambda function (now called Python Preprocessing). It broadly encapsulates the following changes:

Changes

  • Renaming of the Viz Max Values lambda function to Viz Python Preprocessing
  • Renaming of max_values (and variations) to python_preprocessing throughout code base
  • Abstraction of python_preprocessing to function to include multiple "products", with max_values now being one of those products.
  • New high_water_probability product script that supports both 12 hour SRF and 5 day GFS MRF configurations.
  • Removal of high_water_probability pipeline and product file in the source-aws_loosa library
  • Minor changes to viz_shared_functions to support multiple range tokens in file patterns

Testing - I've tested this new functionality with SRF, MRF, and ANA configurations fairly vigorously in the dev historic event version of the pipelines, which was created in August 2023. That said, I have not tested it with coastal configurations, which do rely on max_values for FIM preprocessing (although I don't expect my changes to affect these at all). I have also not thoroughly tested the actual map service component on the EGIS, as I cannot access the portal. I believe it's close enough to deploy to TI for testing this functionality.

Deployment Considerations:

  • We should definitely wait until the upcoming public release deployment is well and settled before merging this into TI, as it includes foundational changes to the pipelines that we do not want to contend with for hot fixes.
  • I changed the mrf service name to include "max", to be consistent with the SRF services. Let me know if that isn't correct, but we'll need to coordinate with WPOD on app changes for that.
  • We will need to include a ingest schema db dump when deploying to UAT.

@TylerSchrag-NOAA TylerSchrag-NOAA added the enhancement New feature or request label Sep 20, 2023
@TylerSchrag-NOAA TylerSchrag-NOAA added this to the V2.1.4 milestone Sep 20, 2023
@TylerSchrag-NOAA TylerSchrag-NOAA self-assigned this Sep 20, 2023
@TylerSchrag-NOAA TylerSchrag-NOAA linked an issue Sep 21, 2023 that may be closed by this pull request
8 tasks
Copy link
Collaborator

@shawncrawley shawncrawley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good - and it is super cool to be moving that much closer to no more EC2 pipeline processing! Well done.

@CoreyKrewson-NOAA CoreyKrewson-NOAA merged commit 206e616 into ti Sep 28, 2023
@CoreyKrewson-NOAA CoreyKrewson-NOAA mentioned this pull request Oct 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Move High Water Probability (and maybe ROF Probability) from EC2 to Lambda
3 participants