Viz EC2 Setup Access and Raster Processing Fixes #455
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
these python scripts were failing due to access issues to the fileshare
unstaging tables happens concurrently with unstaging rasters unstaging rasters is a map that iterates through workspaces
This makes more sense to delete the publish flag after we confirm that the new SD file is created
TylerSchrag-NOAA
approved these changes
May 24, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There were a few additional issues going on with the pipelines.
1 - The Viz EC2 setup script was failing when running some python scripts. This was due to the fact that those scripts were hitting the fileshare and the user running them in the user data did not have access to that machine. I updated the powershell script to run those python scripts using the same user as the pipeline instead. I also moved the delete publish flag out of this powershell script and into the update_data_stores_and_sd_files.py script instead so that flags are deleted after the SD files are successfully created
2- I updated the raster processing and unstaging to occur at different stages. Now the unstaging of the rasters happens at the end with the unstaging of the DB tables so it all happens together. Fixed additional raster bugs in the pipeline as well. Updated step function to have parallel unstaging of rasters and db tables.
3 - Fixed merge issues with peak flow arrival time sql files not referencing new names for high water arrival times
4- Updated check_required_tables_updated to work with checking external table existence as well
5- Fixed bad fim_config for srf coastal inundation