Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds fixes for new target_cols ingest mechanism #512

Merged
merged 3 commits into from
Aug 23, 2023

Conversation

shawncrawley
Copy link
Collaborator

This includes a number of fixes that were required as a result of the new "target_cols" concept that is tied to the "viz_db_ingest" lambda. It's somewhat annoying to manually create the ingest tables for new files - or in this case even files we've been ingesting but which now require additional columns (i.e. "target_cols"). We historically would manually create them since it didn't make sense to create them in the "viz_db_ingest" map items loop. But I added the "iteration_index" argument to the map calls to that function, and thus can check in the function if iteration_index == 0 then we set a create_table variable to True. If either an UndefinedTable (i.e. table does not exist) or BadCopyFileFormat (i.e. a new column is now being ingested that wasn't before) error is thrown and create_table is true, then the table schema will be recreated based exactly on the DataFrame that is being ingested, and the ingest will try again. If the iteration_index is not 0 and one of those same errors is encountered, then the error is caught at the Step Function management level and will retry after waiting 5 seconds.

With this new ingest methodology, a few column names slightly changed. This is because the actual DataFrame column name was used instead of whatever name that was previously manually assigned. Thus, a few product SQL queries had to be updated.

There may be a few remaining issues that I haven't been able to catch yet - since it's only now been less than an hour since fixing the frequent offenders (e.g. lambda_rfc, sns_ana, and man_rnr). If it looks good in the morning, we should immediately deploy this to TI. Once we quickly re-confirm that everything seems to be working, we should then deploy this to UAT.

@CoreyKrewson-NOAA CoreyKrewson-NOAA added this to the V2.1.2 milestone Aug 23, 2023
@nickchadwick-noaa nickchadwick-noaa merged commit 694427f into ti Aug 23, 2023
@nickchadwick-noaa nickchadwick-noaa deleted the ti-fixes-following-rnr-update branch August 23, 2023 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants