Improve peak flow arrival time sql

[CoreyKrewson-NOAA]

The MRF peak flow arrival time sql times out occasionally. This seems to only happen when all MRF sql files are being ran at once so it seems to be just slow enough to timeout but be ok when it is the only thing running. Can we find any efficiency improvements in the sql to stop causing these timeouts?

[TylerSchrag-NOAA]

One reason this service is slow is that it replicates the 'high water return time' logic from the arrival time script, and they often run at the same time (I told Katherine this was OK for now, and that I had this fix in mind for later).

What I plan to do in this next sprint is to add a service dependency column to the admin.services table. I'll setup a check at the start of the postprocess_sql (Service) function that looks for the current reference time in that dependency service table, throws a custom exception if the current reference time isn't present yet, and waits on a retrier. That way I can have this script just wait until high water arrival time runs, and pull that attribute column directly from the publish table.

The extra column could be argued to be messy, but I think this could help us in other ways as well (and I still like that it forces the documentation in one spot, even if the db tables can be a bit messy). And it should only be ~10 lines of code in the lambda function, and remove about 10 in the peak flow sql files.

[CoreyKrewson-NOAA]

Sounds good. I am a little confused on how these are dependent on each other. They both calculate the same metric (arrival time) but on different data (high water flow vs max flow)

[TylerSchrag-NOAA]

They both calculate and report 'High Water Return Time (hours)'... although peak flow still calls it 'Below Bank Return Time (hours)'... which we should make consistent. Maybe 'High Water End Time (hours)' is better for both.

[CoreyKrewson-NOAA]

ahh I see what you mean. Yeah that makes sense