A new diffusive_utils file without features designed only for refactored hydrofabric

[kumdonoaa]

The current diffusive_utils.py includes features utilizing refactored hydrofabric that was originally developed for removing short stream segments of NHDv2.0 hydrofabric of NWMv3.0. The removal was mainly for reducing routing compute time and possible numerical instability. The development of refactored hydrofabric wasn't completed and was stopped. Now the nextgen HYfeature hydrofabric replaces the refactored hydrofabric, so features related to the refactored hydrofabric aren't needed any more. So the cleaned diffusive_utils_v02.py is created to be used for both NHDv2.0 hydrofabric and HYfeature hydrofabric. More features to come in this file for HYfeature pretty soon.

Additions

Removals

Changes

Testing

Screenshots

Notes

Todos

Checklist

• PR has an informative and human-readable title
• Changes are limited to a single goal (no scope creep)
• Code can be automatically merged (no conflicts)
• Code follows project standards (link if applicable)
• Passes all existing automated tests
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future todos are captured in comments
• Visually tested in supported browsers and devices (see checklist below 👇)
• Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)
• Reviewers requested with the Reviewers tool ➡️

Testing checklist

Target Environment support

• Windows
• Linux
• Browser

Accessibility

• Keyboard friendly
• Screen reader friendly

Other

• Is useable without CSS
• Is useable without JS
• Flexible from small to large screens
• No linting errors or warnings
• JavaScript tests are passing

[shorvath-noaa]

This ran fine for me. While you're updating this file, do you think we should fix this warning message?:

t-route/src/troute-routing/troute/routing/diffusive_utils_v02.py:539: PerformanceWarning: DataFrame is highly fragmented. This is usually the result of calling "frame.insert" many times, which has poor performance. Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use "newframe = frame.copy()" usgs_df_complete.insert(i, timestamps[i], -4444.0*np.ones(len(usgs_df)), allow_duplicates=False)

[kumdonoaa]

Yes. Sean. please take a try if you have time after this PR is merged, I think.

[AminTorabi-NOAA]

I have a fix for it: In line 536 instead of for loop replace it with this

missing_timestamps = [ts for ts in timestamps if ts not in usgs_df.columns]
    
        missing_data = pd.DataFrame(-4444.0*np.ones((len(usgs_df), len(missing_timestamps))), 
                                    columns=missing_timestamps, 
                                    index=usgs_df.index)
usgs_df_complete = pd.concat([usgs_df_complete, missing_data], axis=1)`

[shorvath-noaa]

@kumdonoaa , @AminTorabi-NOAA 's fix looks good. Might want to add another line after to make sure columns are in the correct order. Something like:
usgs_df_complete = usgs_df_complete[timestamps]

[AminTorabi-NOAA]

Instead of for loop because we keep adding column to dataframe it slow it down and give warning. This should solve the issue

        missing_timestamps = [ts for ts in timestamps if ts not in usgs_df.columns]
    
  
        missing_data = pd.DataFrame(-4444.0*np.ones((len(usgs_df), len(missing_timestamps))), 
                                    columns=missing_timestamps, 
                                    index=usgs_df.index)
        
       
        usgs_df_complete = pd.concat([usgs_df_complete, missing_data], axis=1)

[kumdonoaa]

@AminTorabi-NOAA @shorvath-noaa Do these lines replace existing lines 537~539?:
https://github.com/kumdonoaa/t-route/blob/4c56cc615b49f6cc75ede3c6b8879689940bee9a/src/troute-routing/troute/routing/diffusive_utils_v02.py#L537-L539

[AminTorabi-NOAA]

Yes. As Sean suggested you can add usgs_df_complete = usgs_df_complete[timestamps] at the end of those lines too

[JurgenZach-NOAA]

I tested it, and it works.