-
Notifications
You must be signed in to change notification settings - Fork 303
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Delete kdtree after saving cache #1227
Conversation
Codecov Report
@@ Coverage Diff @@
## master #1227 +/- ##
=======================================
Coverage 89.77% 89.77%
=======================================
Files 202 202
Lines 30031 30034 +3
=======================================
+ Hits 26961 26964 +3
Misses 3070 3070
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome! Nice catch.
I'm looking at this code in pyresample and I'm a little confused why we set it to a instance attribute anyway. It gets used in the same function that it is created in. Oh maybe dask needs a reference to it to keep it in memory.
One quick question: doesn't this still require computing the indexes on a local threaded scheduler and then after things are cached we are allowed to go distributed? This means you'd have to switch schedulers between computations, doesn't it?
The computations appear to happen automatically in the local thread when saving the cache, so doesn't need manual adjustment for the scheduler. At least with the currend Dask version 😬 |
This simple change (delete the kdtree after saving the cache) makes it possible to use
dask.distributed
withnearest
resampling. The only requirement is thatcache_dir
needs to be used, so this will not be very useful for other than geostationary satellites.flake8 satpy