Code

Lazy netcdf saves#5191

Code

Merged

lbdreyer merged 92 commits intoSciTools:mainSciTools/iris:mainfrom pp-mo:lazy_save_2pp-mo/iris:lazy_save_2Copy head branch name to clipboardApr 21, 2023

+1,700-331

Commits

Commits on Mar 10, 2023

• Basic functional lazy saving.

  pp-mo

  committed
  cd7fa42

  Copy full SHA for cd7fa42

• Simplify function signature which upsets Sphinx.

  pp-mo

  committed
  1f32800

  Copy full SHA for 1f32800

• Non-lazy saves return nothing.

  pp-mo

  committed
  e0f980f

  Copy full SHA for e0f980f

• Now fixed to enable use with process/distributed scheduling.

  pp-mo

  committed
  67b96cf

  Copy full SHA for 67b96cf

• Remove dask.utils.SerializableLock, which I think was a mistake.

  pp-mo

  committed
  8cdbc9b

  Copy full SHA for 8cdbc9b

• Make DefferedSaveWrapper use _thread_safe_nc.

  pp-mo

  committed
  8723f24

  Copy full SHA for 8723f24

• Fixes for non-lazy save.

  pp-mo

  committed
  d19a87f

  Copy full SHA for d19a87f

• Avoid saver error when no deferred writes.

  pp-mo

  committed
  45e0e60

  Copy full SHA for 45e0e60

• Reorganise locking code, ready for shareable locks.

  pp-mo

  committed
  6a83200

  Copy full SHA for 6a83200

• Remove optional usage of 'filelock' for lazy saves.

  pp-mo

  committed
  78a9346

  Copy full SHA for 78a9346

• Document dask-specific locking; implement differently for threads or distributed schedulers.

  pp-mo

  committed
  31b12f7

  Copy full SHA for 31b12f7

• Minor fix for unit-tests.

  pp-mo

  committed
  99b4f41

  Copy full SHA for 99b4f41

• Pin libnetcdf to avoid problems -- see SciTools#5187.

  pp-mo

  committed
  5d0a707

  Copy full SHA for 5d0a707

• Minor test fix.

  pp-mo

  committed
  431036f

  Copy full SHA for 431036f

Commits on Mar 13, 2023

• Move DeferredSaveWrapper into _thread_safe_nc; replicate the NetCDFDataProxy fix; use one lock per Saver; add extra up-scaled test

  pp-mo

  committed
  dc368d9

  Copy full SHA for dc368d9

Commits on Mar 16, 2023

• Update lib/iris/fileformats/netcdf/saver.py

  Show description for 6756a46
  

  pp-mo

  and

  bouweandela

  authored
  6756a46

  Copy full SHA for 6756a46

• Update lib/iris/fileformats/netcdf/_dask_locks.py

  Show description for 80b4b6c
  

  pp-mo

  and

  bouweandela

  authored
  80b4b6c

  Copy full SHA for 80b4b6c

• Update lib/iris/fileformats/netcdf/saver.py

  Show description for 78a8716
  

  pp-mo

  and

  bouweandela

  authored
  78a8716

  Copy full SHA for 78a8716

Commits on Mar 17, 2023

• Small rename + reformat.

  pp-mo

  committed
  47bb08b

  Copy full SHA for 47bb08b

Commits on Mar 18, 2023

• Remove Saver lazy option; all lazy saves are delayed; factor out fillvalue checks and make them delayable.

  pp-mo

  committed
  0ece09a

  Copy full SHA for 0ece09a

• Merge branch 'main' into lazy_save_2

  pp-mo

  authored
  940f544

  Copy full SHA for 940f544

Commits on Mar 20, 2023

• Merge remote-tracking branch 'upstream/main' into lazy_save_2

  pp-mo

  committed
  eb97130

  Copy full SHA for eb97130

• Repurposed 'test__FillValueMaskCheckAndStoreTarget' to 'test__data_fillvalue_check', since old class is gone.

  pp-mo

  committed
  4596081

  Copy full SHA for 4596081

• Disable (temporary) saver debug printouts.

  pp-mo

  committed
  ad49fbe

  Copy full SHA for ad49fbe

• Fix test problems; Saver automatically completes to preserve existing direct usage (which is public API).

  pp-mo

  committed
  b29c927

  Copy full SHA for b29c927

• Fix docstring error.

  pp-mo

  committed
  8f10281

  Copy full SHA for 8f10281

• Fix spurious error in old saver test.

  pp-mo

  committed
  6a564d9

  Copy full SHA for 6a564d9

• Fix Saver docstring.

  pp-mo

  committed
  2fb4d6c

  Copy full SHA for 2fb4d6c

• More robust exit for NetCDFWriteProxy operation.

  pp-mo

  committed
  c84bfdc

  Copy full SHA for c84bfdc

Commits on Mar 21, 2023

• Fix doctests by making the Saver example functional.

  pp-mo

  committed
  5b78085

  Copy full SHA for 5b78085

Commits on Mar 23, 2023

• Improve docstrings; unify terminology; simplify non-lazy save call.

  pp-mo

  committed
  478332e

  Copy full SHA for 478332e

Commits on Mar 27, 2023

• Moved netcdf cell-method handling into nc_load_rules.helpers, and various tests into more specific test folders.

  pp-mo

  committed
  34f154c

  Copy full SHA for 34f154c

• Merge branch 'latest' into lazy_save_2

  pp-mo

  committed
  d3744ba

  Copy full SHA for d3744ba

• Fix lockfiles and Makefile process.

  pp-mo

  committed
  9673ea0

  Copy full SHA for 9673ea0

• Add unit tests for routine _fillvalue_report().

  pp-mo

  committed
  bcbcbc8

  Copy full SHA for bcbcbc8

• Remove debug-only code.

  pp-mo

  committed
  05c04a1

  Copy full SHA for 05c04a1

Commits on Mar 28, 2023

• Added tests for what the save function does with the 'compute' keyword.

  pp-mo

  committed
  679ea47

  Copy full SHA for 679ea47

• Fix mock-specific problems, small tidy.

  pp-mo

  committed
  70ec9dd

  Copy full SHA for 70ec9dd

Commits on Mar 29, 2023

• Restructure hierarchy of tests.unit.fileformats.netcdf

  pp-mo

  committed
  28a4674

  Copy full SHA for 28a4674

• Tidy test docstrings.

  pp-mo

  committed
  67f4b2b

  Copy full SHA for 67f4b2b

• Correct test import.

  pp-mo

  committed
  ebec72f

  Copy full SHA for ebec72f

• Avoid incorrect checking of byte data, and a numpy deprecation warning.

  pp-mo

  committed
  1f5b904

  Copy full SHA for 1f5b904

• Alter parameter names to make test reports clearer.

  pp-mo

  committed
  5045c9f

  Copy full SHA for 5045c9f

Commits on Mar 30, 2023

• Test basic behaviour of _lazy_stream_data; make 'Saver._delayed_writes' private.

  pp-mo

  committed
  393407a

  Copy full SHA for 393407a

• Add integration tests, and distributed dependency.

  pp-mo

  committed
  518360b

  Copy full SHA for 518360b

Commits on Mar 31, 2023

• Docstring fixes.

  pp-mo

  committed
  5c9931f

  Copy full SHA for 5c9931f

Commits on Apr 4, 2023

• Documentation section and whatsnew entry.

  pp-mo

  committed
  7daee68

  Copy full SHA for 7daee68

• Merge branch 'main' into lazy_save_2

  pp-mo

  authored
  97474f9

  Copy full SHA for 97474f9

• Various fixes to whatsnew, docstrings and docs.

  pp-mo

  committed
  64c7251

  Copy full SHA for 64c7251

Commits on Apr 11, 2023

• Minor review changes, fix doctest.

  pp-mo

  committed
  75043f9

  Copy full SHA for 75043f9

• Arrange tests + results to organise by package-name alone.

  pp-mo

  committed
  445fbe2

  Copy full SHA for 445fbe2

• Review changes.

  pp-mo

  committed
  09cb22e

  Copy full SHA for 09cb22e

Commits on Apr 12, 2023

• Review changes.

  pp-mo

  committed
  3445f58

  Copy full SHA for 3445f58

Commits on Apr 13, 2023

• Enhance tests + debug.

  pp-mo

  committed
  cb1e1f7

  Copy full SHA for cb1e1f7

• Support scheduler type 'single-threaded'; allow retries on delayed-save test.

  pp-mo

  committed
  1c81cee

  Copy full SHA for 1c81cee

• Improve test.

  pp-mo

  committed
  370837b

  Copy full SHA for 370837b

• Adding a whatsnew entry for 5224 (SciTools#5234)

  Show description for 2f5f3c2
  

  HGWright

  authored and

  pp-mo

  committed
  2f5f3c2

  Copy full SHA for 2f5f3c2

• Replacing numpy legacy printing with array2string and remaking results for dependent tests

  

  HGWright

  authored and

  pp-mo

  committed
  a55c6f2

  Copy full SHA for a55c6f2

• adding a whatsnew entry

  

  HGWright

  authored and

  pp-mo

  committed
  4914e99

  Copy full SHA for 4914e99

• configure codecov

  

  HGWright

  authored and

  pp-mo

  committed
  bd642cd

  Copy full SHA for bd642cd

• remove results creation commit from blame

  

  HGWright

  authored and

  pp-mo

  committed
  bc5bdd1

  Copy full SHA for bc5bdd1

• fixing whatsnew entry

  

  HGWright

  authored and

  pp-mo

  committed
  301e59e

  Copy full SHA for 301e59e

• Bump scitools/workflows from 2023.04.1 to 2023.04.2 (SciTools#5236)

  Show description for 7b3044d
  

  dependabot[bot]

  authored and

  pp-mo

  committed
  7b3044d

  Copy full SHA for 7b3044d

• Use real array for data of of small netCDF variables. (SciTools#5229)

  Show description for 02f2b66
  pp-mo

  committed
  02f2b66

  Copy full SHA for 02f2b66

• Handle derived coordinates correctly in concatenate (SciTools#5096)

  Show description for a7e0689
  

  schlunma

  authored and

  pp-mo

  committed
  a7e0689

  Copy full SHA for a7e0689

• clarity on whatsnew entry contributors (SciTools#5240)

  

  bjlittle

  authored and

  pp-mo

  committed
  c4e8bbb

  Copy full SHA for c4e8bbb

• Modernize and simplify iris.analysis._Groupby (SciTools#5015)

  Show description for e6661b8
  

  2 peopleauthored and

  pp-mo

  committed
  e6661b8

  Copy full SHA for e6661b8

• Finalises Lazy Data documentation (SciTools#5137)

  Show description for afbdbbd
  

  3 peopleauthored and

  pp-mo

  committed
  afbdbbd

  Copy full SHA for afbdbbd

• Fixes to _discontiguity_in_bounds (attempt 2) (SciTools#4975)

  

  stephenworsley

  authored and

  pp-mo

  committed
  b8bb753

  Copy full SHA for b8bb753

• update ci locks location (SciTools#5228)

  

  bjlittle

  authored and

  pp-mo

  committed
  97cc149

  Copy full SHA for 97cc149

• Updated environment lockfiles (SciTools#5211)

  Show description for f14a321
  

  2 peopleauthored and

  pp-mo

  committed
  f14a321

  Copy full SHA for f14a321

• Increase retries.

  pp-mo

  committed
  f7a0b87

  Copy full SHA for f7a0b87

• Change debug to show which elements failed.

  pp-mo

  committed
  69ddd9d

  Copy full SHA for 69ddd9d

• update cf standard units (SciTools#5244)

  Show description for 8235d60
  

  2 peopleauthored and

  pp-mo

  committed
  8235d60

  Copy full SHA for 8235d60

• libnetcdf <4.9 pin (SciTools#5242)

  Show description for 724c6d2
  

  trexfeathers

  authored and

  pp-mo

  committed
  724c6d2

  Copy full SHA for 724c6d2

• Avoid possible same-file crossover between tests.

  pp-mo

  committed
  4f50dc7

  Copy full SHA for 4f50dc7

• Ensure all-different testfiles; load all vars lazy.

  pp-mo

  committed
  0da68cf

  Copy full SHA for 0da68cf

• Revert changes to testing framework.

  pp-mo

  committed
  e8b7bfd

  Copy full SHA for e8b7bfd

• Remove repeated line from requirements/py*.yml (?merge error), and re-fix lockfiles.

  pp-mo

  committed
  ad48caf

  Copy full SHA for ad48caf

• Revert some more debug changes.

  pp-mo

  committed
  291b587

  Copy full SHA for 291b587

• Merge branch 'latest' into lazy_save_2

  pp-mo

  committed
  b2260ef

  Copy full SHA for b2260ef

Commits on Apr 14, 2023

• Reorganise test for better code clarity.

  pp-mo

  committed
  33a7d86

  Copy full SHA for 33a7d86

• Use public 'Dataset.isopen()' instead of '._isopen'.

  pp-mo

  committed
  db6932d

  Copy full SHA for db6932d

• Create output files in unique temporary directories.

  pp-mo

  committed
  631e001

  Copy full SHA for 631e001

• Tests for fileformats.netcdf._dask_locks.

  pp-mo

  committed
  2869f97

  Copy full SHA for 2869f97

Commits on Apr 21, 2023

• Merge branch 'latest' into lazy_save_2

  pp-mo

  committed
  419727b

  Copy full SHA for 419727b

• Fix attribution names.

  pp-mo

  committed
  2f4458b

  Copy full SHA for 2f4458b

• Merge branch 'latest' into lazy_save_2

  pp-mo

  committed
  88b7a2a

  Copy full SHA for 88b7a2a

• Fixed new py311 lockfile.

  pp-mo

  committed
  98a20e7

  Copy full SHA for 98a20e7

• Fix typos spotted by codespell.

  pp-mo

  committed
  bbc1167

  Copy full SHA for bbc1167

• Add distributed test dep for python 3.11

  pp-mo

  committed
  ed38e43

  Copy full SHA for ed38e43

• Fix lockfile for python 3.11

  pp-mo

  committed
  54ec0f8

  Copy full SHA for 54ec0f8