Code

Monkey patch iris#68

Code

Closed

znicholls wants to merge 40 commits intomasterznicholls/netcdf-scm:masterfrom parallel-crunchingznicholls/netcdf-scm:parallel-crunchingCopy head branch name to clipboard

+249-169

Commits

Commits on Jul 12, 2019

• Reset scratch process script

  znicholls

  committed
  358fce2

  Copy full SHA for 358fce2

• Put parallel loop in

  znicholls

  committed
  2235793

  Copy full SHA for 2235793

• Add comparison of data/marble crunching

  znicholls

  committed
  d9f5e1d

  Copy full SHA for d9f5e1d

• Back to serial for testing purposes

  znicholls

  committed
  83d418f

  Copy full SHA for 83d418f

• Make comparisons a fair test

  znicholls

  committed
  78dd8f8

  Copy full SHA for 78dd8f8

• Make better scratch script

  znicholls

  committed
  d25b518

  Copy full SHA for d25b518

• Print output in scratch

  znicholls

  committed
  c90da9b

  Copy full SHA for c90da9b

• Monkey patch iris netCDF loading

  

  lewisjared

  authored and

  znicholls

  committed
  730f7e6

  Copy full SHA for 730f7e6

• Decide on dask tricks for crunching

  znicholls

  committed
  684fa53

  Copy full SHA for 684fa53

• Update scratch for full IPSL crunching

  znicholls

  committed
  640a63a

  Copy full SHA for 640a63a

• Optimise as far as I can without parallel

  znicholls

  committed
  a064e24

  Copy full SHA for a064e24

• Format

  znicholls

  committed
  4e932f7

  Copy full SHA for 4e932f7

• Add useful benchmarking script

  znicholls

  committed
  9d95d4c

  Copy full SHA for 9d95d4c

• Back to asynchronous reads

  znicholls

  committed
  c394ceb

  Copy full SHA for c394ceb

• Turn off saving to do fair comparison with parallel loop

  znicholls

  committed
  2b6a686

  Copy full SHA for 2b6a686

• Put parallel loop back in

  znicholls

  committed
  3bca515

  Copy full SHA for 3bca515

• Add tougher scratch processing script

  znicholls

  committed
  473a0b7

  Copy full SHA for 473a0b7

• Back to serial processing

  znicholls

  committed
  ca87e2e

  Copy full SHA for ca87e2e

• Turn output writing back on

  znicholls

  committed
  3dcbce4

  Copy full SHA for 3dcbce4

• Update scripts

  znicholls

  committed
  73fedfc

  Copy full SHA for 73fedfc

• Rebase and format

  znicholls

  committed
  d24c18b

  Copy full SHA for d24c18b

• CHANGELOG and tidy up

  znicholls

  committed
  88da7de

  Copy full SHA for 88da7de

• Update scripts/scratch-process.sh

  znicholls

  authored
  5e419f2

  Copy full SHA for 5e419f2

• Add netCDF-SCM version output into processing speed script

  znicholls

  committed
  0828a6b

  Copy full SHA for 0828a6b

• Turn save off

  znicholls

  committed
  484cff3

  Copy full SHA for 484cff3

• Try synchronous output again

  znicholls

  committed
  3a1186d

  Copy full SHA for 3a1186d

• Back to previous setup

  znicholls

  committed
  272b48b

  Copy full SHA for 272b48b

• Fix variable typo

  znicholls

  committed
  af262ae

  Copy full SHA for af262ae

Commits on Jul 13, 2019

• Make processing use map

  znicholls

  committed
  bd9fd4a

  Copy full SHA for bd9fd4a

• Ugly parallel crunching

  znicholls

  committed
  7403f3f

  Copy full SHA for 7403f3f

• Tidy up

  znicholls

  committed
  1235322

  Copy full SHA for 1235322

• Refactor out masking functionality

  znicholls

  authored
  1e628b0

  Copy full SHA for 1e628b0

• Only get masks once

  znicholls

  committed
  00bb3a7

  Copy full SHA for 00bb3a7

• Setup dynamic parallel crunching

  znicholls

  committed
  9044a87

  Copy full SHA for 9044a87

Commits on Jul 14, 2019

• Write speed test to specific folder

  znicholls

  committed
  636f935

  Copy full SHA for 636f935

• Remove speed output files

  znicholls

  committed
  92024ad

  Copy full SHA for 92024ad

• Add upper crunching limit too

  znicholls

  committed
  ffb708e

  Copy full SHA for ffb708e

Commits on Jul 15, 2019

• Go back to serial crunching

  znicholls

  committed
  6f9c2b8

  Copy full SHA for 6f9c2b8

• Make crunching run in parallel

  znicholls

  committed
  74955ea

  Copy full SHA for 74955ea

• Add tqdm progressbar

  znicholls

  committed
  288c16d

  Copy full SHA for 288c16d