Code

beam-refactor reference recipe#486

Code

Merged

cisaacstern merged 74 commits intopangeo-forge:beam-refactorpangeo-forge/pangeo-forge-recipes:beam-refactorfrom norlandrhagen:beam-kerchunknorlandrhagen/pangeo-forge-recipes:beam-kerchunkCopy head branch name to clipboardApr 28, 2023

+2,772-3,000

Commits

Commits on Feb 2, 2023

• start of beam-refactor ref recipe port

  norlandrhagen

  committed
  002600f

  Copy full SHA for 002600f

• progress on combining reference files in MultiZarrToZarr

  norlandrhagen

  committed
  c46bbe8

  Copy full SHA for c46bbe8

Commits on Feb 7, 2023

• progress on combine refs

  norlandrhagen

  committed
  92333db

  Copy full SHA for 92333db

• added path for parquet backend

  norlandrhagen

  committed
  6942407

  Copy full SHA for 6942407

• progress on WriteCombinedReference, CombineReferences, DropKeys

  norlandrhagen

  committed
  ddb8ea6

  Copy full SHA for ddb8ea6

Commits on Feb 13, 2023

• Update pangeo_forge_recipes/transforms.py

  Show description for 792484a
  

  norlandrhagen

  and

  keewis

  authored
  792484a

  Copy full SHA for 792484a

• Update pangeo_forge_recipes/transforms.py

  Show description for 8fd4c37
  

  norlandrhagen

  and

  keewis

  authored
  8fd4c37

  Copy full SHA for 8fd4c37

• update to combine_refs & OpenWithKerchunk dtypes

  norlandrhagen

  committed
  4df8e90

  Copy full SHA for 4df8e90

Commits on Mar 29, 2023

• Merge branch 'beam-refactor' into beam-kerchunk

  norlandrhagen

  committed
  a7976de

  Copy full SHA for a7976de

• added testing script

  norlandrhagen

  committed
  f7114fc

  Copy full SHA for f7114fc

• back to error in WriteCombinedReference

  norlandrhagen

  committed
  8c9d0ca

  Copy full SHA for 8c9d0ca

Commits on Apr 10, 2023

• added DropKeys into pipeline and a E2E test for kerchunk recipe

  norlandrhagen

  committed
  d1200e8

  Copy full SHA for d1200e8

• use CombineFn for kerchunk MultiZarrToZarr

  cisaacstern

  committed
  5837ff4

  Copy full SHA for 5837ff4

Commits on Apr 11, 2023

• kerchunk>=0.1.0 fixes references test for netcdf4

  cisaacstern

  committed
  2075752

  Copy full SHA for 2075752

• fix pre-commit issues

  cisaacstern

  committed
  cd68f6e

  Copy full SHA for cd68f6e

• note docstring fixme in open_with_kerchunk

  cisaacstern

  committed
  06f41a6

  Copy full SHA for 06f41a6

• move write_combine_references to writers module

  cisaacstern

  committed
  b054edf

  Copy full SHA for b054edf

• NetCDF3ToZarr requires filename as str

  cisaacstern

  committed
  f75418f

  Copy full SHA for f75418f

• try to write to parquet

  cisaacstern

  committed
  d693a8d

  Copy full SHA for d693a8d

• give up on parquet for now

  cisaacstern

  committed
  b7a1544

  Copy full SHA for b7a1544

• add ZarrWriterMixin

  cisaacstern

  committed
  f19f966

  Copy full SHA for f19f966

• remove top-level beam_test.py

  cisaacstern

  committed
  1185037

  Copy full SHA for 1185037

• revert terraclimate tutorial to upstream

  cisaacstern

  committed
  1f95bf0

  Copy full SHA for 1f95bf0

Commits on Apr 12, 2023

• Merge remote-tracking branch 'origin/beam-refactor' into beam-kerchunk

  cisaacstern

  committed
  bf95dfb

  Copy full SHA for bf95dfb

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 25a4fba
  pre-commit-ci[bot]

  committed
  25a4fba

  Copy full SHA for 25a4fba

• updates to transforms & combiners as well as cmip6 example ref recipe

  norlandrhagen

  committed
  3b66821

  Copy full SHA for 3b66821

• Merge branch 'beam-kerchunk' of https://github.com/norlandrhagen/pangeo-forge-recipes into beam-kerchunk

  norlandrhagen

  committed
  865e864

  Copy full SHA for 865e864

• removed file pattern truncation

  norlandrhagen

  committed
  027aca3

  Copy full SHA for 027aca3

• reference writer, use target.fs.open and make fname an option

  cisaacstern

  committed
  a428265

  Copy full SHA for a428265

• updated cmip6 ref tutorial

  norlandrhagen

  committed
  29d0162

  Copy full SHA for 29d0162

• added context in recipe type selection

  norlandrhagen

  committed
  970ebe2

  Copy full SHA for 970ebe2

• updated docstring in openers/open_with_kerchunk

  norlandrhagen

  committed
  0ef13db

  Copy full SHA for 0ef13db

• updated scan_grib reference opener with filename

  norlandrhagen

  committed
  8c7dd6c

  Copy full SHA for 8c7dd6c

• GRIB2 based reference tutorial [WIP]

  norlandrhagen

  committed
  23874b8

  Copy full SHA for 23874b8

Commits on Apr 13, 2023

• add grib test based on kerchunk grib test

  cisaacstern

  committed
  95ed34b

  Copy full SHA for 95ed34b

Commits on Apr 14, 2023

• start of kerchunk unit testing

  norlandrhagen

  committed
  d2c5121

  Copy full SHA for d2c5121

• latest version of grib/reference notebook failing due to path issue

  norlandrhagen

  committed
  8244125

  Copy full SHA for 8244125

Commits on Apr 18, 2023

• updated python->python3 for start_http_server

  norlandrhagen

  committed
  1382f1b

  Copy full SHA for 1382f1b

• added remote protocol in open_with_kerchunk

  norlandrhagen

  committed
  b2123ca

  Copy full SHA for b2123ca

• updates to ref/grib tutorial

  norlandrhagen

  committed
  74c3ce9

  Copy full SHA for 74c3ce9

• added tests for openwithkerchunk

  norlandrhagen

  committed
  3694823

  Copy full SHA for 3694823

Commits on Apr 19, 2023

• added pass for zarr filetype in test_OPenWithKerchunk_direct

  norlandrhagen

  committed
  9a337ee

  Copy full SHA for 9a337ee

• updated kerchunk for grib2 fix

  norlandrhagen

  committed
  0c12f4a

  Copy full SHA for 0c12f4a

Commits on Apr 21, 2023

• start of test test_inline_threshold - should be moved to test_openers

  norlandrhagen

  committed
  e6037d0

  Copy full SHA for e6037d0

• added missing inline_threshold arg

  norlandrhagen

  committed
  46da58d

  Copy full SHA for 46da58d

Commits on Apr 24, 2023

• moved inline_thresh test to test_openers & shared fixtures to conftest

  norlandrhagen

  committed
  fcc6de3

  Copy full SHA for fcc6de3

• factor out url_or_file_obj preprocessor into standalone func

  cisaacstern

  committed
  555be04

  Copy full SHA for 555be04

Commits on Apr 25, 2023

• grib integration test first commit

  cisaacstern

  committed
  f1ff239

  Copy full SHA for f1ff239

• test_grib integration cont

  cisaacstern

  committed
  e542c3d

  Copy full SHA for e542c3d

Commits on Apr 26, 2023

• get multi message grib test to pass

  cisaacstern

  committed
  0005138

  Copy full SHA for 0005138

• make drop_keys default behavior in OpenWithKerchunk

  cisaacstern

  committed
  51f423b

  Copy full SHA for 51f423b

• fix tests for changes in kerchunk opener

  cisaacstern

  committed
  b0d1357

  Copy full SHA for b0d1357

• if url includes remote_protocol, don't re-add it

  cisaacstern

  committed
  216f91d

  Copy full SHA for 216f91d

• rename test_grib test for specificity

  cisaacstern

  committed
  07262eb

  Copy full SHA for 07262eb

Commits on Apr 27, 2023

• add translate and maybe_eager_combine methods to mzz combinefn

  cisaacstern

  committed
  bb47200

  Copy full SHA for bb47200

• pass eager_combine param in first grib integration test

  cisaacstern

  committed
  f846885

  Copy full SHA for f846885

• revert methods on mzz combiner, use 'precombine_inputs' as param name

  cisaacstern

  committed
  4d79800

  Copy full SHA for 4d79800

• fix first hrrr test with precombine_inputs param name

  cisaacstern

  committed
  e1c9373

  Copy full SHA for e1c9373

• finishing touches on second hrrr integration test

  cisaacstern

  committed
  a2f979f

  Copy full SHA for a2f979f

• fix combinerefs test for change in combinerefs api

  cisaacstern

  committed
  e2ed607

  Copy full SHA for e2ed607

• Merge remote-tracking branch 'origin/beam-refactor' into beam-kerchunk

  cisaacstern

  committed
  537d964

  Copy full SHA for 537d964

• add integration test workflow

  cisaacstern

  committed
  614b55a

  Copy full SHA for 614b55a

• in integration tests, move if block to prepare-env job

  cisaacstern

  committed
  d3f8ef0

  Copy full SHA for d3f8ef0

Commits on Apr 28, 2023

• use kerchunk_open_kwargs instead of explicit kws on kerchunk opener

  cisaacstern

  committed
  f5a9a78

  Copy full SHA for f5a9a78

• fix docstrings + arg order in transforms

  cisaacstern

  committed
  353ea4a

  Copy full SHA for 353ea4a

• add docstring to pythia hrrr integration test

  cisaacstern

  committed
  fe0010e

  Copy full SHA for fe0010e

• move narrative comments on reference recipes to user guide section

  cisaacstern

  committed
  e37da6a

  Copy full SHA for e37da6a

• upgrade myst-nb to try to fix docs

  cisaacstern

  committed
  6e27e8f

  Copy full SHA for 6e27e8f

• upgrade myst-parser to match myst-nb upgrade

  cisaacstern

  committed
  361437b

  Copy full SHA for 361437b

• upgrade sphinx to fix docs build

  cisaacstern

  committed
  9dcb30a

  Copy full SHA for 9dcb30a

• readthedocs use py39

  cisaacstern

  committed
  16e6790

  Copy full SHA for 16e6790

• revert docs reqs changes

  cisaacstern

  committed
  f1ec41b

  Copy full SHA for f1ec41b

• update ref tutorials for the latest reference api

  cisaacstern

  committed
  6d0d697

  Copy full SHA for 6d0d697

• update ref tutorials for the latest reference api

  cisaacstern

  committed
  8e6deef

  Copy full SHA for 8e6deef