beam-refactor reference recipe

.github/workflows/test-integration.yaml

@@ -0,0 +1,63 @@
+name: Integration tests
+
+on:
+  push:
+    branches: [ "beam-refactor" ]  # FIXME: change to default branch post-merge
+  pull_request:
+    branches: [ "beam-refactor" ]  # FIXME: change to default branch post-merge
+    types: [ opened, reopened, synchronize, labeled ]
+
+env:
+  PYTEST_ADDOPTS: "--color=yes"
+
+jobs:
+  prepare-env:
+    # run on:
+    #  - all pushes to specified branch(es)
+    #  - a PR was just labeled 'test-integration'
+    #  - a PR with 'test-integration' label was opened, reopened, or synchronized
+    if: |
+      github.event_name == 'push' ||
+      github.event.label.name == 'test-integration' ||
+      contains( github.event.pull_request.labels.*.name, 'test-integration')
+    uses: ./.github/workflows/prepare-env.yaml
+  integration-tests:
+    needs: prepare-env
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.9", "3.10"]
+        dependencies: ["releases-only", "upstream-dev"]
+    steps:
+      - uses: actions/checkout@v2
+
+      # generic steps to load env from cache
+      - name: 🎯 Set cache number
+        id: cache-number
+        # cache will last 3 days by default
+        run: echo CACHE_NUMBER=`expr $(date +'%j') / 3` >> $GITHUB_ENV
+      - name: 🎯 Set environment file
+        id: env-file
+        run: echo "env_file=ci/py${{ matrix.python-version }}.yml" >> $GITHUB_ENV
+      - uses: actions/cache@v2
+        name: 🗃 Loaded Cached environment
+        with:
+          path: /usr/share/miniconda3/envs/pangeo-forge-recipes
+          key: ${{ runner.os }}-conda-${{ matrix.python-version }}-${{ hashFiles( env.env_file ) }}-${{ matrix.dependencies }}-${{ env.CACHE_NUMBER }}
+        id: conda-cache
+      - name: 🤿  Bail out if no cache hit
+        if: steps.conda-cache.outputs.cache-hit != 'true'
+        run: false
+      - name: 🎯 Set path to include conda python
+        run: echo "/usr/share/miniconda3/envs/pangeo-forge-recipes/bin" >> $GITHUB_PATH
+
+      # custom testing steps unique to this workflow
+      - name: 🌈 Install pangeo-forge-recipes package
+        shell: bash -l {0}
+        run: |
+          python -m pip install --no-deps -e  .
+      - name: 🏄‍♂️ Run Tests
+        shell: bash -l {0}
+        run: |
+          pytest --timeout=600 tests-integration/ -v

.readthedocs.yml

@@ -3,8 +3,14 @@ version: 2
 sphinx:
   configuration: docs/conf.py
 
-# Optionally set the version of Python and requirements required to build your docs
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.9"
+
 python:
-  version: 3.8
   install:
+    # Install package too, so autodoc works
+    - method: pip
+      path: .
     - requirements: docs/requirements.txt

ci/py3.10.yml

@@ -7,7 +7,7 @@ dependencies:
   - apache-beam
   - black
   - boto3
-  - cfgrib<0.9.9.0
+  - cfgrib
   - cftime
   - codecov
   - dask
@@ -20,7 +20,7 @@ dependencies:
   - hdf5
   - intake
   - intake-xarray
-  - kerchunk>=0.0.6
+  - kerchunk>=0.1.1
   - lxml    # Optional dep of pydap
   - matplotlib  # needed for building tutorial notebooks
   - netcdf4

ci/py3.9.yml

@@ -7,7 +7,7 @@ dependencies:
   - apache-beam
   - black
   - boto3
-  - cfgrib<0.9.9.0
+  - cfgrib
   - cftime
   - codecov
   - dask
@@ -20,7 +20,7 @@ dependencies:
   - hdf5
   - intake
   - intake-xarray
-  - kerchunk>=0.0.6
+  - kerchunk>=0.1.1
   - lxml    # Optional dep of pydap
   - matplotlib  # needed for building tutorial notebooks
   - netcdf4

docs/pangeo_forge_recipes/recipe_user_guide/recipes.md

@@ -93,13 +93,37 @@ All recipes need a place to store the target dataset. Refer to {doc}`storage` fo
 Once your recipe is defined and has its storage targets assigned, you're ready to
 move on to {doc}`execution`.
 
-### HDF Reference Recipes
+### Reference Recipes
 
-Like the Xarray to Zarr recipes, this category allows us to more efficiently access data from a bunch of NetCDF / HDF files.
-However, such a recipe does not actually copy the original source data.
-Instead, it generates metadata files which reference and index the original data, allowing it to be accessed more quickly and easily.
-For more background, see [this blog post](https://medium.com/pangeo/fake-it-until-you-make-it-reading-goes-netcdf4-data-on-aws-s3-as-zarr-for-rapid-data-access-61e33f8fe685).
+Like the Xarray to Zarr recipes, this category of recipes allows us to efficiently access data from a
+collection of source files. Unlike the standard Zarr recipes, these reference recipes utilize
+[kerchunk](https://fsspec.github.io/kerchunk/) to generate metadata files which reference and index the
+original data, allowing it to be accessed more quickly and easily, without duplicating it.
 
-There is currently one tutorial for this recipe:
+Whereas the standard Zarr recipe creates a copy of the original dataset in the Zarr format, the
+kerchunk-based reference recipe does not copy the data and instead creates a Kerchunk mapping, which
+allows archival formats (including NetCDF, GRIB2, etc.) to be read as if they were Zarr datasets.
+More details about how Kerchunk works can be found in the
+[kerchunk docs](https://fsspec.github.io/kerchunk/detail.html)
+and [this blog post](https://medium.com/pangeo/fake-it-until-you-make-it-reading-goes-netcdf4-data-on-aws-s3-as-zarr-for-rapid-data-access-61e33f8fe685).
+
+There are currently two tutorials for reference recipes:
 
 - {doc}`../tutorials/hdf_reference/reference_cmip6`
+- {doc}`../tutorials/grib_reference/reference_HRRR`
+
+When choosing whether to create a reference recipe, it is important to consider questions such as:
+
+_**Where are the archival (i.e. source) files for this dataset currently stored?**_ If the original data
+are not already in the cloud (or some other high-bandwidth storage device, such as an on-prem data
+center), the performance benefits of using a reference recipe may be limited, because network speeds
+to access the original data will constrain I/O throughput.
+
+_**Does this dataset require preprocessing?**_ With reference recipes, modification of the underlying
+data is not possible. For example, the chunking schema of a dataset cannot be modified with Kerchunk, so
+you are limited to the chunk schema of the archival data. If you need to optimize your datasets chunking
+schema for space or time, the standard Zarr recipe is the only option. While you cannot modify chunking
+in a reference recipe, changes in the metadata (attributes, encoding, etc.) can be applied.
+
+These caveats aside, for archival data stored on highly-throughput storage devices, for which
+preprocessing is not required, reference recipes are an ideal and storage-efficient option.