Port "Pangeo Forge Recipes" documentation section to use beam-refactor approach

[derekocallaghan]

The current documentation (#445) describes the Beam approach in the Introduction Tutorial section. The Pangeo Forge Recipes section will also need to be ported to replace e.g. XarrayZarrRecipe references with the recommended Beam equivalent.

I guess the final API is still a work in progress (e.g. see #473). If it's useful, I can add a new/update an existing Recipe Tutorial as this may help settle the approach recommended to recipe authors. We could the update the Recipes User Guide, API Reference etc afterwards.

I have a local CCMP recipe that includes some custom processing (#453), which I can use to create a new recipe tutorial.

The objective is to have an accurate (if reduced) version of the documentation by Feb 26th 2023. The following sections will be ported:

• Recipes User Guide (Port Recipes User Guide to beam-refactor #499)

• Recipe Tutorials:

  • Xarray-to-Zarr Sequential Recipe: NOAA OISST notebook (Port tutorial notebooks to use Beam #487)
  • NetCDF Zarr Sequential Recipe: CMIP6 (Port tutorial notebooks to use Beam #487)
  • NetCDF Zarr Multi-Variable Sequential Recipe: NOAA World Ocean Atlas notebook (Port tutorial notebooks to use Beam #487)
  • Complex NetCDF to Zarr Recipe: TerraClimate (Port of Terraclimate example to beam-refactor #490)
  • NARR: Subsetting and OPeNDAP (Update opendap tutorial #500)
  • HDF Reference Recipe for CMIP6 (beam-refactor reference recipe #486)

• API Reference: check docstrings are accurate

• Development

  • No Beam-related changes required.

[norlandrhagen]

Hey there @derekocallaghan, I would love to help out on porting over some of the recipe tutorials. Do you have any you prefer to work on or have already started on?

[derekocallaghan]

Hi @norlandrhagen, I've just created #487 that contains an initial port of the NetCDF Zarr Multi-Variable Sequential Recipe: NOAA World Ocean Atlas notebook to use the beam-refactor approach. Although it contains references to the Xarray-to-Zarr Sequential Recipe: NOAA OISST notebook, I haven't ported this yet as it contains functionality which may no longer be provided by the beam-refactor branch. I'm going to look at this next.

In the meantime, the ported NOAA World Ocean Atlas notebook in #487 may be useful as an example for porting other tutorial notebooks. There are currently five tutorial notebooks, so if you wanted to look at any of the following that'd be great:

• CMIP6
• TerraClimate
• NARR: Subsetting and OPeNDAP

It's possible these may take a number of iterations depending on API changes. I guess we can coordinate in this which notebooks are being worked on to avoid duplication.

[derekocallaghan]

Hi @rabernat, I'm looking into porting the Xarray-to-Zarr Sequential Recipe: NOAA OISST notebook to use the Beam approach. Some of the steps are currently no longer fully relevant as they use functionality from pangeo_forge_recipes.recipes.xarray_zarr such as open_chunk() e.g.:

• Step 4: Play with the recipe
• Step 5: Create storage targets
• Step 6: Examine some chunks

Do we still want to have steps similar to these in the Beam version of the notebook, or replace them with steps that look at the output Zarr instead?

[cisaacstern]

Per conversation today during Coordination meeting, we've agreed to cut out the sections referenced above, which are no longer relevant in the context of beam-refactor.

[norlandrhagen]

@derekocallaghan here is the TerraClimate recipe.

[derekocallaghan]

An initial port of the Xarray-to-Zarr Sequential Recipe: NOAA OISST notebook has been created (#487).

[derekocallaghan]

As there are a few sections to port, I've added a task list to this issue description to coordinate activity.

[derekocallaghan]

I'm currently updating the remaining documentation to use the updated content from the tutorial notebooks (#487).

[cisaacstern]

Notes from Coordination Meeting today:

• Terraclimate blocked by open bug report (@cisaacstern link after meeting)
• Open separate issue for migrating tutorials from notebooks -> scripts (with CI testing)

Note: Latter is not a release blocker.

[derekocallaghan]

Notes from Coordination Meeting today:

* Terraclimate blocked by open bug report (@cisaacstern link after meeting)

* Open separate issue for migrating tutorials from notebooks -> scripts (with CI testing)

Note: Latter is not a release blocker.

Thanks @cisaacstern, apologies I couldn't make the last couple of meetings. I can help out with some of the notebook -> script migration.

[cisaacstern]

#490 is now in! I'll take one more spin through the docs this afternoon, and then merge beam-refactor 🚀 .

[cisaacstern]

Having skimmed the docs, my conclusion is that while there are certainly many opportunities for clarification and greater depth, there are no glaring inaccuracies, so I'm going to call this issue closed, and move on to releasing beam-refactor.