snapshot.fetch() errors since Zenodo migration

[khaeru]

On 2023-10-13, Zenono migrated to a new platform/backend. Since that time, test_snapshot.py::test_load[0] has failed. The symptoms vary:

1. Here, pooch appears to get an response from Zenodo, but the response appears to be malformed, resulting in KeyError: 'key' within pooch. When this occurs, the Zenodo record URL also appears accessible through the browser.

2. At other times, Zenodo record URLs (in particular https://zenodo.org/record/5793870) appear not to resolve. In a browser, this can manifest as:

   • 504 Gateway Time-out, displayed as a plain HTML page.
   • 503 Application Not Available, displayed as an error page that identifies a CERN "OKD" cluster.

   …in either case, this results in pooch raising ValueError: Archive with doi:10.5281/zenodo.5793870 not found (see https://zenodo.org/record/5793870). Is the DOI correct?

Currently:

• https://zenodo.org has a header message stating "Oct 15 08:30 UTC: We are continuing to work on resolving identified issues" without any link or explanation of what those issues are.
• The "Status" link from the Zenodo footer leads to https://stats.uptimerobot.com/vlYOVuWgM/, which displays "Files upload/download: Down" and "Search: Down".

Some combination of the following seems likely. Respectively with the symptoms above:

1. Either:
   • A. Zenodo is operational, but the migration has resulted in unintended/unadvertised changes in its API responses that cause the pooch error. (Zenodo has What's changed? and What's new? pages describing the migration, but they don't indicate any API changes. Its API documentation doesn't indicate a new version.)
   • B. Zenodo is operational, and as part of its migration has well-defined changes to its API; pooch has not adapted to these. Compare Invenio RDM support fatiando/pooch#350, which suggests the new platform "InvenioRDM" has a distinct API that is not yet supported by pooch.
2. Zenodo is not operational.

Mitigation

• File this issue and KeyError: 'key' in ZenodoRepository.download_url() after Zenodo migration fatiando/pooch#371 upstream.
• Temporarily xfail the test.

Fix

• Track resolution at Zenodo and upstream.
• Un-xfail the test and confirm.

[khaeru]

Pooch 1.8.0 was released with a fix for the upstream bug; today, the scheduled pytest run gave XPASS for the test that was previously FAIL/XFAIL-ing.

So this is likely fixed, and the last step is to remove the XFAIL mark.

[glatterf42]

The test in question is passing again, even though other tests on #137 are still failing at the moment. Thus, I'm going to close this issue.