Update datasets to use API

[anthayes92]

Context:
Datasets are currently downloaded by hitting the S3 bucket (via CloudFront) where the .h5 files are stored. This PR updates the way datasets are downloaded by directing download requests to the Software Cloud managed Datasets Service.

Description of the Change:
The qml.data.load function now queries the Datasets API in order to download datasets. Additionally:

• Updates parameter formatting for the new API
• Removes URL escaping functionality since the download URLS retrieved from the API are already escaped

Benefits:

• Removes almost all dependency on the foldermap and data_struct files, alleviating the need to manually manage them.

⚠️ There is a lingering foldermap dependency for the list_datasets() function which will likely be removed in 0.40.0

• Facilitates tracking dataset downloads for analytics.

Possible Drawbacks:

• Introduces network dependency for accessing the external API.

Related GitHub Issues:

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.38%. Comparing base (43cb979) to head (ef8c82f).
Report is 321 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6126      +/-   ##
==========================================
- Coverage   99.38%   99.38%   -0.01%     
==========================================
  Files         450      451       +1     
  Lines       42619    42670      +51     
==========================================
+ Hits        42359    42408      +49     
- Misses        260      262       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[astralcai]

What's the decision regarding the old list_datasets? If we're deprecating it, it should raise a deprecation warning, and both the changelog and docs/development/deprecations.rst should mention this deprecation.

[anthayes92]

What's the decision regarding the old list_datasets? If we're deprecating it, it should raise a deprecation warning, and both the changelog and docs/development/deprecations.rst should mention this deprecation.

We'll be maintaining list_datasets() with the existing functionality in this release. We currently plan to revisit this in 0.40.0 to remove the dependency on the foldermap, the list_datasets() will not be deprecated regardless.

[astralcai]

LGTM!

[DSGuala]

Thanks for pushing the recent updates! Defaults are now working nicely 👍
Some additional QA feedback from testing:

1. This code gives an obscure error message: [ds] = qml.data.load('qchem', molname='fail')
ValueError: max_workers must be greater than 0
   Previously we would have this error message:
   ValueError: molname value(s) ['fail'] are not available. Available values are: ['BH3', 'BeH2', 'C2',
2. Not a high priority, but there is some difference in how we store the files now (new folder structure and file names). Users may have to redownload their existing datasets.
   old:
   
   vs new:
   
3. The 'full' keyword is no longer valid:
   [ds] = qml.data.load('qchem', molname='H2', basis='STO-3G', bondlength='full')
TypeError: Object of type ParamArg is not JSON serializable

[anthayes92]

Thanks for testing this out and leaving this useful feedback @DSGuala! To address your points:

This code gives an obscure error message: [ds] = qml.data.load('qchem', molname='fail')
ValueError: max_workers must be greater than 0
Previously we would have this error message:
ValueError: molname value(s) ['fail'] are not available. Available values are: ['BH3', 'BeH2', 'C2',

I've updated this to give a general error message:

ValueError: No datasets exist for the provided configuration.
Please check the available datasets by using the ``qml.data.list_datasets()`` function.

Since retrieving the specific values available for each parameter presents a similar issue to the problem with updating list_datasets(), an intermediate solution for now is to point the user to the existing list_datasets() function to check the available datasets.

Not a high priority, but there is some difference in how we store the files now (new folder structure and file names). Users may have to redownload their existing datasets.

This could be quite involved given the new formatting of the download IDs. Users can also specify these paths manually if desired. Given this is lower priority and considering the current time constraints this could be updated in a followup if we still want it.

The 'full' keyword is no longer valid:
[ds] = qml.data.load('qchem', molname='H2', basis='STO-3G', bondlength='full')
TypeError: Object of type ParamArg is not JSON serializable

Good catch, this was overlooked. A fix has been added now thanks!

[brownj85]

Looks good!

[DSGuala]

Looks good to me! Thanks for addressing everything quickly 🙏