kedro-datasets: Datasets accept non-primitive parameters in the __init__

[ElenaKhaustova]

Description

Catalog to config solution is based on extracting dataset __init__ parameters and their values passed when creating a dataset.

The solution does not cover the case when non-primitive parameters are accepted in the dataset constructor:

• Connection (from google.oauth2.credentials import Credentials) - https://docs.kedro.org/projects/kedro-datasets/en/kedro-datasets-5.1.0/_modules/kedro_datasets/pandas/gbq_dataset.html#GBQQueryDataset
• type[AbstractDataset] - https://docs.kedro.org/projects/kedro-datasets/en/kedro-datasets-
  5.1.0/_modules/kedro_datasets/partitions/incremental_dataset.html#IncrementalDataset
• add other (in progress)

Possible Implementation

(Preferable solution) Force datasets to only have static, primitive parameters in the __init__: kedro-org/kedro#4329 (comment)

Possible Alternatives

Extend parent AbstractDataset.to_config() at the dataset level to serialize those objects.

[astrojuanlu]

In [16]: catalog = KedroDataCatalog()

In [17]: catalog.add_feed_dict({"ds": GBQQueryDataset("SELECT 1", "test_table", credentials=Credentials("abc"))})

In [18]: catalog.list()
Out[18]: ['ds']

In [19]: catalog.to_config()
Out[19]: ({}, {}, {}, None)

Should I have seen an error here @ElenaKhaustova ?

[astrojuanlu]

It does happen with a dataset only instantiated with primitive properties though 🤔

In [20]: catalog = KedroDataCatalog()

In [21]: catalog.add_feed_dict({"ds": GBQQueryDataset("SELECT 1", "test_table")})

In [22]: catalog.list()
Out[22]: ['ds']

In [23]: catalog.to_config()
Out[23]: ({}, {}, {}, None)

[ElenaKhaustova]

.add_feed_dict({"ds": GBQQueryDataset("SELECT 1", "test_table", credentials=Credentials("abc"))})

I'm not sure what credentials you passed here cause Credentials is an abstract.

But even if GBQQueryDataset()object is instantiated properly you won't see it in the serialization result because you used add_feed_dict method to add it to the catalog. This method adds everything as a MemoryDataset which is skipped during the serialization.

It works as expected in case you add dataset to the catalog via __setitem__ and dataset only instantiated with primitive properties:

But in case non-primitives are used, no error occurs as we do not dump the result dictionary and objects remains as they are - not serialized:

[ElenaKhaustova]

So the error should not occur until dumping from the result dictionary. We do not validate it now but we have a note in catalog.to_config() docstrings:

This method is only applicable to catalogs that contain datasets initialized with static, primitive
parameters. For example, it will work fine if one passes credentials as dictionary to
`GBQQueryDataset` but not as `google.auth.credentials.Credentials` object. See
https://github.com/kedro-org/kedro-plugins/issues/950 for the details.

[astrojuanlu]

I'm not sure what credentials you passed here cause Credentials is an abstract.

Took from google.oauth2.credentials import Credentials

you won't see it in the serialization result because you used add_feed_dict method

Hm I missed that KedroDataCatalog.add_feed_dict behaves differently than DataCatalog.add_feed_dict 🤔

So the error should not occur until dumping from the result dictionary

Indeed, looks like the problem would, in any case, going from dictionary to file.

Not sure we should do anything here.

[ElenaKhaustova]

Not sure we should do anything here.

I agree, but since the docstring refers to this ticket, we can keep it open until people start using the new catalog in case it becomes a common problem.