Related topic: dataframe protocol for data interchange/export

[rgommers]

In March'20 there was a very detailed discussion about introduction a new __dataframe__ protocol: https://discuss.ossdata.org/t/a-dataframe-protocol-for-the-pydata-ecosystem/267. The purpose of it is being able to exchange data between different implementations, or export data to (e.g.) Apache Arrow or NumPy.

There's a strawman implementation at wesm/dataframe-protocol#1.

The discussion went a little all over the place, with many people misunderstanding the main purpose was data exchange rather than providing an API to manipulate or do computations with a dataframe. That would be a much larger topic, and something this consortium aims to deliver an RFC for.

That said, the __dataframe__ topic is very much related, and is also a potentially interesting example of a cross-dataframe-library topic that could really benefit from having a detailed RFC with requirements and use cases. We should consider picking up that topic, and consider lessons from it in community engagement.

[szha]

It seems feasible to build this on top of array exchange. Having clarity first on array exchange seems like a low hanging fruit. To me the __cuda_array_interface__ doesn't seem generalizable to the future accelerators (e.g. it doesn't make sense to make __tpu{1,2,3,4}_array_interface__) so the pydata API definition would be a good opportunity to fix it.

[rgommers]

Time to close this. A lot of work has been done here; see https://data-apis.org/dataframe-protocol/latest/ and the "interchange protocol" label.