WIP: Implement DataFrame Interchange Protocol

[cnpryer]

Note: This is a draft and is currently in progress. Some details may change over time.

About

The DataFrame Interchange Protocol allows for the exchange of dataframes using a standard spec.

Some protocol documentation:

• See Purpose and Scope
• See Design concepts and requirements
• See The API of the __dataframe__ protocol

tl;dr (but you really should read if you're interested) Implements a public from_dataframe function for creating DataFrame objects using the __dataframe__ protocol and the __dataframe__ API itself.

Proposal

Objectives

• Add __dataframe__ API
• Add from_dataframe
• Finalize tests for API and compliance

This is a Python protocol. So we can implement this in the Python package. Note that this is not the Array API. Using "exchange" is how Pandas implements this. I propose instead we call this namespace "interchange" for correctness.

Reading through their PR it does look like it should be given a decent amount of attention and review. At the same time I want this to follow current Polars wrapper design patterns.

We can start by focusing on a similarly verbatim startup that focuses on the core protocol requirements. With this PR we can include unit tests, but a followup PR should implement some compliance testing.

Additional resources

• The Pandas implementation from the dataframe-api repo
  • Its initial commit
• PR for actual Pandas implementation
  • The project's public API
  • Early prototype implementation
  • No, indices are not protocol-supported
• Issue on relationship with array specs
• cuDF prototype
• Python Buffer Protocol
• __array_interface__ Protocol
• Arrow C Protocol note: Arrow2 implementation
• DLPack
• Array Interchange Protocol

Other

I'm not planning on rushing through this, so if you're browsing and wondering if you can leapfrog me with an implementation feel free!

[codecov-commenter]

Codecov Report

Merging #3727 (c05edcc) into master (25b58e2) will decrease coverage by 46.96%.
The diff coverage is 58.67%.

@@             Coverage Diff             @@
##           master    #3727       +/-   ##
===========================================
- Coverage   78.04%   31.07%   -46.97%     
===========================================
  Files         448      453        +5     
  Lines       74199    74395      +196     
===========================================
- Hits        57908    23118    -34790     
- Misses      16291    51277    +34986     

Impacted Files	Coverage Δ
py-polars/polars/interchange/dataframe.py	36.50% <36.50%> (ø)
py-polars/polars/interchange/buffer.py	48.14% <48.14%> (ø)
py-polars/polars/interchange/column.py	53.70% <53.70%> (ø)
py-polars/polars/interchange/_utils.py	96.00% <96.00%> (ø)
py-polars/polars/interchange/__init__.py	100.00% <100.00%> (ø)
py-polars/src/lazy/utils.rs	0.00% <0.00%> (-100.00%)	⬇️
py-polars/src/py_modules.rs	0.00% <0.00%> (-100.00%)	⬇️
polars/polars-io/src/tests.rs	0.00% <0.00%> (-100.00%)	⬇️
polars/polars-lazy/src/lib.rs	0.00% <0.00%> (-100.00%)	⬇️
polars/polars-time/src/lib.rs	0.00% <0.00%> (-100.00%)	⬇️
... and 369 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 25b58e2...c05edcc. Read the comment docs.

[cnpryer]

Should the Buffer class use pyarrow here? From what I can tell it doesn't implement the __array_interface__. So Buffer.ptr needs to be rolled here and I'm guessing that includes handling when arrays are chunked.

[cnpryer]

Another thought I've had working on this is that, unless I'm misunderstanding something, we're going to have to ensure contiguous memory for underlying data from the interchange. So I'd think we'd have to use copies if needed. Maybe I can be corrected here.

[cnpryer]

After reading into this more, there are some areas maybe unexplored with an arrow-backed Buffer.

Originally I wanted to establish a private API for polars using the spec's objects Buffer, Column, and DataFrame. After spending more time on this, starting with a public implementation for from_dataframe might provide more opportunity to learn about what a direction using arrow-backed buffers (or maybe even a polars data structure) could look like.

I gutted the current refactored interchange namespace to focus on a from_dataframe first approach here.

[cnpryer]

Since I'm struggling to find the time to work on this I'll close it for now and open it back up if I can get back to it.