Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: interchange protocol with nullable datatypes a non-null validity #57665

Merged
merged 11 commits into from
Mar 7, 2024

Conversation

MarcoGorelli
Copy link
Member

@MarcoGorelli MarcoGorelli commented Feb 28, 2024

This deals with pandas nullable dtypes

pyarrow nullables ones will require changes to the PandasBuffer class, so I'll handle those separately

Comment on lines -420 to +419
@pytest.mark.parametrize(
"dtype", ["Int8", pytest.param("Int8[pyarrow]", marks=td.skip_if_no("pyarrow"))]
)
def test_nullable_integers(dtype: str) -> None:
def test_nullable_integers() -> None:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

splitting the pyarrow dtype out of this test so I can xfail it

@MarcoGorelli MarcoGorelli added the Interchange Dataframe Interchange Protocol label Feb 28, 2024
@MarcoGorelli MarcoGorelli marked this pull request as ready for review February 28, 2024 21:57
def test_nullable_integers(dtype: str) -> None:
def test_nullable_integers() -> None:
# https://github.com/pandas-dev/pandas/issues/55069
df = pd.DataFrame({"a": [1]}, dtype="Int8")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you include a None in here to test null?

And parametrize over ["Int8", "Float32", "boolean"]?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

boolean isn't supported yet, there's a separate issue about that: #55332

sure, I've added a separate test test_pandas_nullable_w_missing_values which includes a null

@@ -19,6 +19,7 @@ Enhancements

Fixed regressions
~~~~~~~~~~~~~~~~~
- :meth:`DataFrame.__dataframe__` was producing incorrect data buffers when the a column's type was a pandas nullable on with missing values (:issue:`56702`)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh actually. I guess we need a 2.2.2 whatsnew first

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah right, 2.2.1 is already out, sorry about that, thanks for spotting it

@MarcoGorelli MarcoGorelli added this to the 2.2.2 milestone Mar 2, 2024
@MarcoGorelli MarcoGorelli requested a review from mroeschke March 7, 2024 14:13
@mroeschke mroeschke merged commit 03717bc into pandas-dev:main Mar 7, 2024
47 checks passed
@mroeschke
Copy link
Member

Thanks @MarcoGorelli

Copy link

lumberbot-app bot commented Mar 7, 2024

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

  1. Checkout backport branch and update it.
git checkout 2.2.x
git pull
  1. Cherry pick the first parent branch of the this PR on top of the older branch:
git cherry-pick -x -m1 03717bcc5ae762d8a0ab8d259ca000af66e8ba82
  1. You will likely have some merge/cherry-pick conflict here, fix them and commit:
git commit -am 'Backport PR #57665: BUG: interchange protocol with nullable datatypes a non-null validity'
  1. Push to a named branch:
git push YOURFORK 2.2.x:auto-backport-of-pr-57665-on-2.2.x
  1. Create a PR against branch 2.2.x, I would have named this PR:

"Backport PR #57665 on branch 2.2.x (BUG: interchange protocol with nullable datatypes a non-null validity)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.

MarcoGorelli added a commit to MarcoGorelli/pandas that referenced this pull request Mar 7, 2024
…pandas-dev#57665)

* BUG: interchange protocol with nullable datatypes a non-null validity provides nonsense results

* whatsnew

* 🏷️ typing

* parametrise over more types

* move whatsnew

(cherry picked from commit 03717bc)
MarcoGorelli added a commit to MarcoGorelli/pandas that referenced this pull request Mar 7, 2024
…pandas-dev#57665)

* BUG: interchange protocol with nullable datatypes a non-null validity provides nonsense results

* whatsnew

* 🏷️ typing

* parametrise over more types

* move whatsnew

(cherry picked from commit 03717bc)
MarcoGorelli added a commit that referenced this pull request Mar 8, 2024
…llable datatypes a non-null validity) (#57769)

BUG: interchange protocol with nullable datatypes a non-null validity (#57665)

* BUG: interchange protocol with nullable datatypes a non-null validity provides nonsense results

* whatsnew

* 🏷️ typing

* parametrise over more types

* move whatsnew

(cherry picked from commit 03717bc)
pmhatre1 pushed a commit to pmhatre1/pandas-pmhatre1 that referenced this pull request May 7, 2024
…pandas-dev#57665)

* BUG: interchange protocol with nullable datatypes a non-null validity provides nonsense results

* whatsnew

* 🏷️ typing

* parametrise over more types

* move whatsnew
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Interchange Dataframe Interchange Protocol
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BUG: interchange protocol with nullable datatypes a non-null validity provides nonsense results
4 participants