Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Numpy bool msgpack bugfix #18395

Merged
merged 5 commits into from
Nov 23, 2017
Merged

Numpy bool msgpack bugfix #18395

merged 5 commits into from
Nov 23, 2017

Conversation

PhyNerd
Copy link
Contributor

@PhyNerd PhyNerd commented Nov 20, 2017

This fixes issue #18390.
It allows for running DataFrame.to_msgpack() with dataframes containing fields of the numpy.bool_ datatype.

@codecov
Copy link

codecov bot commented Nov 21, 2017

Codecov Report

Merging #18395 into master will decrease coverage by 0.01%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #18395      +/-   ##
==========================================
- Coverage   91.35%   91.34%   -0.02%     
==========================================
  Files         163      163              
  Lines       49691    49691              
==========================================
- Hits        45397    45388       -9     
- Misses       4294     4303       +9
Flag Coverage Δ
#multiple 89.14% <ø> (ø) ⬆️
#single 39.67% <ø> (-0.07%) ⬇️
Impacted Files Coverage Δ
pandas/io/gbq.py 25% <0%> (-58.34%) ⬇️
pandas/core/frame.py 97.8% <0%> (-0.1%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fedc503...378a218. Read the comment docs.

@jreback jreback added the Dtype Conversions Unexpected or buggy dtype conversions label Nov 21, 2017
@jreback jreback added this to the 0.21.1 milestone Nov 21, 2017
@@ -133,7 +134,7 @@ cdef class Packer(object):
while True:
if o is None:
ret = msgpack_pack_nil(&self.pk)
elif isinstance(o, bool):
elif isinstance(o, (bool, bool_)):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use np.bool_

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do.

@@ -25,6 +26,7 @@ def testPack(self):
(), ((),), ((), None,),
{None: 0},
(1 << 23),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a round-trip test for the example in the issue itself.

Copy link
Contributor Author

@PhyNerd PhyNerd Nov 21, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test should already test for the buggy behavior when it comes to np.bool_.

Should I revert the changes to this test and add a one checking round-trip behavior or should I make that test a additional one?
Should the test for the to_msgpack() round-trip behavior or packer roundtrip behavior?
If it should check for to_msgpack() behavior where would I best add this test? (There is no /test/io/formats/test_to_msgpack.py)

@jreback jreback removed this from the 0.21.1 milestone Nov 21, 2017
@PhyNerd
Copy link
Contributor Author

PhyNerd commented Nov 21, 2017

I applied the requested changes.
I'm not sure why but one of the Travis CI builds timed-out.

@jreback
Copy link
Contributor

jreback commented Nov 22, 2017

can you add the tests from the original issue as well, e.g. the round-trip of the object array containing bools. rebase and ping on green.

@PhyNerd PhyNerd force-pushed the NumpyBoolMsgpack branch 2 times, most recently from e82439a to dd73128 Compare November 22, 2017 13:07
@PhyNerd
Copy link
Contributor Author

PhyNerd commented Nov 22, 2017

I added the requested check.

@jreback jreback added this to the 0.21.1 milestone Nov 22, 2017
@jreback jreback added Compat pandas objects compatability with Numpy or Python functions Needs Backport labels Nov 22, 2017
@jreback
Copy link
Contributor

jreback commented Nov 22, 2017

lgtm. just needs a rebase. ping when pushed.

This allows for running to_msgpack with dataframes containing field of numpy.bool_ datatype
Added test for scaler numpy bool, added numpy bool to mixed list test, removed modifications to pack test and changend bool_ to np.bool_
@PhyNerd
Copy link
Contributor Author

PhyNerd commented Nov 23, 2017

rebased.

@jreback jreback merged commit 6c074d1 into pandas-dev:master Nov 23, 2017
@jreback
Copy link
Contributor

jreback commented Nov 23, 2017

thanks @PhyNerd

TomAugspurger pushed a commit to TomAugspurger/pandas that referenced this pull request Dec 8, 2017
TomAugspurger pushed a commit that referenced this pull request Dec 11, 2017
(cherry picked from commit 6c074d1)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Compat pandas objects compatability with Numpy or Python functions Dtype Conversions Unexpected or buggy dtype conversions
Projects
None yet
Development

Successfully merging this pull request may close these issues.

to_msgpack fails for Dataframes with numpy bools
3 participants