ERR: Remove bitwise operations on dtype=object?

[eyurtsev]

Code Sample, a copy-pastable example if possible

1) This is okay:

~pd.Series([False, False, True, False], dtype=bool)

Out[76]:
0     True
1     True
2    False
3     True
dtype: bool

2) This looks like a problem:

~pd.Series([False, False, True, False], dtype=bool).shift(1).dropna()
1    -1
2    -1
3    -2
dtype: object

Problem description

.shift and .dropna are common pandas operations.

.shift(1) converts dtype from bool to object, so the bitwise operation is applied to each object (~False).

The output in the latter case is extremely surprising. It might be safer to raise an Exception rather than allow bitwise operations implemented on objects.

~pd.Series([1.0], dtype=object)

Expected Output

Exception!

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 2.7.13.final.0 python-bits: 64 OS: Linux OS-release: 4.10.0-21-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: None.None

pandas: 0.19.0
nose: 1.3.7
pip: 9.0.1
setuptools: 35.0.2
Cython: 0.25.1
numpy: 1.10.1
scipy: 0.16.0
statsmodels: 0.6.1
xarray: None
IPython: 5.3.0
sphinx: 1.2.2
patsy: 0.4.0
dateutil: 2.6.0
pytz: 2015.6
blosc: None
bottleneck: None
tables: 3.3.0
numexpr: 2.4.6
matplotlib: 2.0.1
openpyxl: 2.3.3
xlrd: 0.9.4
xlwt: None
xlsxwriter: 0.8.4
lxml: 3.5.0
bs4: 4.5.3
html5lib: 0.999999999
httplib2: 0.9.2
apiclient: 1.4.2
sqlalchemy: 1.0.8
pymysql: None
psycopg2: None
jinja2: 2.9.6
boto: 2.20.1
pandas_datareader: None

[jreback]

yeah this is a tough one. We don't normally infer object dtypes before other ops. And of course this is object because we don't have first class NA for bools :< Though these are bitwise ops so we could infer and if not bool raise a TypeError.

do you want to have a go and see how much impact this would have? IOW add some tests and make a change and see what else breaks?

[david-zwicker]

I just stumbled across a similar and likely connected error when dealing with boolean data that was for some reason stored with dtype=object. Here is a short sample demonstrating the problem:

>>> ~pd.DataFrame([True, True], dtype=object)
      0
0    -2
1    -2

This is of course rather unexpected, in particular since boolean indexing cannot be used with this result.

[toobaz]

Might be worth mentioning that while Series and dtype-specific indexes typically support the same arithmetic operations, the standard Index class (so dtype=object) does not try to make inference and just fails:

In [2]: -pd.Index([2, 4, 6], dtype='object')
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-7c929da771a5> in <module>()
----> 1 -pd.Index([2, 4, 6], dtype='object')

~/nobackup/repo/pandas/pandas/core/ops.py in invalid_op(self, other)
    183     def invalid_op(self, other=None):
    184         raise TypeError("cannot perform {name} with this index type: "
--> 185                         "{typ}".format(name=name, typ=type(self).__name__))
    186 
    187     invalid_op.__name__ = name

TypeError: cannot perform __neg__ with this index type: Index

The choice we take for Series(., dtype=object) and for Index(., dtype=object) should probably be consistent (although this is not, as of now, a direct concern for ~, which is entirely unsupported for indexes - see #22336)