BUG: Unexpected behaviour comparison dataframes with None values

[rosekoopman]

• [x ] I have checked that this issue has not already been reported.

• [ x] I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

np.nan == np.nan
# False

pd.Series(index=[0], data=[np.nan]) == pd.Series(index=[0], data=[np.nan])
# False

None == None
# True

pd.Series(index=[0], data=[None]) == pd.Series(index=[0], data=[None])
# False

Problem description

If None==None equals True, I had expected that pd.Series(index=[0], data=[None]) == pd.Series(index=[0], data=[None]) would also equal True.

I have pandas version 1.0.5

Expected Output

pd.Series(index=[0], data=[None]) == pd.Series(index=[0], data=[None])

True

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.6.10.final.0
python-bits : 64
OS : Linux
OS-release : 4.14.171-105.231.amzn1.x86_64
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.0.5
numpy : 1.16.2
pytz : 2019.3
dateutil : 2.8.0
pip : 20.0.2
setuptools : 46.1.3.post20200330
Cython : None
pytest : 5.3.5
hypothesis : None
sphinx : 2.4.3
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.5.0
html5lib : None
pymysql : None
psycopg2 : 2.8.4 (dt dec pq3 ext lo64)
jinja2 : 2.11.1
IPython : 7.12.0
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : 4.5.0
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 0.15.1
pytables : None
pytest : 5.3.5
pyxlsb : None
s3fs : 0.4.0
scipy : 1.4.1
sqlalchemy : 1.3.13
tables : None
tabulate : 0.8.6
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
numba : None

[mroeschke]

This is expected behavior. See: https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html#values-considered-missing

[dsaxton]

@mroeschke While this is expected for None there's an almost identical issue for object NA which does seem like a bug:

In [3]: ser = pd.Series([pd.NA])                                                                                                                                                                             

In [4]: ser == ser                                                                                                                                                                                           
Out[4]: 
0    False
dtype: bool

Does it make sense to modify this / leave it open to track this? I think it's due to the same line:

pandas/pandas/_libs/ops.pyx

Line 167 in 6a6faf5

if checknull(x) or checknull(y):

[mroeschke]

Might be better to open up a separate issue for object NA.

I haven't followed NA semantics too closely but shouldn't that comparison also resolve to False or does the spec say it should be True?

[dsaxton]

Might be better to open up a separate issue for object NA.

I haven't followed NA semantics too closely but shouldn't that comparison also resolve to False or does the spec say it should be True?

The comparison should resolve to NA since the equality is "unknown", @rosekoopman would you be interested in opening up an issue for that?

[rosekoopman]

@mroeschke : thanks for the explanation.

@dsaxton : yes I can open a new issue. However, the documentation linked by @mroeschke seems to suggest that False is the expected behaviour:

One has to be mindful that in Python (and NumPy), the nan's don’t compare equal, but None's do. Note that pandas/NumPy uses the fact that np.nan != np.nan, and treats None like np.nan.

[dsaxton]

@rosekoopman Actually let's not worry about opening another, looks like the issue is known and tracked here: #32931