Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: Timestamp == date match stdlib #36131

Merged
merged 64 commits into from
Jan 1, 2021
Merged
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
4c5eddd
REF: remove unnecesary try/except
jbrockmendel Aug 21, 2020
c632c9f
Merge branch 'master' of https://github.com/pandas-dev/pandas into re…
jbrockmendel Aug 21, 2020
9e64be3
Merge branch 'master' of https://github.com/pandas-dev/pandas into re…
jbrockmendel Aug 21, 2020
42649fb
TST: add test for agg on ordered categorical cols (#35630)
mathurk1 Aug 21, 2020
47121dd
TST: resample does not yield empty groups (#10603) (#35799)
tkmz-n Aug 21, 2020
1decb3e
revert accidental rebase
jbrockmendel Aug 22, 2020
57c5dd3
Merge branch 'master' of https://github.com/pandas-dev/pandas into ma…
jbrockmendel Aug 22, 2020
a358463
Merge branch 'master' of https://github.com/pandas-dev/pandas into ma…
jbrockmendel Aug 23, 2020
ffa7ad7
Merge branch 'master' of https://github.com/pandas-dev/pandas into ma…
jbrockmendel Aug 23, 2020
e5e98d4
Merge branch 'master' of https://github.com/pandas-dev/pandas into ma…
jbrockmendel Aug 24, 2020
408db5a
Merge branch 'master' of https://github.com/pandas-dev/pandas into ma…
jbrockmendel Aug 24, 2020
9b1b3f3
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Aug 26, 2020
a6d4228
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Sep 4, 2020
5517a68
BUG: Timestamp == date comparison match stdlib
jbrockmendel Sep 4, 2020
5ce9683
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Sep 11, 2020
f43e920
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Sep 12, 2020
1bffa44
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Sep 13, 2020
8c6012e
Deprecate
jbrockmendel Sep 13, 2020
ec827a0
isort troubleshoot
jbrockmendel Sep 13, 2020
fbb1576
catch warnings
jbrockmendel Sep 13, 2020
efae3ad
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Sep 13, 2020
e53dc16
suggested edit
jbrockmendel Sep 13, 2020
e504852
troubleshoot isort
jbrockmendel Sep 14, 2020
5b34905
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Sep 20, 2020
f0060e3
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Sep 21, 2020
f97848e
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Sep 22, 2020
41ae9cd
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Sep 26, 2020
f0add43
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Sep 26, 2020
da0888c
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Sep 29, 2020
27c8005
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Oct 1, 2020
8036c99
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Oct 3, 2020
a8fed20
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Oct 11, 2020
198d9e0
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Oct 14, 2020
67429a1
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Oct 31, 2020
3513f3b
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Oct 31, 2020
6df2513
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Oct 31, 2020
dbb3086
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Nov 10, 2020
87ff44a
split+fixturize
jbrockmendel Nov 10, 2020
c97a1e7
asserts for warnings
jbrockmendel Nov 10, 2020
67b3fb2
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Nov 11, 2020
da9426c
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Nov 25, 2020
13f170b
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Nov 25, 2020
ba18c96
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Nov 28, 2020
de69fb4
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Dec 6, 2020
d412ac0
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Dec 8, 2020
cf85a4f
move whatnsew
jbrockmendel Dec 8, 2020
a1e32d7
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Dec 17, 2020
938ad02
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Dec 23, 2020
06589ca
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Dec 23, 2020
4cd919c
tc1 -> cat_series1
jbrockmendel Dec 23, 2020
947daac
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Dec 27, 2020
5196b1e
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Dec 27, 2020
4d7aa41
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Dec 29, 2020
a307a0c
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Dec 29, 2020
a92409d
revert drop_duplicates tests
jbrockmendel Dec 30, 2020
164e2b4
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Dec 30, 2020
a321fdd
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Dec 30, 2020
0cb6019
Update pandas/_libs/tslibs/timestamps.pyx
jbrockmendel Dec 30, 2020
3557c7a
Merge branch 'bug-timestamp-cmp-date' of github.com:jbrockmendel/pand…
jbrockmendel Dec 30, 2020
af49cb5
catch warning
jbrockmendel Dec 30, 2020
7a423f4
lint fixup
jbrockmendel Dec 30, 2020
677760f
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Dec 31, 2020
b8c67e9
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Dec 31, 2020
fbfdaff
suppress warning
jbrockmendel Dec 31, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.2.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,7 @@ Deprecations
- Deprecated parameter ``inplace`` in :meth:`MultiIndex.set_codes` and :meth:`MultiIndex.set_levels` (:issue:`35626`)
- Deprecated parameter ``dtype`` in :~meth:`Index.copy` on method all index classes. Use the :meth:`Index.astype` method instead for changing dtype(:issue:`35853`)
- Date parser functions :func:`~pandas.io.date_converters.parse_date_time`, :func:`~pandas.io.date_converters.parse_date_fields`, :func:`~pandas.io.date_converters.parse_all_fields` and :func:`~pandas.io.date_converters.generic_parser` from ``pandas.io.date_converters`` are deprecated and will be removed in a future version; use :func:`to_datetime` instead (:issue:`35741`)
- Deprecated comparison of :class:`Timestamp` object with ``datetime.date`` objects. Instead of e.g. ``ts <= mydate`` use ``ts <= pd.Timestamp(mydate)`` (:issue:`36131`)
jbrockmendel marked this conversation as resolved.
Show resolved Hide resolved

.. ---------------------------------------------------------------------------

Expand Down
17 changes: 15 additions & 2 deletions pandas/_libs/tslibs/timestamps.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,15 @@ from numpy cimport int8_t, int64_t, ndarray, uint8_t

cnp.import_array()

from cpython.datetime cimport ( # alias bc `tzinfo` is a kwarg below
from cpython.datetime cimport (
PyDate_Check,
PyDateTime_Check,
PyDateTime_IMPORT,
PyDelta_Check,
PyTZInfo_Check,
datetime,
time,
tzinfo as tzinfo_type,
tzinfo as tzinfo_type, # alias bc `tzinfo` is a kwarg below
)
from cpython.object cimport Py_EQ, Py_NE, PyObject_RichCompare, PyObject_RichCompareBool

Expand Down Expand Up @@ -275,6 +276,18 @@ cdef class _Timestamp(ABCTimestamp):
return np.zeros(other.shape, dtype=np.bool_)
return NotImplemented

elif PyDate_Check(other):
# returning NotImplemented defers to the `date` implementation
# which incorrectly drops tz and normalizes to midnight
# before comparing
# We follow the stdlib datetime behavior of never being equal
warnings.warn(
"Comparison of Timestamp with datetime.date is deprecated in "
"order to match the standard library behavior. "
"In a future version these will be considered non-comparable."
"Use ts == pd.Timestamp(dt) instead.", FutureWarning
jbrockmendel marked this conversation as resolved.
Show resolved Hide resolved
)
return NotImplemented
else:
return NotImplemented

Expand Down
5 changes: 4 additions & 1 deletion pandas/tests/frame/indexing/test_indexing.py
Original file line number Diff line number Diff line change
Expand Up @@ -1931,7 +1931,10 @@ def test_setitem_datetime_coercion(self):
assert pd.Timestamp("2008-08-08") == df.loc[0, "c"]
assert pd.Timestamp("2008-08-08") == df.loc[1, "c"]
df.loc[2, "c"] = date(2005, 5, 5)
assert pd.Timestamp("2005-05-05") == df.loc[2, "c"]
with tm.assert_produces_warning(FutureWarning):
# Comparing Timestamp to date obj is deprecated
assert pd.Timestamp("2005-05-05") == df.loc[2, "c"]
assert pd.Timestamp("2005-05-05").date() == df.loc[2, "c"]

def test_setitem_datetimelike_with_inference(self):
# GH 7592
Expand Down
4 changes: 3 additions & 1 deletion pandas/tests/frame/test_constructors.py
Original file line number Diff line number Diff line change
Expand Up @@ -2628,7 +2628,9 @@ def test_datetime_date_tuple_columns_from_dict(self):
# GH 10863
v = date.today()
tup = v, v
result = DataFrame({tup: Series(range(3), index=range(3))}, columns=[tup])
with tm.assert_produces_warning(FutureWarning, check_stacklevel=False):
# GH#36131 comparison of Timestamp vs date
result = DataFrame({tup: Series(range(3), index=range(3))}, columns=[tup])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to avoid this warning in this case? (it's not something a user can do something about?)
And will the behaviour change here after the deprecationg has been enforced?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any thoughts here @jbrockmendel

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just the warning not to use date objects, or removing the Categorical behavior of passing convert_dates=True to maybe_infer_to_datetimelike

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should not be warning at the top level

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this the user/test is passing date objects, and gets a warning that behavior with date objects is going to change.

I can add a test in which the user-facing behavior is going to change (and is currently wrong)

v = date.today()
tup = v, v

ts = pd.Timestamp(v) + pd.Timedelta(hours=1)
tup2 = ts, ts

ser = pd.Series(range(3))

result = pd.DataFrame({tup2: ser}, columns=[tup])

>>> result
   (2020-12-17, 2020-12-17)
0                         0
1                         1
2                         2

But after this deprecation is enforced, we're going to end up with

>>> result
Empty DataFrame
Columns: [(2020-12-17, 2020-12-17)]
Index: []

which btw is analogous to what we'd get today if we swapped tup and tup2 above:

result2 = pd.DataFrame({tup: ser}, columns=[tup2])

>>> result2
Empty DataFrame
Columns: [(2020-12-17 01:00:00, 2020-12-17 01:00:00)]
Index: []

expected = DataFrame([0, 1, 2], columns=Index(Series([tup])))
tm.assert_frame_equal(result, expected)

Expand Down
4 changes: 3 additions & 1 deletion pandas/tests/groupby/test_apply.py
Original file line number Diff line number Diff line change
Expand Up @@ -1065,7 +1065,9 @@ def test_apply_with_date_in_multiindex_does_not_convert_to_timestamp():
)

grp = df.groupby(["A", "B"])
result = grp.apply(lambda x: x.head(1))
with tm.assert_produces_warning(FutureWarning, check_stacklevel=False):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we are exposing this to the user?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any idea why this happens?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when it tries to make a MultiIndex out of the A and B columns, it goes through Categorical, which passes convert_dates=True to maybe_infer_to_datetimelike,

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should not be warnign at the top level

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we could finagle it so that we dont pass convert_dates=True to maybe_infer_to_datetimelike in this case, but that gets pretty complicated, especially compared to just warning the user that date objects are finicky

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a while back you were in favor of telling people not to pass dates to the MultiIndex constructors.

# GH#36131 comparison of Timestamp vs date inside _make_concat_multiindex
result = grp.apply(lambda x: x.head(1))

expected = df.iloc[[0, 2, 3]]
expected = expected.reset_index()
Expand Down
36 changes: 36 additions & 0 deletions pandas/tests/scalar/timestamp/test_comparisons.py
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,42 @@ def test_compare_invalid(self):
assert val != np.float64(1)
assert val != np.int64(1)

@pytest.mark.parametrize("tz", [None, "US/Pacific"])
def test_compare_date(self, tz):
# GH#36131 comparing Timestamp with date object is deprecated
ts = Timestamp.now(tz)
dt = ts.to_pydatetime().date()
# These are incorrectly considered as equal because they
# dispatch to the date comparisons which truncates ts

for left, right in [(ts, dt), (dt, ts)]:
with tm.assert_produces_warning(FutureWarning):
assert left == right
with tm.assert_produces_warning(FutureWarning):
assert not left != right
with tm.assert_produces_warning(FutureWarning):
assert not left < right
with tm.assert_produces_warning(FutureWarning):
assert left <= right
with tm.assert_produces_warning(FutureWarning):
assert not left > right
with tm.assert_produces_warning(FutureWarning):
assert left >= right

# Once the deprecation is enforced, the following assertions
# can be enabled:
# assert not left == right
# assert left != right
#
# with pytest.raises(TypeError):
# left < right
# with pytest.raises(TypeError):
# left <= right
# with pytest.raises(TypeError):
# left > right
# with pytest.raises(TypeError):
# left >= right

def test_cant_compare_tz_naive_w_aware(self, utc_fixture):
# see GH#1404
a = Timestamp("3/12/2012")
Expand Down