make DateOffset immutable #21341

jbrockmendel · 2018-06-06T16:10:10Z

Returning to the long-standing goal of making DateOffsets immutable (recall: DateOffset.__eq__ calls DateOffset._params which is very slow. _params can't be cached ATM because DateOffset is mutable`)

Earlier attempts in this direction tried to make the base class a cdef class, but that has run into pickle problems that I haven't been able to sort out so far. This PR goes the patch-__setattr__ route instead.

Note: this PR does not implement the caching that is the underlying goal.

I'm likely to make some other PRs in this area, will try to keep them orthogonal.

codecov · 2018-06-06T18:51:00Z

Codecov Report

❗ No coverage uploaded for pull request base (master@e24da6c). Click here to learn what that means.
The diff coverage is 100%.

@@            Coverage Diff            @@
##             master   #21341   +/-   ##
=========================================
  Coverage          ?   91.91%           
=========================================
  Files             ?      153           
  Lines             ?    49550           
  Branches          ?        0           
=========================================
  Hits              ?    45546           
  Misses            ?     4004           
  Partials          ?        0

Flag	Coverage Δ
#multiple	`90.31% <100%> (?)`
#single	`41.79% <100%> (?)`

Impacted Files	Coverage Δ
pandas/tseries/offsets.py	`97.16% <100%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e24da6c...f931224. Read the comment docs.

shoyer · 2018-06-06T20:37:36Z

pandas/_libs/tslibs/offsets.pyx

+        object.__setattr__(self, "_cache", {})
+
+    def __setattr__(self, name, value):
+        raise AttributeError("DateOffset objects are immutable.")


The standard approach is usually to use private attribute like _n and provide access via a properties. Is there a reason why that wouldn't make sense here?

That would absolutely make sense, but since micro-benchmarks are a concern I prefer to avoid the property overhead. The best-case would be the implementation in ~~#18824~~#18224, but I can't figure out the pickle errors there.

OK, seems reasonable to me.

sorenwacker · 2018-06-07T00:25:22Z

OK, its working again.

jreback · 2018-06-14T10:26:09Z

@jbrockmendel ok pls rebase, I think you have some residual comments about normalize (your PR was merged so maybe need changing)

jbrockmendel · 2018-06-14T16:34:02Z

This is better than nothing since it will allow us to move forward with some serious perf improvements to PeriodIndex ops, but is still pretty ugly. If there's a decent chance of getting someone more competent than me to look at the pickle issue holding up ~~#18824~~#18224 that would be the best-case.

jreback · 2018-06-19T00:29:56Z

pandas/_libs/tslibs/offsets.pyx

@@ -304,6 +304,15 @@ class _BaseOffset(object):
    _day_opt = None
    _attributes = frozenset(['n', 'normalize'])

+    def __init__(self, n=1, normalize=False):
+        n = self._validate_n(n)
+        object.__setattr__(self, "n", n)


I would just set these as regular attributes, I doubt this actually makes a signfiicant perf difference.

Not sure I understand. These are regular attributes, but the PR overrides __setattr__ so we have to use object.__setattr__.

oh sorry you are right.

can you set these as cdef readonly? (and maybe type them) or not yet?

I really really wish I could, but until someone smarter than me looks at ~~#18824~~#18224 cdef class causes pickle test failures.

@jbrockmendel what pickle issue are you referring to in that change? That just looks like a master tracker but not sure the exact issue contained within

@WillAyd thanks for catching that, should be #18224. Will fix above.

jreback · 2018-06-19T00:30:07Z

pandas/_libs/tslibs/offsets.pyx

+        n = self._validate_n(n)
+        object.__setattr__(self, "n", n)
+        object.__setattr__(self, "normalize", normalize)
+        object.__setattr__(self, "_cache", {})


what is _cache for?

If we don't create this explicitly in __init__ then @cache_readonly lookups will try to create it, which will raise.

can you put this on the class? (or is that really weird)?

pd.DateOffset._cache = {} off = pd.DateOffset(n=1) hour = pd.offsets.BusinessHour(n=2) >>> off._cache is hour._cache True

right so we can't use the @cache_readonly decorator here? (or is that what you are doing by pre-emptively setting the cache)?

cache_readonly.__get__ checks for a _cache attribute and creates one if it does not exist. Creating it in __init__ ensures that one exists, so a new one does not have to be created (since attempting to do so would raise)

i.e. this is necessary in order to retain the existing usages of cache_readonly on DateOffset subclasses.

ok makes sense

jreback · 2018-06-19T01:19:09Z

@jbrockmendel ok this is ok. merge before your set_state/get_state change? does it matter?

jbrockmendel · 2018-06-19T02:19:06Z

There will be a merge conflict I'll need to address. Since I'm holding out hope WillAyd can help make #18224 viable (and this unnecessary), I'd rather the the getstate/setstate one get merged first.

jreback · 2018-06-19T10:51:49Z

ok dokie

jreback

I think this needs a whatsnew as now technically DateOffsets are immutable yes? do we have some direct tests for this?

jbrockmendel · 2018-06-20T18:01:51Z

I think this needs a whatsnew as now technically DateOffsets are immutable yes? do we have some direct tests for this?

added

jreback · 2018-06-21T10:24:29Z

thanks @jbrockmendel

make DateOffset immutable

8699d01

gfyoung added Internals Related to non-user accessible pandas implementation Timedelta Timedelta data type labels Jun 6, 2018

gfyoung requested a review from jreback June 6, 2018 16:38

shoyer reviewed Jun 6, 2018

View reviewed changes

Merge branch 'master' of https://github.com/pandas-dev/pandas into imm

b11026e

Merge branch 'master' of https://github.com/pandas-dev/pandas into imm

dfcf83f

jreback requested changes Jun 19, 2018

View reviewed changes

Merge branch 'master' of https://github.com/pandas-dev/pandas into imm

88f8cff

jreback approved these changes Jun 19, 2018

View reviewed changes

jreback added this to the 0.24.0 milestone Jun 19, 2018

Merge branch 'master' of https://github.com/pandas-dev/pandas into imm

e561f36

jreback requested changes Jun 20, 2018

View reviewed changes

whatsnew, tests

71204a4

jreback added the Frequency DateOffsets label Jun 21, 2018

jreback approved these changes Jun 21, 2018

View reviewed changes

jreback added 2 commits June 21, 2018 06:22

doc

cfc409d

Merge branch 'master' into PR_TOOL_MERGE_PR_21341

f931224

jreback merged commit 1638331 into pandas-dev:master Jun 21, 2018

This was referenced Jun 21, 2018

cache DateOffset attrs now that they are immutable #21582

Merged

COMPAT: Scalar types immutable #17116

Closed

jbrockmendel deleted the imm branch June 22, 2018 03:27

tswast mentioned this pull request Jun 25, 2018

Cannot run tests on HEAD due to object.__init__() takes no parameters on BaseOffset.__init__(self, n, normalize) #21626

Closed

Sup3rGeo pushed a commit to Sup3rGeo/pandas that referenced this pull request Oct 1, 2018

make DateOffset immutable (pandas-dev#21341)

cbb7cc0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

make DateOffset immutable #21341

make DateOffset immutable #21341

jbrockmendel commented Jun 6, 2018

codecov bot commented Jun 6, 2018 •

edited

Loading

shoyer Jun 6, 2018

jbrockmendel Jun 7, 2018 •

edited

Loading

shoyer Jun 7, 2018

sorenwacker commented Jun 7, 2018

jreback commented Jun 14, 2018

jbrockmendel commented Jun 14, 2018 •

edited

Loading

jreback Jun 19, 2018

jbrockmendel Jun 19, 2018

jreback Jun 19, 2018

jbrockmendel Jun 19, 2018 •

edited

Loading

WillAyd Jun 19, 2018

jbrockmendel Jun 19, 2018

jreback Jun 19, 2018

jbrockmendel Jun 19, 2018

jreback Jun 19, 2018

jbrockmendel Jun 19, 2018

jreback Jun 19, 2018

jbrockmendel Jun 19, 2018

jbrockmendel Jun 19, 2018

jreback Jun 19, 2018

jreback commented Jun 19, 2018

jbrockmendel commented Jun 19, 2018

jreback commented Jun 19, 2018

jreback left a comment

jbrockmendel commented Jun 20, 2018

jreback commented Jun 21, 2018

make DateOffset immutable #21341

make DateOffset immutable #21341

Conversation

jbrockmendel commented Jun 6, 2018

codecov bot commented Jun 6, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

jbrockmendel Jun 7, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sorenwacker commented Jun 7, 2018

jreback commented Jun 14, 2018

jbrockmendel commented Jun 14, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbrockmendel Jun 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jun 19, 2018

jbrockmendel commented Jun 19, 2018

jreback commented Jun 19, 2018

jreback left a comment

Choose a reason for hiding this comment

jbrockmendel commented Jun 20, 2018

jreback commented Jun 21, 2018

codecov bot commented Jun 6, 2018 •

edited

Loading

jbrockmendel Jun 7, 2018 •

edited

Loading

jbrockmendel commented Jun 14, 2018 •

edited

Loading

jbrockmendel Jun 19, 2018 •

edited

Loading