Should Pandas adopt a naming convention for its protocol methods

[ghost]

numpy has defined various protocol extensions over the years, and has consistently
named them with an array prefix, and "dunder" notation:

• __array__
• __array_prepare__, __array_wrap__, Subclassing ndarray
• __aray_ufunc__ NEP13
• __array_function__ NEP18

pandas has't been as diligent. The one example I've been dealing with, is _reduce. Originally an internal Series method (not a protocol), it now dispatches to _reduce for subclasses of ExtensionArray.

pandas/pandas/core/series.py

Lines 3743 to 3745 in baa77c3

	elif isinstance(delegate, ExtensionArray):
	# dispatch to ExtensionArray interface
	return delegate._reduce(name, skipna=skipna, **kwds)

When reading the code for a new EA project, its hard to pick out that _reduce is actually an override of the parent class, instead of just an internal function written by the EA author. Something like __pandas_reduce__ would have made this clearer.

Due to inexperience with EA, I lost a bit of time figuring out why s.sum() wasn't invoking the EA implementation I thought it would.

Granted, it's not exactly a protocol. For the forseeable future, this will always be an ExtensionArray subclass, rather than a duck typed object. But, for readability,
and while EA is experimental, this might be the right time to clean it up.

[TomAugspurger]

Typically dunder methods are associated with a top-level method intended for users to call. Since there isn't a single reduce method that corresponds to ExtensionArray._reduce, I don't think a dunder method is appropriate.

[jorisvandenbossche]

Note that this is partly also historically. Until recently with the EAs, we never had candidates for such methods. And the specific example of _reduce: this already existed internally before it was exposed as part of the EA interface, so this was mainly added to the EA interface to quickly have something that EAs can tap into.

But I think I agree with @TomAugspurger that at this moment, a full dunder method is maybe not warranted. It are indeed only overrides of the parent class, not actual protocols (and good documentation / comments in the source code can make this clear I think).

[TomAugspurger]

I'm OK with the current setup.

…

On Tue, Jun 18, 2019 at 9:01 PM pilkibun ***@***.***> wrote: Typically dunder methods are associated with a top-level method intended for users to call. not so. __array__? __array_ufunc__? none of the numpy methods have to do with top level methods. As I see it, duner methods are about protocols and/or duck typing. I don't think a dunder method is appropriate. @TomAugspurger <https://github.com/TomAugspurger>. Not necessarily a dunder method (though I think it fits), but a rename to indicate association with pandas. _reduce is rather generic. so this was mainly added to the EA interface to quickly have something that EAs can tap into. sure, but now should it stay that way? Remember, this is the first time where pandas extensions have been externally developed. Numpy looks for specific methods in an external object to detect support, pandas EA does the same (well, it just calls it). I think it should be thought of as a protocol. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#26915?email_source=notifications&email_token=AAKAOIWMHDHSCWD6EK2WOIDP3GHQFA5CNFSM4HY3XLX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYAORUI#issuecomment-503376081>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKAOISMF7UHKXKJST7YFWDP3GHQFANCNFSM4HY3XLXQ> .

[ghost]

#26935

[jorisvandenbossche]

Typically dunder methods are associated with a top-level method intended for users to call.

not so. array? array_ufunc? none of the numpy methods have to do with top level methods. As I see it, duner methods are about protocols and/or duck typing.

They are (but with top-level functions, not methods as Tom said, that was maybe confusing): __array__ is a way to tap into np.(as)array(..), __array_ufunc__ are a way to tap into np.add, np.subtract, .. top-level ufuncs.

I don't think we are ready at all to make official dunder methods. It took many years for numpy to get to this. Let's first further experiment with what we have, and for sure, look into what is still missing for other numerical ops (the things being discussed in the other issues you opened)

[jorisvandenbossche]

This issue is not actually about _reduce, but about the suggestion in #26935, that we build a single dispatch interface which unifies all the series methods,

Well, but the example we were discussing was _reduce. And we have many similar other methods that are part of our "EA interface", such as _from_sequence, _from_factorized, _concat_same_type, ... I don't think there is any point to start renaming those at this point.

To be clear, that is not to say the discussion in #26935 is not worth having. Absolutely not, that are problems we need to find a solution for. But just that the starting to rename the already existing methods is currently not that useful, I think (not that we could have chosen better names if we would start over). So closing this, let's continue the discussion about the actual method dispatch problem in #26935.