API: expose pandas.errors #15541

jreback · 2017-03-01T19:13:01Z

this doesn't actually deprecate the existing (as we have discussed in #15181) but baby steps.

I also didn't add the 'hated'! SettingWithCopy* ones; these will be removed in pandas 2.0 anyhow, so no point in adding them.

jreback · 2017-03-01T19:13:08Z

@shoyer xref #15537 (comment)

jreback · 2017-03-01T19:50:21Z

@jorisvandenbossche xref #15181 (comment)

I think this actually works.

In [1]: from pandas.api.exceptions import UnsortedIndexError

In [2]: try:
   ...:     raise UnsortedIndexError()
   ...: except UnsortedIndexError:
   ...:     print("caught")
   ...:     
caught

TomAugspurger · 2017-03-01T21:50:08Z

@jreback second one should be except exceptions.UnsortedIndexError, ;) Missing the exceptions.

jreback · 2017-03-01T21:58:11Z

thanks @TomAugspurger yes that works too

jorisvandenbossche · 2017-03-01T23:57:57Z

I think this actually works.

Yes, but that is because you didn't do any deprecations of the original ones. Referring from two locations to the same exception is no problem, but it was deprecating the old one (eg by using a decorator) that caused the problems discussed before.

I am in favor of putting this just in pandas.exceptions

jreback · 2017-03-02T00:17:10Z

@jorisvandenbossche

I am in favor of putting this just in pandas.exceptions

why would you create another namespace ? when we already have a perfectly good one.

jreback · 2017-03-03T15:08:55Z

@jorisvandenbossche can you comment on your objection?

jorisvandenbossche · 2017-03-03T15:27:12Z

can you comment on your objection?

Well, that comes back to my initial objections to the api namespace :-) (so I should have pressed that more earlier ..)
I am in favour of removing things of the top-level namespace (functions / sub-namespaces) that people shouldn't use (as we are doing), but in favour of adding things users may use.

In this specific case, what I also would do is creating an actual module (let it be pandas.api.exceptions or pandas.exceptions, that's even a bit different discussion) with the code in there, and let our other modules (in the io code, indexing code, etc) import the exceptions from there (instead the other way around as it is now). That gathers all our custom exceptions in one place (but on the other hand moves them away from where they are used, so that's not necessarily a net win), but also gives a consistent api to users. If they encounter such an exception, it is of the same type (I mean with same class path name) as the one they can use to catch it. And you minimize the chance to 'misuse' a private import.

And to be clear, big +1 to make those exceptions more officially public!

jreback · 2017-03-03T16:03:28Z

@jorisvandenbossche sure, but these are really 2 separate things.

an API for exceptions to the user
move / change the internal usage

I don't think the 2nd part is relevant here. I mean we can always change how they are imported into the public namespace. The point here is to have a separate public API layer (just as we did with pandas.api.types. The actual exceptions being defined elsewhere is fine.

shoyer · 2017-03-03T16:38:20Z

I agree with Joris that it would be better to define the exceptions in `pandas.exceptions` or whereever the public API location is (personally I like the shorter `pandas.errors`). Unlike __repr__ I don't think Python provides any builtin way to indicate the canonical location of errors. I like making a immediately visible location the API location -- it makes it much harder to unintentionally use the wrong path.

…

On Fri, Mar 3, 2017 at 8:03 AM Jeff Reback ***@***.***> wrote: @jorisvandenbossche <https://github.com/jorisvandenbossche> sure, but these are really 2 separate things. - an API for exceptions to the user - move / change the internal usage I don't think the 2nd part is relevant here. I mean we can always change *how* they are imported into the public namespace. The point here is to have a separate public API layer (just as we did with pandas.api.types. The actual exceptions being defined elsewhere is fine. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15541 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABKS1vdT_M_Eh7sMXWUJz9IsDt1PlGqAks5riDnTgaJpZM4MQFAO> .

jreback · 2017-03-03T19:58:41Z

@shoyer @jorisvandenbossche what is the reason to deviate from the established pandas.api namespace?

find with exceptions -> errors. actually was maybe going to split out warnings separately.

This is the point of the top-level api namespace, IOW.

api.types, api.errors, api.warnings, api.* whatever (imaging .algos, .lib, .includes, .src) etc.

this is going to eventually proliferate. We cannot have pandas.* namespace be anything but non-public.

We have 1 and only 1 public namespace (or at least moving in that direction).

shoyer · 2017-03-03T20:13:01Z

If the default for namespaces like pandas.* going forward is public, what's the advantage of nesting them like pandas.api.*?

Most libraries don't use special designated api submodules, e.g., consider all the scipy submodules like scipy.stats, scipy.ndimage, scipy.linalg, etc.

jreback · 2017-03-03T20:17:58Z

no, pandas.* namespace is NOT public (or at least it shouldn't be). My statement is that is de-facto currently.

I am thinking about statsmodels.api

The point is that pandas NEEDS a specifically public api, that is not just everything the kitchen sink.

just because scipy/numpy does something does not make it the 'correct' way to do things. sure its a nice reference, but pandas has 2 sets of users, API and 'regular' ones. These are different and need to be treated differently.

jreback · 2017-03-03T20:19:26Z

The problem is that for years people building things on top of pandas have just reached in and used anything. That cannot continue to happen. How better to provide implementation flexibility than to have a specific public API? I thought we had settled this discussion a while back.

shoyer · 2017-03-03T21:29:57Z

Ultimately, I don't think this matters too much, but given a choice between:

pandas.* indicates private namespace, pandas.api.* includes a public namespace, and
pandas.* includes public namespace, pandas._internals (and so forth) indicates private namespace

I think the later (option 2) is more user friendly and consistent with the top level namespace pandas already being public facing. In contrast, note that with statsmodels, users only use statsmodels.api, not the top level namespace statsmodels.

On the other hand, it's true that option 2 is a little harder to transition to (for us), so maybe it's not worth the trouble.

I don't really understand your point about "API" vs "regular" users. I don't think it's a terribly advanced use case to catch appropriate exception types. I do agree that we want to reduce noise in the top level namespace.

jreback · 2017-03-03T21:33:55Z

I don't really understand your point about "API" vs "regular" users. I don't think it's a terribly advanced use case to catch appropriate exception types. I do agree that we want to reduce noise in the top level namespace.

I would this is obvious.

'regular' users are the ones who pretty much use pd.* as there namespace. In theory they may use pandas.api.exceptions (e.g. specifically catching a parser exception).
'API' users are using (formerly) pandas internals to build other things on top of pandas. This is the point of an API.

Ultimately these are part of the same namespace that is publicly exposed. The point is that it is completely silly that people can just use any of the namespace. Sure you can, but pandas has gotten so big, that you simply cannot live that that when you need to move things around to change implementation details.

jreback · 2017-03-03T21:35:32Z

pandas.* includes public namespace, pandas._internals (and so forth) indicates private namespace

how exactly is this more friendly? I would argue that this is much less friendly, simply because you don't know where to find things that you might need. This is a huge surface area, both for dev and for usage. Limiting this scope is hugely important.

codecov-io · 2017-03-08T14:04:30Z

Codecov Report

Merging #15541 into master will decrease coverage by 0.02%.
The diff coverage is 92.3%.

@@            Coverage Diff             @@
##           master   #15541      +/-   ##
==========================================
- Coverage   90.98%   90.96%   -0.03%     
==========================================
  Files         143      145       +2     
  Lines       49477    49483       +6     
==========================================
- Hits        45019    45014       -5     
- Misses       4458     4469      +11

Flag	Coverage Δ
#multiple	`88.72% <92.3%> (-0.01%)`	⬇️
#single	`40.63% <82.05%> (-0.12%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/indexing.py	`94.1% <ø> (ø)`	⬆️
pandas/tslib.py	`100% <ø> (ø)`	⬆️
pandas/lib.py	`100% <ø> (ø)`	⬆️
pandas/api/lib/__init__.py	`0% <0%> (ø)`
pandas/tseries/index.py	`95.43% <100%> (ø)`	⬆️
pandas/io/excel.py	`79.7% <100%> (+0.03%)`	⬆️
pandas/computation/align.py	`97.91% <100%> (+0.02%)`	⬆️
pandas/io/parsers.py	`95.52% <100%> (ø)`	⬆️
pandas/core/common.py	`90.68% <100%> (-0.29%)`	⬇️
pandas/core/frame.py	`97.56% <100%> (-0.1%)`	⬇️
... and 15 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 456e729...e5fbdc8. Read the comment docs.

jreback · 2017-03-09T19:26:25Z

/home/joris/miniconda3/envs/dev/lib/python3.5/site-packages/feather/api.py:43: FutureWarning: pandas.lib.infer_dtype is deprecated. Please use pandas._libs.lib.infer_dtype instead.
  inferred_type = pd.lib.infer_dtype(col)

certainly could do a PR upstream to fix this (e.g. with conditional imports). But might be nice
to have
from pandas.api.lib import infer_dtype

as a starting location for some library code. This is a useful enough routine that its nice to expose to the outside world.

jreback · 2017-03-09T19:26:44Z

cc @wesm

wesm · 2017-03-13T03:37:00Z

I've been running into these internal API / external API issues lately (e.g. needing to use DatetimeTZDtype in an external library), so it might be worth documenting the pandas < 0.19 way to get some of these APIs

jorisvandenbossche · 2017-03-13T10:51:10Z

We indeed need to document this better.

To come back on the discussion above, I am also more in favor of "option 2" from #15541 (comment). It's indeed maybe a bit harder to get to that point, but IMO it would be much more clear to be able to say: everything accessible from the pandas.* namespace (directly or in a submodule) is public (rather than eg have to say pandas.types is private, but pandas.api.types you may use). But that is also a more general discussion not only related to exceptions, cfr #13634

jreback · 2017-03-27T13:58:23Z

@shoyer
updated

removed PandasError completely (barely used)
added doc-string to infer_dtype

shoyer · 2017-03-27T17:43:08Z

@jreback looks great to me -- thanks!

One question on the docstring for infer_dtype: what do mixed and `mixed-integer mean? It would be nice to give some guidance on these. Here's my guess:

I think mixed means "not anything else" (i.e., dtype=object from numpy), not necessarily mixed types (e.g., you get "mixed" for an list of all tuples). So perhaps it would be better called "unknown-or-mixed" or something like that. Maybe this isn't worth changing, but it's definitely worth documenting.
I have no idea what 'mixed-integer' means exactly or how to trigger it. I thought this might mean a mixture of integer types (e.g., int32 vs int64) but those just turn up 'integer'.

jreback · 2017-03-27T18:07:22Z

@shoyer these have long-time meaning in pandas (could be changed, but this is what they are).

mixed* are just non-strings (unicode/bytes) that are actual mixed python objects.

In [9]: infer_dtype(['a', 1])
Out[9]: 'mixed-integer'

In [10]: infer_dtype(['a', 1.5])
Out[10]: 'mixed'

In [11]: infer_dtype([1.5, 1])
Out[11]: 'mixed-integer-float'

In [12]: infer_dtype(['a', pd.Timestamp('20130101')])
Out[12]: 'mixed'

In [14]: infer_dtype([(1, 2, 3)])
Out[14]: 'mixed'

jreback · 2017-03-27T18:19:24Z

pushed some examples.

jorisvandenbossche

minor comment, looks good to me

jorisvandenbossche · 2017-03-27T20:39:20Z

pandas/_libs/src/inference.pyx

+    -------
+    string describing the common type of the input data.
+    Results can include:
+       - floating


small rst thing: no indentation needed, and white line before the list

done, though I don't think we actually have a doc-generation section for pandas.api ATM. (i'll make an issue).

jreback · 2017-03-29T15:36:01Z

@jorisvandenbossche if you have a chance.

chris-b1 · 2017-04-03T17:50:48Z

pandas/lib.py

@@ -2,6 +2,7 @@

 import warnings
 warnings.warn("The pandas.lib module is deprecated and will be "


Would it be possible to specially catch infer_dtype here and give a more specific message since it has moved and is public? Noticed I'm using it some real code, I think based on your SO answer here - http://stackoverflow.com/a/20670901/3657742.

In [1]: pd.lib.infer_dtype('foo') /Users/jreback/miniconda3/envs/pandas/bin/ipython:1: FutureWarning: pandas.lib.infer_dtype is deprecated. Please use pandas._libs.lib.infer_dtype instead. #!/Users/jreback/miniconda3/envs/pandas/bin/python Out[1]: 'string'

hmm, let me see.

yep

In [1]: pd.lib.infer_dtype('foo') /Users/jreback/miniconda3/envs/pandas/bin/ipython:1: FutureWarning: pandas.lib is deprecated and will be removed in a future version. You can access infer_dtype in pandas.api.lib.infer_dtype #!/Users/jreback/miniconda3/envs/pandas/bin/python Out[1]: 'string'

done in latest push

add infer_dtype to pandas.api.lib

move all exception / warning impls (and references) from pandas.core.common and pandas.io.common closes pandas-dev#14800

jorisvandenbossche

Looks good to me!

* API: expose dtypes in pandas.api.types xref #16015 xref apache/arrow#585 xref #16042 xref #15541 (comment)

* API: expose dtypes in pandas.api.types xref pandas-dev#16015 xref apache/arrow#585 xref pandas-dev#16042 xref pandas-dev#15541 (comment)

jreback added the API Design label Mar 1, 2017

jreback added this to the 0.20.0 milestone Mar 1, 2017

jreback force-pushed the exceptions branch 3 times, most recently from ff94598 to 4386409 Compare March 1, 2017 19:45

jreback force-pushed the exceptions branch 2 times, most recently from 998b171 to 56e8631 Compare March 8, 2017 14:04

jreback force-pushed the exceptions branch from 56e8631 to f0fb174 Compare March 9, 2017 19:19

jreback force-pushed the exceptions branch from 6efcc6d to b6947a1 Compare March 27, 2017 18:18

jreback force-pushed the exceptions branch from b6947a1 to 5c2543f Compare March 27, 2017 19:38

jorisvandenbossche reviewed Mar 27, 2017

View reviewed changes

jreback force-pushed the exceptions branch from 6c21f63 to 7f18b9a Compare April 2, 2017 22:58

chris-b1 reviewed Apr 3, 2017

View reviewed changes

jreback added 12 commits April 3, 2017 13:54

add pandas.api.lib

eec40cd

add infer_dtype to pandas.api.lib

API: expose pandas.errors

991fbb4

move all exception / warning impls (and references) from pandas.core.common and pandas.io.common closes pandas-dev#14800

corrections

92b2fdc

remove need for PandasError sub-class

7e8432d

DOC: better docs on infer_type

e91901d

more examples

35d225f

fix doc-string

5754630

remove AmbiguousIndexError, completely unused

2bb1fbd

more docs for exceptions

3dc4b9a

document removed exceptions

d636ef7

typo on pandas.errors in whatsnew

ab4525b

give nicer deprecation / message on infer_dtype moving

e5fbdc8

jreback force-pushed the exceptions branch from 7f18b9a to e5fbdc8 Compare April 3, 2017 18:13

jorisvandenbossche approved these changes Apr 3, 2017

View reviewed changes

jreback closed this in da0523a Apr 3, 2017

jorisvandenbossche mentioned this pull request Apr 5, 2017

API: pandas.api #13634

Closed

jorisvandenbossche mentioned this pull request Apr 18, 2017

API: scope of the public types module #16042

Closed

jreback mentioned this pull request Apr 23, 2017

API: expose dtypes in pandas.api.types #16099

Merged

jreback added a commit that referenced this pull request Apr 24, 2017

API: expose dtypes in pandas.api.types (#16099)

844013b

* API: expose dtypes in pandas.api.types xref #16015 xref apache/arrow#585 xref #16042 xref #15541 (comment)

pcluo pushed a commit to pcluo/pandas that referenced this pull request May 22, 2017

API: expose dtypes in pandas.api.types (pandas-dev#16099)

a4709e2

* API: expose dtypes in pandas.api.types xref pandas-dev#16015 xref apache/arrow#585 xref pandas-dev#16042 xref pandas-dev#15541 (comment)

		@@ -2,6 +2,7 @@

		import warnings
		warnings.warn("The pandas.lib module is deprecated and will be "

API: expose pandas.errors #15541

API: expose pandas.errors #15541

Conversation

jreback commented Mar 1, 2017 • edited Loading

jreback commented Mar 1, 2017 • edited Loading

jreback commented Mar 1, 2017 • edited Loading

TomAugspurger commented Mar 1, 2017 • edited Loading

jreback commented Mar 1, 2017

jorisvandenbossche commented Mar 1, 2017

jreback commented Mar 2, 2017

jreback commented Mar 3, 2017

jorisvandenbossche commented Mar 3, 2017

jreback commented Mar 3, 2017

shoyer commented Mar 3, 2017 via email

jreback commented Mar 3, 2017

shoyer commented Mar 3, 2017

jreback commented Mar 3, 2017 • edited Loading

jreback commented Mar 3, 2017

shoyer commented Mar 3, 2017

jreback commented Mar 3, 2017

jreback commented Mar 3, 2017

codecov-io commented Mar 8, 2017 • edited by codecov bot Loading

Codecov Report

jreback commented Mar 9, 2017

jreback commented Mar 9, 2017

wesm commented Mar 13, 2017

jorisvandenbossche commented Mar 13, 2017

jreback commented Mar 27, 2017

shoyer commented Mar 27, 2017

jreback commented Mar 27, 2017 • edited Loading

jreback commented Mar 27, 2017

jorisvandenbossche left a comment

Choose a reason for hiding this comment

jorisvandenbossche Mar 27, 2017

Choose a reason for hiding this comment

jreback Mar 27, 2017

Choose a reason for hiding this comment

jreback commented Mar 29, 2017

chris-b1 Apr 3, 2017

Choose a reason for hiding this comment

jreback Apr 3, 2017

Choose a reason for hiding this comment

jreback Apr 3, 2017

Choose a reason for hiding this comment

jreback Apr 3, 2017

Choose a reason for hiding this comment

jorisvandenbossche left a comment

Choose a reason for hiding this comment

jreback commented Mar 1, 2017 •

edited

Loading

jreback commented Mar 1, 2017 •

edited

Loading

jreback commented Mar 1, 2017 •

edited

Loading

TomAugspurger commented Mar 1, 2017 •

edited

Loading

jreback commented Mar 3, 2017 •

edited

Loading

codecov-io commented Mar 8, 2017 •

edited by codecov bot

Loading

jreback commented Mar 27, 2017 •

edited

Loading