BUG: __getitem__ raise blank KeyError for IntervalIndex and missing keys

[phofl]

• closes ERR: missing key not displayed in exception message for IntervalIndex #27365
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

Without tolist error would look like KeyError: array([6])

[jbrockmendel]

Tracking this down, it looks like the KeyError raised in Loc._validate_read_indexer has the well-written version of the message, and it looks like it is special-casing Categoricalndex for no apparent reason (CategoricalIndex and IntervalIndex being the only two Index subclasses that do the check this PR is looking at).

Not yet totally sure, but it may be that we should always have this exception be raised in _validate_read_indexer

[phofl]

@jbrockmendel I was thinking along the same line. One problem is, that we arrive in _get_listlike_indexer sometimes through _getitem_iterable where raise_missing ist set to False. This leads to supressed KeyErrors in some cases when not raising in IntervalIndex implementation. If we could change this to True, we do not need the raise above. But comment says will stop working in the future, so I thought we can not change this now.

[jbrockmendel]

But comment says will stop working in the future, so I thought we can not change this now.

Can you track down when these comments got added? I think its likely that they are out of date.

[jbrockmendel]

@phofl i came to the conclusion that the future-behavior you mentioned was supposed to have already been implemented, so those comments should be removed/updated.

moreover, if we just remove IntervalIndex._convert_list_indexer altogether (and disable it from being called from CategoricalIndex._convert_list_indexer, we end up falling though to raise raise KeyError(f"None of [{key}] are in the [{axis_name}]") in Loc._validate_read_indexer, like we would for any other Index.

The remaining trouble is that the exception message isn't quite right for IntervalIndex bc it isnt a matter of the key not being present, but the key not being contained in any of the intervals.

[phofl]

@jbrockmendel ok great. Will remove the code paths and update this pr with it. Should come to this tomorrow evening

[phofl]

Removing

IntervalIndex._convert_list_indexer

completly does not work out of the box.

ser = Series(np.arange(5), IntervalIndex.from_breaks(np.arange(6)))
result = s.loc[[1.5, 2.5, 3.5]]

where result has index
idx=Index([1.5,2.5,3.5]) instead of intervals.

Additionally, if disable calling it from CategoricalIndex._convert_list_indexer

df = DataFrame({"A": range(10)})
s = pd.cut(df.A, 5)
df["B"] = s
df = df.set_index("B")
result = df.loc[[4]]

raises {KeyError}'a list-indexer must only include values that are in the categories' which is probably wrong because we have an interval containing 4?

[jbrockmendel]

completly does not work out of the box.

Thanks for giving it a try. I may have been overly optimistic about reducing code here.

[phofl]

cc @jbrockmendel
Looked a bit more into this and got a follow up question.

df = DataFrame({"A": range(10)})
s = pd.cut(df.A, 5)
df["B"] = s
df = df.set_index("B")
result = df.loc[[4]]

This raises because

Int64Index([4], dtype='int64').difference(IntervalIndex([(-0.009, 1.8], (1.8, 3.6], (3.6, 5.4], (5.4, 7.2], (7.2, 9.0]],  closed='right', dtype='interval[float64]')

returns

Int64Index([4], dtype='int64')

I am not that familiar with interval support in this operation, but this seems at least odd. I would have expected one of the following:

• Raising because incompatible
• Empty return

Is my assumption correct? Then fixing this would bring us one step closer to removing this as you proposed

[jbrockmendel]

This raises because

This example doesn't raise for me on master. Do you mean in this branch?

I am not that familiar with interval support in this operation, but this seems at least odd. I would have expected one of the following:

indexing on a CategoricalIndex is difficult, and I think not entirely consistent (pretty sure there's an issue about this).

IntervalIndex.get_loc/get_indexer behavior is pretty much unconnected to the .difference operation (though that would make sense for e.g. 2 Int64Indexes). So im not sure i understand the question.

[phofl]

Sorry, my question was rubbish without context.

If we remove

     if self.categories._defer_to_indexing:
            indexer = self.categories._convert_list_indexer(keyarr)
            return Index(self.codes).get_indexer_for(indexer)

from _convert_list_indexer in category.py as discussed above, my example raises. In this case it gets piped through to the code snippet posted above with the differences return.

[jbrockmendel]

from _convert_list_indexer in category.py as discussed above, my example raises. In this case it gets piped through to the code snippet posted above with the differences return.

yah lets ignore this as orthogonal to this PR. can revisit later if we get ambitious about code simplification.

[jreback]

this looks good, can you merge master @phofl if we want to open an issue about the actual error message ok by me.

[jreback]

thanks @phofl very nice, keep em coming (as you have been!)

[simonjayhawkins]

if we want to open an issue about the actual error message ok by me.

#25996 already exists for key error messages in general.

[phofl]

Thanks very much, its fun working with you all