ENH: add NA scalar for missing value indicator, use in StringArray. #29597
Conversation
pandas/core/na_scalar.py
Outdated
|
|
||
| def _create_binary_propagating_op(name): | ||
| def method(self, other): | ||
| if isinstance(other, numbers.Number) or other is NA or isinstance(other, str): |
There was a problem hiding this comment.
Question is what type of objects we should recognize here.
I think we want to recognize scalar values that can be stored in dataframes. But of course we don't have control over what kind of scalars people put in a dataframe.
For now I did numbers + string (numbers also covers numpy scalars). Our own scalars (Timedelta, Timestamp, etc) could also be added, or they can also handle that on their side)
There was a problem hiding this comment.
you might want to look at how NaT handles these, in particular for arraylike others
There was a problem hiding this comment.
Do we want it to be recognized by arrays? (I somewhat purposefully left those out, but didn't think it fully through)
For our own internal arrays (eg IntegerArray), that can be handled on the array level? (so returning NotImplemented here)
For numpy arrays the question is what to do with it, as you cannot represent NAs in a numpy array (except in an object array). So not handling it might be fine?
There was a problem hiding this comment.
For numpy arrays the question is what to do with it, as you cannot represent NAs in a numpy array (except in an object array). So not handling it might be fine?
Actually, by not handling it and deferring to numpy, this already happens (-> converting to object):
In [61]: np.array([1, 2, 3]) + pd.NA
Out[61]: array([NA, NA, NA], dtype=object)
In [62]: pd.NA + np.array([1, 2, 3])
Out[62]: array([NA, NA, NA], dtype=object)
There was a problem hiding this comment.
thats fine for now, but longer-term this will need to be optimized
There was a problem hiding this comment.
but longer-term this will need to be optimized
Can you explain what you mean with this?
| if isinstance(other, np.int64): | ||
| # for numpy scalars we get a deprecation warning and False as result | ||
| # for equality or error for larger/lesser than | ||
| continue |
There was a problem hiding this comment.
So numpy scalars we don't have full control over, so this means that if they are the left operand, we get some other behaviour:
In [27]: np.int64(1) == pd.NA
/home/joris/miniconda3/envs/dev/bin/ipython:1: DeprecationWarning: elementwise comparison failed; this will raise an error in the future.
#!/home/joris/miniconda3/envs/dev/bin/python
Out[27]: False
In [28]: pd.NA == np.int64(1)
Out[28]: NA
In [29]: np.int64(1) < pd.NA
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-29-87134fac2734> in <module>
----> 1 np.int64(1) < pd.NA
~/scipy/pandas/pandas/core/na_scalar.py in __bool__(self)
37
38 def __bool__(self):
---> 39 raise TypeError("boolean value of NA is ambiguous")
40
41 def __hash__(self):
TypeError: boolean value of NA is ambiguous
In [30]: pd.NA > np.int64(1)
Out[30]: NA
(for the first case, not sure what the behaviour will be once the change in numpy is done)
|
Any chance you could change StringDtype.na_value to use this, and see what breaks? There are a few places where we use I suspect that |
pandas/core/na_scalar.py
Outdated
| elif other is False or other is NA: | ||
| return NA | ||
| else: | ||
| return NotImplemented |
There was a problem hiding this comment.
other NA objects? NaT? NaN? None?
There was a problem hiding this comment.
For logical ops, I am not sure it should handle those, as logical ops typically involve booleans (and NaT/NaN/None cannot be stored in a boolean array)
Eg for float is also raises right now:
In [69]: pd.NA | 1.5
...
TypeError: unsupported operand type(s) for |: 'NAType' and 'float'
In [70]: pd.NA | np.nan
...
TypeError: unsupported operand type(s) for |: 'NAType' and 'float'
(for comparison ops, though, we should probably add those)
|
core.missing and _libs.missing will need to recognize this |
|
Playing with using this in StringArray, will push something (here or a PR against this) in a bit. Anecdotally, having (Pdb) pp left_value
<StringArray>
[NA, NA]
Length: 2, dtype: string
(Pdb) np.all(left_value == right_value)
*** TypeError: boolean value of NA is ambiguousBut maybe that's a good thing. I'd like to develop some more experience here. |
|
@jbrockmendel I moved it to cython (well, still mostly copy paste of the python version into the cython file) and let missing functions recognize this (isna), so feedback on that part is very welcome |
We will need to rethink any/all in general as well. In principle, those are reducing operations, and reductions skip NAs, so your example should be equivalent to |
|
If anyone wants a preview of what getting this working with StringArray looks like: https://github.com/jorisvandenbossche/pandas/pull/1/files. It'll mean a larger diff, but I think we'll want to merge that PR against Joris' branch into here, so that we can get a feel for what actually using this is going to look like before merging it into master. |
util.is_nan is specifically for float-nan. The scalar-recognizing functions that need to recognize this are in _libs.missing. |
|
Joris merged in my changes to make StringArray use I want to call out one behavior change we may want to make: Right now, I'm proposing that we change that behavior to always return an integer-na dtype in those cases for |
+1 It's also not a "breaking" change, as string dtype is still new. In general, with those new dtypes I think we should try to return as much as possible also new dtypes for the result (eg also for boolean results, once boolean array has landed) |
|
Just to scope this PR, I think that using |
pandas/_libs/missing.pyx
Outdated
| def _create_binary_propagating_op(name, divmod=False): | ||
|
|
||
| def method(self, other): | ||
| if isinstance(other, numbers.Number) or other is NA or isinstance(other, str): |
There was a problem hiding this comment.
can you put the numbers.Number check last, as it will be least performant
|
I added some initial docs (whatsnew + section in missing_data |
|
@jorisvandenbossche I've looked at your doc changes. You may want to say something about that when you read data, the new |
| This also means that ``pd.NA`` cannot be used in a context where it is | ||
| evaluated to a boolean, such as ``if condition: ...`` where ``condition`` can | ||
| potentially be ``pd.NA``. In such cases, :func:`isna` can be used to check | ||
| for ``pd.NA`` or it could be prevented that ``condition`` can be ``pd.NA`` |
There was a problem hiding this comment.
| for ``pd.NA`` or it could be prevented that ``condition`` can be ``pd.NA`` | |
| for ``pd.NA`` or ``condition`` being ``pd.NA`` can be avoided, for example by filling missing values beforehand. |
(can't apply this directly, since it wraps two lines)
|
cc @pandas-dev/pandas-core more comments on this? |
|
|
||
| Starting from pandas 1.0, an experimental ``pd.NA`` value (singleton) is | ||
| available to represent scalar missing values. At this moment, it is used in | ||
| the nullable integer and boolean data types and the dedicated string data type |
There was a problem hiding this comment.
use commas rather than 3 ands, ideally if you can link to those sections?
| pd.NA | True | ||
|
|
||
| On the other hand, if one of the operands is ``False``, the result depends | ||
| on the value of the other operand. Therefore, in thise case ``pd.NA`` |
| ``NA`` in a boolean context | ||
| --------------------------- | ||
|
|
||
| Since the actual value of an NA is unknown, it is ambiguous to convert NA |
There was a problem hiding this comment.
I think we have a section gotchas.truth that is very similiar, could link.
|
Updated for the doc comments, thanks! @Dr-Irv I added a sentence on that. But we should probably add a more extensive section on that in a page dedicated to those new data types (for a follow-up). @jreback I run the |
|
thanks @jorisvandenbossche nice addition. unless I missed this, you didn't update nullable integers to return that, assume that's a followon. |
Yes, as being discussed, see #29556 for all follow-ups |
…ndexing-1row-df * upstream/master: (49 commits) repr() (pandas-dev#29959) DOC : Typo fix in userguide/Styling (pandas-dev#29956) CLN: small things in pytables (pandas-dev#29958) API/DEPR: Change default skipna behaviour + deprecate numeric_only in Categorical.min and max (pandas-dev#27929) DEPR: DTI/TDI/PI constructor arguments (pandas-dev#29930) CLN: fix pytables passing too many kwargs (pandas-dev#29951) Typing (pandas-dev#29947) repr() (pandas-dev#29948) repr() (pandas-dev#29950) Added space at the end of the sentence (pandas-dev#29949) ENH: add NA scalar for missing value indicator, use in StringArray. (pandas-dev#29597) CLN: BlockManager.apply (pandas-dev#29825) TST: add test for rolling max/min/mean with DatetimeIndex over different frequencies (pandas-dev#29932) CLN: explicit signature for to_hdf (pandas-dev#29939) CLN: make kwargs explicit for pytables read_ methods (pandas-dev#29935) Convert core/indexes/base.py to f-strings (pandas-dev#29903) DEPR: dropna multiple axes, fillna int for td64, from_codes with floats, Series.nonzero (pandas-dev#29875) CLN: make kwargs explicit in pytables constructors (pandas-dev#29936) DEPR: tz_convert in the Timestamp constructor raises (pandas-dev#29929) STY: F-strings and repr (pandas-dev#29938) ...
xref #28095, #28778
This PR adds a
pd.NAsingleton with the behaviour as discussed in above issues. For now, it's only used in StringArray in this PR.