Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG/API: fillna converts column of Timestamps to datetime64 #8189

Closed
TomAugspurger opened this issue Sep 5, 2014 · 2 comments
Closed

BUG/API: fillna converts column of Timestamps to datetime64 #8189

TomAugspurger opened this issue Sep 5, 2014 · 2 comments
Labels
Bug Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Timezones Timezone data dtype
Milestone

Comments

@TomAugspurger
Copy link
Contributor

Not sure if this is intentional / known.

In [58]: tz = pd.DatetimeIndex(start='2010-01-01', periods=10, freq='d', tz='US/Central')

In [59]: df = pd.DataFrame(np.random.randn(10, 2), columns=['A', 'B'])

In [60]: df['time'] = tz

In [62]: df.dtypes
Out[62]: 
A       float64
B       float64
time     object
dtype: object

In [63]: df.fillna(0).dtypes
Out[63]: 
A              float64
B              float64
time    datetime64[ns]
dtype: object

Notice that the dtype of the time column changes from object (column of Timestamps) to datetime64[ns].
I think what's happening is the timestamps are being downcast:

In [64]: df.fillna(0, downcast=False).dtypes
Out[64]: 
A       float64
B       float64
time     object
dtype: object

The reason I worry about this as the default is because the timestamp information is lost.

@jreback
Copy link
Contributor

jreback commented Sep 5, 2014

you have a timestamp with a timezone, it gets converted to utc on the fillna.

its prob a bug (this needs to preserve object dtype thruou and not infer it).

@jreback jreback added this to the 0.15.1 milestone Sep 6, 2014
@jreback
Copy link
Contributor

jreback commented Sep 6, 2014

actually this is a dupe of #7630

@jreback jreback closed this as completed Sep 6, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Timezones Timezone data dtype
Projects
None yet
Development

No branches or pull requests

2 participants