ENH: Feature request, date type

[zbrookle]

There currently isn’t any native date dtype in pandas, which makes it impossible to integrate this type into file formats like parquet, where schema is defined

[jbrockmendel]

Period[D] behaves a lot like date. Could that be adapted to solve the parquet issue?

[zbrookle]

That seems kind of like a work around, is there a real reason that pandas doesn’t have a date type? It’s a pretty essential and standard type for most data platforms and frameworks

[jreback]

That seems kind of like a work around, is there a real reason that pandas doesn’t have a date type? It’s a pretty essential and standard type for most data platforms and frameworks

@zbrookle you are welcome to spend the time to develop one

normalized Timestamps are pretty easy to understand and easily act like a date type

[zbrookle]

@jreback Okay awesome I’d definitely love to work on it

[jreback]

our current setup actually would be pretty reasonable to support a dtype and extension array of type

datetime64[D] (and maybe other freqs); may also have to be named slightly different

could be backed by a nullable integer array that represent ordinala from epoch

i don’t think it’s that hard actually

[jorisvandenbossche]

As reference, Apache Arrow has two different date types: https://github.com/apache/arrow/blob/21a6474fd6e4b5c0444d8364ebfdefcf856ef7fd/format/Schema.fbs#L144-L157
There is date64 (milliseconds since 1970 in 64bit) and date32 (days since 1970 in 32bit).

For pandas, having two different types feels a bit overkill though. And compatibility with numpy's datetime64[D] would also be nice, but so this doesn't map to any of the Arrow types (since the days are in 32bit).

[zbrookle]

@jorisvandenbossche I think the compatibility with NumPy is probably the most important since that's what the backend of pandas is. It wouldn't be too hard to have logic that converts to the appropriate pyarrow format when a dataframe writes to parquet. I think there would just have to be an error that would be raised if they tried to write out to a date that is beyond 32 bits, which I don't think will happened very frequently

[jorisvandenbossche]

I think the compatibility with NumPy is probably the most important since that's what the backend of pandas is

Compatibility with numpy is important, but I am not fully sure the numpy being the backend is important in this case. I think many of the datetime-specific functionalities are implemented in pandas, and are not coming from numpy (although I am not very much up to date with the exact details here)

[zbrookle]

@jorisvandenbossche I'm actually very confident that the backend for at least the datetime64 object in pandas is NumPy (because a lot of my implementation of the date was based off the datetime type) and so I think the biggest priority in terms of conversion is that these two types at the very least work well and efficiently together

[jorisvandenbossche]

Yes, we use the numpy datetime64 dtype, but that doesn't mean we use much of numpy to do datetime-related things with it (in the end, it's just an int64 array with an annotation).

Eg the code to get "fields" from a date (like the month, of the day of the month, etc): https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/tslibs/fields.pyx

[jbrockmendel]

I think many of the datetime-specific functionalities are implemented in pandas, and are not coming from numpy (although I am not very much up to date with the exact details here)

This is correct, largely because the relevant C functions in numpy are not exposed. If they become exposed (xref numpy/numpy#16364) i expect we'll start using them (once our minimum np version catches up) instead of having our own mostly-copy/pasted versions.

[jbrockmendel]

Eg the code to get "fields" from a date (like the month, of the day of the month, etc): https://github.com/pandas-dev/pandas/blob/master/pandas/_libs/tslibs/fields.pyx

Those functions all assume the int64s being passed represent nanosecond unit timestamps, which wouldn't apply here. What would apply is wrapping Period[D]

[zbrookle]

I’m aware that the actual date functions and abstractions are not through NumPy, but the backing array containing the data is, and that's what will impact conversion between the different data types within pandas using the .astype method. I think the only real decision here is whether to have the DateDtype backed by an int32 array or an int64 array.

• If it's an int32 array, then when converting to another type of date, there would need to be a type conversion in addition to the conversion from datetime epoch to date epoch
• If it's an int64 array, then only the epoch conversion would be needed, but additional conversion and handling would be necessary for pyarrow
  So really it's a trade off between pyarrow and numpy, so whatever you all think is most important I'll use

[tswast]

Copying my comment from here: googleapis/python-db-dtypes-pandas#30

[In the db-dtypes package], we did try datetime64[D]. It made the conversion to timestamp much more difficult. Also, I suspect adding time to date to get a timestamp will be a common operation, so the numpy datetime64[ns] backing array was a better fit for that too.

[jbrockmendel]

we now have pyarrow-backed date dtypes. im curious if that handles this use case

[mroeschke]

I think this is handled by the ArrowDtype pa.date type so closing