BUG:to_numpy() returns numpy array with Object dtype, for multiple columns with the same dtype ("Float64" or "float64[pyarrow]").

[dpmcauliffe]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

df = pd.DataFrame(dict(a=np.linspace(0, 2, 4, dtype=np.float64), 
                       b=np.linspace(2, 4, 4, dtype=np.float64)))

# This behaves as expeceted
orig_2D_arr = df[['a', 'b']].to_numpy()
assert orig_2D_arr.dtype == np.float64

# Convert to "numpy_nullable" dtypes i.e. Float64 
df_np = df.convert_dtypes(dtype_backend="numpy_nullable")

# 2 Columns with the same dtype result in numpy array with 'Object' Dtype
np_backend_2_col_arr = df_np[['a', 'b']].to_numpy()
assert np_backend_2_col_arr.dtype != np.float64 # NOK - numpy dtype is 'Object' - not consistent with to_numpy() docs

# Single Column OK numpy array dtype == float64
np_backend_1_col_arr = df_np[['a']].to_numpy()
assert np_backend_1_col_arr.dtype == np.float64 # OK = numpy dtype is 'float64'

# Same Behaviour for pyarrow backend types
df_pa = df.convert_dtypes(dtype_backend="pyarrow")

# Single Column OK numpy array dtype == float64
pa_backend_1_col_arr = df_pa[['a']].to_numpy()
assert pa_backend_1_col_arr.dtype == np.float64 # OK = numpy dtype is 'float64'

# 2 Columns with the same dtype result in numpy array with 'Object' Dtype
pa_backend_2_col_arr = df_pa[['a', 'b']].to_numpy()
assert pa_backend_2_col_arr.dtype != np.float64  # NOK - numpy dtype is 'Object' - not consistent with to_numpy() docs

Issue Description

The array returned by the to_numpy() method has a dtype that is not consistent with the documentation, and in my interpretation is a bug.

When using the to_numpy() method on a DataFrame with 2 or more columns with the same dtype the returned numpy array has dtype Object.

This occurs when the columns are backed by numpy_nullable or pyarrow dtype backends.

For example, with 2 columns ['a', 'b'] with dtype float64[pyarrow]:

• df[['a', 'b']].to_numpy() - the dtype of the numpy array is of type Object
• df[['a']].to_numpy() - the dtype of the numpy array is of type np.float64

The documentation states:

Convert the DataFrame to a NumPy array.

By default, the dtype of the returned array will be the common NumPy dtype of all types in the DataFrame. For example, if the dtypes are float16 and float32, the results dtype will be float32. This may require copying data and coercing values, which may be expensive.

The observed behaviour is in contradiction with this - we get the expected numpy dtype for a single column, but an Objectdtype when called with multiple columns (of the same dtype)

Expected Behavior

When columns 'a' and 'b' have the same dtype we would expect:

df[['a', 'b']].to_numpy().dtype == df[['a']].to_numpy().dtype == df[['b']].to_numpy().dtype

And ideally when columns 'a' and 'b' have the same dtype, of either Float64 or float64[pyarrow]

df[['a', 'b']].to_numpy().dtype == np.float64

Installed Versions

INSTALLED VERSIONS

commit : 0691c5c
python : 3.12.3
python-bits : 64
OS : Darwin
OS-release : 23.6.0
Version : Darwin Kernel Version 23.6.0: Mon Jul 29 21:14:46 PDT 2024; root:xnu-10063.141.2~1/RELEASE_ARM64_T6031
machine : arm64
processor : arm
byteorder : little
LC_ALL : None
LANG : en_IE.UTF-8
LOCALE : en_IE.UTF-8

pandas : 2.2.3
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.9.0.post0
pip : 24.2
Cython : 3.0.10
sphinx : 8.0.2
IPython : 8.26.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
blosc : None
bottleneck : 1.4.0
dataframe-api-compat : None
fastparquet : None
fsspec : 2024.6.1
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.4
lxml.etree : None
matplotlib : 3.9.0
numba : 0.60.0
numexpr : None
odfpy : None
openpyxl : 3.1.5
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : 16.1.0
pyreadstat : None
pytest : 8.2.2
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.14.1
sqlalchemy : None
tables : None
tabulate : None
xarray : 2024.6.0
xlrd : None
xlsxwriter : None
zstandard : None
tzdata : 2024.1
qtpy : None
pyqt5 : None

[rhshadrach]

Thanks for the report, I think this is part of #32265