[Python] Converting from NumPy to large_string or large_binary returns not implemented

[phofl]

Describe the bug, including details regarding any error messages, version, and platform.

arr = np.array(["a", "b"])
pa.array(arr, type=pa.large_string())

This returns

    pa.array(arr, type=pa.large_string())
  File "pyarrow/array.pxi", line 316, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array
  File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: NumPyConverter doesn't implement <large_string> conversion. 

I think this should work?

Component(s)

Python

[jorisvandenbossche]

Yes, there is indeed no specific reason for this to not work (apart from someone implementing it).

Looking at the code, this is default fallback because the NumPyConverter only is implemented for the non-large StringType. We have this:

arrow/python/pyarrow/src/arrow/python/numpy_to_arrow.cc

Lines 661 to 666 in c38c778

	Status NumPyConverter::Visit(const StringType& type) {
	util::InitializeUTF8();
	::arrow::internal::ChunkedStringBuilder builder(kBinaryChunksize, pool_);
	auto data = reinterpret_cast<const uint8_t*>(PyArray_DATA(arr_));

But no equivalent Visit(const LargeStringType& type). The implementation for StringType is based on ChunkedStringBuilder, which is a chunked version of StringBuilder. We already have LargeStringBuilder, so it should certainly be possible to add a ChunkedLargeStringBuilder as well, so we can template the NumPyConverter to work with both builders.