Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reading complex types with decimals inside with 'hive.parquet-batch-read-optimization-enabled=true' results in error #23877

Closed
denodo-research-labs opened this issue Oct 23, 2024 · 0 comments · Fixed by #24440
Labels

Comments

@denodo-research-labs
Copy link
Contributor

denodo-research-labs commented Oct 23, 2024

Reading a Parquet file with complex types with decimals inside with hive.parquet-batch-read-optimization-enabled is true results in error.
It works when hive.parquet-batch-read-optimization-enabled is false.

Parquet definition of the complex type with decimals inside:

as_array_big_decimal: OPTIONAL F:1
.list:                REPEATED F:1
..element:            OPTIONAL INT32 L:DECIMAL(1,0) R:1 D:3

Your Environment

  • Presto 0.289
  • Storage S3
  • Connector used: Hive connector
  • Trace:

java.lang.UnsupportedOperationException: com.facebook.presto.common.block.IntArrayBlock at com.facebook.presto.common.block.Block.getLong(Block.java:81) at com.facebook.presto.common.type.ShortDecimalType.getObjectValue(ShortDecimalType.java:77) at com.facebook.presto.common.type.ArrayType.arrayBlockToObjectValues(ArrayType.java:160) at com.facebook.presto.common.type.ArrayType.lambda$getObjectValue$0(ArrayType.java:147) at com.facebook.presto.common.block.AbstractArrayBlock.apply(AbstractArrayBlock.java:298) at com.facebook.presto.common.type.ArrayType.getObjectValue(ArrayType.java:147) at com.facebook.presto.server.protocol.RowIterable$RowIterator.computeNext(RowIterable.java:77) at com.facebook.presto.server.protocol.RowIterable$RowIterator.computeNext(RowIterable.java:50) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)

Expected Behavior

Vectorized reading should work.

Current Behavior

The following error is obtained:

[Error: java.lang.UnsupportedOperationException: com.facebook.presto.common.block.IntArrayBlock
    at com.facebook.presto.common.block.Block.getLong(Block.java:81)
    at com.facebook.presto.common.type.ShortDecimalType.getObjectValue(ShortDecimalType.java:77)...]

Steps to Reproduce

  1. CREATE TABLE "hive"."default".hive_data_reader_array_primitives (
    as_array_int array(integer),
    as_array_long array(bigint),
    as_array_byte array(tinyint),
    as_array_short array(smallint),
    as_array_boolean array(boolean),
    as_array_float array(real),
    as_array_double array(double),
    as_array_string array(varchar),
    as_array_binary array(varbinary),
    as_array_big_decimal array(decimal(1,0))
    ) WITH (FORMAT = 'PARQUET', external_location = 's3a://path/to/parquet/')

  2. Parquet file: https://github.com/prestodb/presto/blob/master/presto-delta/src/test/resources/delta_v3/data-reader-array-primitives/part-00000-13921577-6635-457e-9e3a-f32dc7fea982.c000.snappy.parquet

  3. Hive catalog properties with hive.parquet-batch-read-optimization-enabled=true

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant