Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue: "Invalid flatbuffers message" Error When Opening Arrow File Generated by pg2arrow in Python #857

Open
a-hirota opened this issue Nov 29, 2024 · 2 comments

Comments

@a-hirota
Copy link

a-hirota commented Nov 29, 2024

Description
When attempting to open an Arrow file generated by pg2arrow in Python, the following error is raised:
pyarrow.version:16.1.0

pyarrow.lib.ArrowInvalid: Invalid flatbuffers message
@a-hirota
Copy link
Author

a-hirota commented Dec 2, 2024

The Arrow file could be successfully read using PyArrow versions from 0.0.1 to 14.0.2. But an error occurred starting with version from 15.0.0.

import pyarrow as pa
import pyarrow.ipc as ipc

file_path = '/home/kiban/gpu_dump/testtable_pg2arrow.arrow'

try:
    with open(file_path, 'rb') as f:
        reader = ipc.open_file(f)
        table = reader.read_all()
    print("Arrow Table Schema:")
    print(table.schema)
    print("Arrow Table as Pandas DataFrame:")
    print(table.to_pandas())
except pa.ArrowInvalid as e:
    print(f"Invalid Arrow file: {e}")
except Exception as e:
    print(f"Unexpected error: {e}")

pyarrow.version:0.0.1
pyarrow.version:14.0.2

Arrow Table Schema:
user_cd: decimal128(4, 0)
・・・・・・・・

@a-hirota
Copy link
Author

a-hirota commented Dec 2, 2024

The following versions might be related.

GH-35497 - [C++] Use the latest tagged version of flatbuffers (#38192)
GH-38823 - Fix TestArrowReaderAdHoc.ReadFloat16Files to use new uncompressed files (#38825)

https://arrow.apache.org/release/15.0.0.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant