How does vaex read parquet file?

[yohplala]

Hi,
I have been trying to see how/where vaex reads parquet file in its sources.
I have found out that it seems to be located in
vaex/packages/vaex-core/vaex/arrow/dataset.py

In pyarrow.parquet, the method to read parquet files is read_table().
So i have been tryin gto Ctrl-F read_table, but I could not fin any call to this method.

Please, where/how in the source is vaex reading parquet file?

Thanks for your help.
Have a good day,
Bests,

[maartenbreddels]

Hi,

We're using the arrow dataset API, where we iterate over the fragments, and request the data using fragment.to_table(..)
Is there any problem you're working on?

cheers,

Maarten

[yohplala]

Is there any problem you're working on?

Hi @maartenbreddels, thanks for the quick answer!
No, no worries ;).

I am contributing to fastparquet project.
So far, these contributions have been 'feature oriented', but I have few works in mind to improve its speed.
When I am there, I will check what these improvements brings in term of speed.

Maybe (seeing long time in advance) a next step could have been to interface vaex to fastparquet to benefit from these speed-ups in vaex as well (pandas and dask are for instance both interfaced to it).

Well, only a far idea at the moment. I don't know if it does really worth it, as fastparquet issues data in pandas format, so to interface it with vaex, there would be a subsequent pandas to pyarrow table step.
Again, only a far idea at the moment.

[maartenbreddels]

Cool, let me know if you need some pointers to use fastparquet from vaex. Also, for high bandwidth communication feel free to hop on slack (see link in main README.md)