Error: Can't synchronously read data while reading lzf compressed dataset

[PatrickHallek]

I am trying to read a Dataset that was compressed via lzf. It works great and reads out the data correctly without directly throwing an error.
However, I get the following log message:

HDF5-DIAG: Error detected in HDF5 (1.14.2) thread 0:
  #000: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5D.c line 1061 in H5Dread(): can't synchronously read data
    major: Dataset
    minor: Read failed
  #001: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5D.c line 1008 in H5D__read_api_common(): can't read data
    major: Dataset
    minor: Read failed
  #002: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5VLcallback.c line 2092 in H5VL_dataset_read_direct(): dataset read failed
    major: Virtual Object Layer
    minor: Read failed
  #003: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5VLcallback.c line 2048 in H5VL__dataset_read(): dataset read failed
    major: Virtual Object Layer
    minor: Read failed
  #004: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5VLnative_dataset.c line 363 in H5VL__native_dataset_read(): can't read data
    major: Dataset
    minor: Read failed
  #005: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5Dio.c line 383 in H5D__read(): can't read data
    major: Dataset
    minor: Read failed
  #006: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5Dchunk.c line 2856 in H5D__chunk_read(): unable to read raw data chunk
    major: Low-level I/O
    minor: Read failed
  #007: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5Dchunk.c line 4468 in H5D__chunk_lock(): data pipeline read failed
    major: Dataset
    minor: Filter operation failed
  #008: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5Z.c line 1356 in H5Z_pipeline(): required filter 'lzf' is not registered
    major: Data filters
    minor: Read failed
  #009: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5PLint.c line 267 in H5PL_load(): can't find plugin. Check either HDF5_VOL_CONNECTOR, HDF5_PLUGIN_PATH, default location, or path set by H5PLxxx functions
    major: Plugin for dynamically loaded library
    minor: Object not found
  #010: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5PLpath.c line 804 in H5PL__find_plugin_in_path_table(): search in path /usr/local/hdf5/lib/plugin encountered an error
    major: Plugin for dynamically loaded library
    minor: Can't get value
  #011: /__w/libhdf5-wasm/libhdf5-wasm/build/1.14.2/_deps/hdf5-src/src/H5PLpath.c line 857 in H5PL__find_plugin_in_path(): can't open directory (/usr/local/hdf5/lib/plugin). Please verify its existence
    major: Plugin for dynamically loaded library
    minor: Can't open directory or file

I created the hdf5 file with a python script using h5py.
Using gzip or no compression works without outputting this error.

I am using typescript with ts-node, this is my code:

import { Dataset, File } from 'h5wasm'

export const importDynamic = new Function('modulePath', 'return import(modulePath)')

const main = async () => {
    const { h5wasm } = await importDynamic('h5wasm/node')
    await h5wasm.ready
    const filePath = 'path/to/data.hdf5'
    const file: File = new h5wasm.File(filePath)
    const img = (file.get('image_1') as Dataset).to_array() as number[][]
    console.log(img.length, img[0].length) // This will output the correct image size without throwing
}
main()

[bmaranville]

Are you trying to load the LZF plugin? That one is not built into the base distribution of h5wasm, unlike GZIP and SZIP. There is a whole ecosystem of compression/data management plugins for HDF5. A subset of them has been compiled to webassembly for use with h5wasm here: https://github.com/h5wasm/h5wasm-plugins

Hopefully that will allow you to load your dataset without error! If not, please file an issue on that repository. (I also help maintain it)