Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to read local files using browser (no-build) #43

Closed
arpanda opened this issue Jan 4, 2023 · 3 comments
Closed

How to read local files using browser (no-build) #43

arpanda opened this issue Jan 4, 2023 · 3 comments

Comments

@arpanda
Copy link

arpanda commented Jan 4, 2023

Hi,
I am trying to read a HDF5 file stored locally.
The file is available at "http://localhost:8080/examples/h5_files/test_data/test.h5". So providing the relative path as input for the file like /examples/h5_files/test_data/test.h5

Test code.
file: HDF5_Reader.js

import h5wasm from "https://cdn.jsdelivr.net/npm/h5wasm@0.4.0/dist/esm/hdf5_hl.js";

// the WASM loads asychronously, and you can get the module like this:
const Module = await h5wasm.ready;

class Read_HDF5{
    
    constructor(h5_file){
        self.h5_file = h5_file
    }
    
    async getKeys(){
        let f = new h5wasm.File(self.h5_file, "r");
        console.log(f.keys())
    }
}


export {Read_HDF5};
<!DOCTYPE html>
<html lang="en">
    <head>
        <title>View Hdf5</title>
        <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.6.0/jquery.min.js"></script>
    </head>
    
<body>
    <h1>View the keys of of HDF5 files</h1>
    
    <script type="module">
        import {Read_HDF5} from "./js/HDF5_Reader.js";
        let h5_file = "/examples/h5_files/test_data/test.h5";
        const h5_reader = new Read_HDF5(h5_file);
        h5_reader.getKeys();
    </script>

</body>
</html>

but getting error something like this.
image

@bmaranville
Copy link
Member

If you're loading files in the browser, you have to do a little bit more - (you have to load the file data to an Arraybuffer, then into a Uint8Array and write that to the virtual emscripten file system), If you're using the no-build browser example, every step in that section of the readme is needed in order to load a file from a URL!

I suppose it might be nice to consider adding a helper function

async h5wasm.fromURL(url: string): Promise<h5wasm.File>

that awaits a fetch, then writes to a random local file, and returns an h5wasm.File object but I don't have anything like that in the library at the moment.

@arpanda
Copy link
Author

arpanda commented Jan 4, 2023

Thanks. Hopefully, you will have something soon. I did the following and it can access the keys. 🎉 ✅

import h5wasm from "https://cdn.jsdelivr.net/npm/h5wasm@0.4.0/dist/esm/hdf5_hl.js";


class Read_HDF5{
    
    constructor(h5_file){
        this.h5_file = h5_file;
        this.random_name = "random.h5"
    }
    
    async getKeys(){
        // the WASM loads asychronously, and you can get the module like this:
        const Module = await h5wasm.ready;

        // then you can get the FileSystem object from the Module:
        const { FS } = Module;
        console.log(this.h5_file)
        let response = await fetch(this.h5_file)
        let ab = await response.arrayBuffer();

        FS.writeFile(this.random_name, new Uint8Array(ab));
        let f = new h5wasm.File(this.random_name, "r");
        console.log(f.keys())
        let data = f.get("12_rd_p");
        console.log(data)
        console.log(data.value) 
    }
}

export {Read_HDF5};

As it's write the data to the virtual emscripten file system, is there any limit for the input files. Will ~250 MB HDF5 file create any issue for the performance?

@bmaranville
Copy link
Member

You shouldn't have any issues with files that size. Files > 2GB will have issues with the max size of an ArrayBuffer in javascript. There are ways to read even larger files using HTTP range requests, or web workers and special emscripten filesystems (see discussion in #40 for instance), etc. but it gets more complicated. You can use direct filesystem random access with nodejs instead of a browser if you want to make a native tool/program without limits on file size.

@arpanda arpanda closed this as completed Jan 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants