Big-endian data is wrongly decoded and therefore wrongly plotted

[RubelMozumder]

Describe the bug

The values of numpy array elements are being change abruptly. For example, the real numpy arrays are [[-2.6798e-08, -2.67987e-08, -2.67994e-08, -2.68001e-08..]] which is represented as [[3.205e+18, −8.250e+20, ...]]. The plot does not make any sense and does not contain any important information.

I tried to check this file in h5wasm https://h5web.panosc.eu/h5wasm and got the same behavior as mentioned above.

To Reproduce

Here in the attached zip file I have attache a screen shot of a data set plotted by matplotlib.
And a nexus output file (.nxs) which can be plot by h5wasm web application.

Degug-20230418T085917Z-001.zip

Expected behaviour

The expected behavior for a data set from several datasets has been added in the zip file.

Screenshots

In zip file

Context

• OS: [Ubuntu 20.04]
• Browser: [Chrome, firefox]
• Version: [Version 112.0.5615.49 (Official Build) (64-bit),
  Version 112.0 (64-bit)]

[RubelMozumder]

@axelboc, Please take a look on it. Thanks.

[loichuder]

Hello,

This is indeed a problem with h5wasm. I will report it to the maintainer and see what we can do.

The file appears to be displayed fine with h5grove: https://h5web.panosc.eu/h5grove?file=spm_test_output.test.nxs

[loichuder]

To be more precise about this, the issue stems from the fact that the data is stored with big-endian floats.

So it is possible to circumvent the issue by storing the data with little-endian floats instead.

[bmaranville]

I think it's probably most efficient to let the HDF library handle this conversion - I can try to make sure all values are read out to little-endian (even if stored big-endian)

[loichuder]

Up to you, I'll leave this to your expertise 😄

[RubelMozumder]

Hello,

This is indeed a problem with h5wasm. I will report it to the maintainer and see what we can do.

The file appears to be displayed fine with h5grove: https://h5web.panosc.eu/h5grove?file=spm_test_output.test.nxs

@loichuder, Thanks a lot for such a repartee. Still, there are at least two issue to be resolved:

1. The plot rendered by h5grove is not even near to the plot generated by (please have a look on the screen short) impshow() function of maplotlib. As fur as I can understand the plot algorithm in h5grove can't differentiate among different points with little difference in magnitude.
2. Fact, I mentioned the h5wasm for reproducibility of problem. Is there any way to solve it in my local installation of h5web application.

[loichuder]

The plot rendered by h5grove is not even near to the plot generated by (please have a look on the screen short) impshow() function of maplotlib. As fur as I can understand the plot algorithm in h5grove can't differentiate among different points with little difference in magnitude.

Not sure I understand your point here. I redid the plot with matplotlib with Viridis, a colormap also available in H5Web, and both plots look quite similar (left: matplotlib, right: H5Web):

Fact, I mentioned the h5wasm for reproducibility of problem. Is there any way to solve it in my local installation of h5web application.

It depends in which way you are using H5Web:

• In NOMAD: correct me @sanbrock but I think it uses h5grove under the hood so it should not have any issues with big-endians
• In JupyterLab: same thing for jupyterlab-h5web
• In VSCode: vscode-h5web uses h5wasm so we have to wait for a fix in h5wasm

[RubelMozumder]

Thanks for support. Yes, I got the plot just now but in Firefox, but when I was writing my last comment, I check it in Chrome. The right->Firefox and left->Chrome. In that case there a new issue for different browsers.

The original issue has been evolved in jupyter-lab. Here, is the python lib installation relevant with jupyterlab and h5*:

pip freeze | grep 'h5\|jup'
h5grove==1.2.0
h5py==3.8.0
h5py-wrapper==1.1.0
jupyter==1.0.0
jupyter-console==6.6.3
jupyter-events==0.6.3
jupyter-server==1.23.6
jupyter-ydoc==0.2.2
jupyter_client==8.0.3
jupyter_core==5.2.0
jupyter_server_fileid==0.8.0
jupyter_server_ydoc==0.6.1
jupyterlab==3.6.1
jupyterlab-h5web==7.0.0
jupyterlab-pygments==0.2.2
jupyterlab-widgets==3.0.5
jupyterlab_server==2.20.0

The below screen-shot from Jupyterlab:

[RubelMozumder]

Thanks for support. Yes, I got the plot just now but in Firefox, but when I was writing my last comment, I check it in Chrome. The right->Firefox and left->Chrome. In that case there a new issue for different browsers.

The original issue has been evolved in jupyter-lab. Here, is the python lib installation relevant with jupyterlab and h5*:

pip freeze | grep 'h5\|jup'
h5grove==1.2.0
h5py==3.8.0
h5py-wrapper==1.1.0
jupyter==1.0.0
jupyter-console==6.6.3
jupyter-events==0.6.3
jupyter-server==1.23.6
jupyter-ydoc==0.2.2
jupyter_client==8.0.3
jupyter_core==5.2.0
jupyter_server_fileid==0.8.0
jupyter_server_ydoc==0.6.1
jupyterlab==3.6.1
jupyterlab-h5web==7.0.0
jupyterlab-pygments==0.2.2
jupyterlab-widgets==3.0.5
jupyterlab_server==2.20.0

The below screen-shot from Jupyterlab:

note that, the same plot has been found for jupyterlab in Chrome and Firefox.

[bmaranville]

There's a fix in the main branch of h5wasm, but I haven't made a release yet (there are some other new features, and I haven't written the documentation yet)

[bmaranville]

v0.4.11 released just now should fix the problem

[loichuder]

Thanks for support. Yes, I got the plot just now but in Firefox, but when I was writing my last comment, I check it in Chrome. The right->Firefox and left->Chrome. In that case there a new issue for different browsers.

The original issue has been evolved in jupyter-lab.

The below screen-shot from Jupyterlab

Oooh, interesting. The same issue found for h5wasm is indeed happening in JupyterLab, whatever the browser. I can reproduce it as well.

I will have a look at it.

[loichuder]

@RubelMozumder It should be fixed by the latest Jupyterlab-h5web version (7.1.1)

Thanks for the report !

[RubelMozumder]

@loichuder Thank you very much. I hope it will be fixed and I will try it. Unfortunately, I could not try it yet.