Compress the lossless part of MGARD data with Zstd. #4339

pnorbert · 2024-09-09T19:16:13Z

It was uncompresed until now.

$ ctest -R MGARD.*/.*BP5.Serial
$ find testing/adios2/engine/bp/operations/bp5 -name "*bp" -exec du -sh {} \;
48K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD3DSel_0.00001.bp
40K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD1DSel_0.00001.bp
60K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD2D_0.01.bp
36K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD3D_0.00001.bp
44K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD2DSel_0.0001.bp
40K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD1D_0.00001.bp
36K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD3D_0.0001.bp
40K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD1DSel_0.0001.bp
32K     testing/adios2/engine/bp/operations/bp5/BPWRMGARDNull_0.01.bp
40K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD1D_0.001.bp
60K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD2D_0.00001.bp
32K     testing/adios2/engine/bp/operations/bp5/BPWRMGARDNull_0.001.bp
44K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD2DSel_0.01.bp
40K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD1DSel_0.001.bp
60K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD2D_0.001.bp
32K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD3D_0.001.bp
44K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD2DSel_0.001.bp
40K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD1D_0.0001.bp
40K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD1D_0.01.bp
44K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD3DSel_0.001.bp
44K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD3DSel_0.01.bp
32K     testing/adios2/engine/bp/operations/bp5/BPWRMGARDNull_0.0001.bp
32K     testing/adios2/engine/bp/operations/bp5/BPWRMGARDNull_0.00001.bp
60K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD2D_0.0001.bp
32K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD3D_0.01.bp
44K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD2DSel_0.00001.bp
40K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD1DSel_0.01.bp
44K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD3DSel_0.0001.bp

Before this commit:

$ find testing/adios2/engine/bp/operations/bp5 -name "*bp" -exec du -sh {} \;
352K    testing/adios2/engine/bp/operations/bp5/BPWRMGARD3DSel_0.00001.bp
176K    testing/adios2/engine/bp/operations/bp5/BPWRMGARD1DSel_0.00001.bp
60K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD2D_0.01.bp
96K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD3D_0.00001.bp
176K    testing/adios2/engine/bp/operations/bp5/BPWRMGARD2DSel_0.0001.bp
176K    testing/adios2/engine/bp/operations/bp5/BPWRMGARD1D_0.00001.bp
96K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD3D_0.0001.bp
176K    testing/adios2/engine/bp/operations/bp5/BPWRMGARD1DSel_0.0001.bp
100K    testing/adios2/engine/bp/operations/bp5/BPWRMGARDNull_0.01.bp
176K    testing/adios2/engine/bp/operations/bp5/BPWRMGARD1D_0.001.bp
60K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD2D_0.00001.bp
100K    testing/adios2/engine/bp/operations/bp5/BPWRMGARDNull_0.001.bp
176K    testing/adios2/engine/bp/operations/bp5/BPWRMGARD2DSel_0.01.bp
176K    testing/adios2/engine/bp/operations/bp5/BPWRMGARD1DSel_0.001.bp
60K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD2D_0.001.bp
96K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD3D_0.001.bp
176K    testing/adios2/engine/bp/operations/bp5/BPWRMGARD2DSel_0.001.bp
176K    testing/adios2/engine/bp/operations/bp5/BPWRMGARD1D_0.0001.bp
176K    testing/adios2/engine/bp/operations/bp5/BPWRMGARD1D_0.01.bp
352K    testing/adios2/engine/bp/operations/bp5/BPWRMGARD3DSel_0.001.bp
344K    testing/adios2/engine/bp/operations/bp5/BPWRMGARD3DSel_0.01.bp
100K    testing/adios2/engine/bp/operations/bp5/BPWRMGARDNull_0.0001.bp
100K    testing/adios2/engine/bp/operations/bp5/BPWRMGARDNull_0.00001.bp
60K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD2D_0.0001.bp
96K     testing/adios2/engine/bp/operations/bp5/BPWRMGARD3D_0.01.bp
176K    testing/adios2/engine/bp/operations/bp5/BPWRMGARD2DSel_0.00001.bp
176K    testing/adios2/engine/bp/operations/bp5/BPWRMGARD1DSel_0.01.bp
352K    testing/adios2/engine/bp/operations/bp5/BPWRMGARD3DSel_0.0001.bp

…ed until now. $ ctest -R MGARD.*/.*BP5.Serial $ find testing/adios2/engine/bp/operations/bp5 -name "*bp" -exec du -sh {} \; 48K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3DSel_0.00001.bp 40K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1DSel_0.00001.bp 60K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2D_0.01.bp 36K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3D_0.00001.bp 44K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2DSel_0.0001.bp 40K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1D_0.00001.bp 36K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3D_0.0001.bp 40K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1DSel_0.0001.bp 32K testing/adios2/engine/bp/operations/bp5/BPWRMGARDNull_0.01.bp 40K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1D_0.001.bp 60K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2D_0.00001.bp 32K testing/adios2/engine/bp/operations/bp5/BPWRMGARDNull_0.001.bp 44K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2DSel_0.01.bp 40K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1DSel_0.001.bp 60K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2D_0.001.bp 32K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3D_0.001.bp 44K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2DSel_0.001.bp 40K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1D_0.0001.bp 40K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1D_0.01.bp 44K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3DSel_0.001.bp 44K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3DSel_0.01.bp 32K testing/adios2/engine/bp/operations/bp5/BPWRMGARDNull_0.0001.bp 32K testing/adios2/engine/bp/operations/bp5/BPWRMGARDNull_0.00001.bp 60K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2D_0.0001.bp 32K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3D_0.01.bp 44K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2DSel_0.00001.bp 40K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1DSel_0.01.bp 44K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3DSel_0.0001.bp Before this commit: $ find testing/adios2/engine/bp/operations/bp5 -name "*bp" -exec du -sh {} \; 352K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3DSel_0.00001.bp 176K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1DSel_0.00001.bp 60K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2D_0.01.bp 96K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3D_0.00001.bp 176K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2DSel_0.0001.bp 176K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1D_0.00001.bp 96K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3D_0.0001.bp 176K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1DSel_0.0001.bp 100K testing/adios2/engine/bp/operations/bp5/BPWRMGARDNull_0.01.bp 176K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1D_0.001.bp 60K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2D_0.00001.bp 100K testing/adios2/engine/bp/operations/bp5/BPWRMGARDNull_0.001.bp 176K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2DSel_0.01.bp 176K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1DSel_0.001.bp 60K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2D_0.001.bp 96K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3D_0.001.bp 176K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2DSel_0.001.bp 176K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1D_0.0001.bp 176K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1D_0.01.bp 352K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3DSel_0.001.bp 344K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3DSel_0.01.bp 100K testing/adios2/engine/bp/operations/bp5/BPWRMGARDNull_0.0001.bp 100K testing/adios2/engine/bp/operations/bp5/BPWRMGARDNull_0.00001.bp 60K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2D_0.0001.bp 96K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3D_0.01.bp 176K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2DSel_0.00001.bp 176K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1DSel_0.01.bp 352K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3DSel_0.0001.bp

…ed until now. (ornladios#4339) $ ctest -R MGARD.*/.*BP5.Serial $ find testing/adios2/engine/bp/operations/bp5 -name "*bp" -exec du -sh {} \; 48K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3DSel_0.00001.bp 40K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1DSel_0.00001.bp 60K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2D_0.01.bp 36K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3D_0.00001.bp 44K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2DSel_0.0001.bp 40K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1D_0.00001.bp 36K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3D_0.0001.bp 40K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1DSel_0.0001.bp 32K testing/adios2/engine/bp/operations/bp5/BPWRMGARDNull_0.01.bp 40K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1D_0.001.bp 60K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2D_0.00001.bp 32K testing/adios2/engine/bp/operations/bp5/BPWRMGARDNull_0.001.bp 44K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2DSel_0.01.bp 40K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1DSel_0.001.bp 60K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2D_0.001.bp 32K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3D_0.001.bp 44K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2DSel_0.001.bp 40K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1D_0.0001.bp 40K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1D_0.01.bp 44K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3DSel_0.001.bp 44K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3DSel_0.01.bp 32K testing/adios2/engine/bp/operations/bp5/BPWRMGARDNull_0.0001.bp 32K testing/adios2/engine/bp/operations/bp5/BPWRMGARDNull_0.00001.bp 60K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2D_0.0001.bp 32K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3D_0.01.bp 44K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2DSel_0.00001.bp 40K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1DSel_0.01.bp 44K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3DSel_0.0001.bp Before this commit: $ find testing/adios2/engine/bp/operations/bp5 -name "*bp" -exec du -sh {} \; 352K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3DSel_0.00001.bp 176K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1DSel_0.00001.bp 60K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2D_0.01.bp 96K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3D_0.00001.bp 176K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2DSel_0.0001.bp 176K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1D_0.00001.bp 96K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3D_0.0001.bp 176K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1DSel_0.0001.bp 100K testing/adios2/engine/bp/operations/bp5/BPWRMGARDNull_0.01.bp 176K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1D_0.001.bp 60K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2D_0.00001.bp 100K testing/adios2/engine/bp/operations/bp5/BPWRMGARDNull_0.001.bp 176K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2DSel_0.01.bp 176K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1DSel_0.001.bp 60K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2D_0.001.bp 96K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3D_0.001.bp 176K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2DSel_0.001.bp 176K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1D_0.0001.bp 176K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1D_0.01.bp 352K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3DSel_0.001.bp 344K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3DSel_0.01.bp 100K testing/adios2/engine/bp/operations/bp5/BPWRMGARDNull_0.0001.bp 100K testing/adios2/engine/bp/operations/bp5/BPWRMGARDNull_0.00001.bp 60K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2D_0.0001.bp 96K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3D_0.01.bp 176K testing/adios2/engine/bp/operations/bp5/BPWRMGARD2DSel_0.00001.bp 176K testing/adios2/engine/bp/operations/bp5/BPWRMGARD1DSel_0.01.bp 352K testing/adios2/engine/bp/operations/bp5/BPWRMGARD3DSel_0.0001.bp (cherry picked from commit d1d871b)

* release_210: (33 commits) Bump version to v2.10.2 Merge pull request #4379 from vicentebolea/increace-cmake-version-dep-blosc Fix missing int64_t definition (#4365) Merge pull request #4346 from anagainaru/memSelLayout Merge pull request #4343 from anagainaru/gpuSpan Compress the lossless part of MGARD data with Zstd. It was uncompressed until now. (#4339) Fix an issue when reading blocks in streaming mode, when some producers did not write any block to output. (#4332) Fix in python API: (#4331) Merge pull request #4303 from anagainaru/build-frontier Merge pull request #4293 from anagainaru/check_parser_lexer Derived variables of type StoreData should not record the expression string (#4247) Update ReadMe.md (#4243) Switch from ROCM_VERSION_MAJOR to HIP_VERSION_MAJOR (#4222) Merge pull request #4207 from anagainaru/hide-symbols Patch for when both xrootd and derived variables are activated (#4200) Add defines for BP3, BP4 and BP5 (#4191) Overload NdCopy instead of adding a new argument for the 2.10 release Revert "Backport #3759 to Release 2.10 branch. (cherry-picked from master branch #4340) (#4341)" Revert "Merge pull request #4189 from pnorbert/campaign-s3-support" ci,python: enable python win2022 serial build ...

pnorbert added the backport:release label Sep 9, 2024

pnorbert requested a review from eisenhauer September 9, 2024 19:16

eisenhauer approved these changes Sep 9, 2024

View reviewed changes

pnorbert merged commit d1d871b into ornladios:master Sep 9, 2024
38 checks passed

pnorbert deleted the mgard-compress-lossless-part branch September 9, 2024 21:41

pnorbert added this to the v2.10.2 milestone Oct 1, 2024

vicentebolea mentioned this pull request Oct 23, 2024

Backport from master #4374

Merged

vicentebolea removed the backport:release label Oct 25, 2024

vicentebolea mentioned this pull request Oct 26, 2024

Bump version to v2.10.2 #4383

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compress the lossless part of MGARD data with Zstd. #4339

Compress the lossless part of MGARD data with Zstd. #4339

pnorbert commented Sep 9, 2024

Compress the lossless part of MGARD data with Zstd. #4339

Compress the lossless part of MGARD data with Zstd. #4339

Conversation

pnorbert commented Sep 9, 2024