Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix in python API: #4331

Merged
merged 3 commits into from
Aug 30, 2024
Merged

Conversation

pnorbert
Copy link
Contributor

a. BlocksInfo(name, step)
b. stream.write() sets selection if either 'start' or 'count' is provided (fix for local arrays).

This code below shows the original problem when enumerating the blocks of the v3 variable in the localArray example, where the number of blocks changes across the steps.

#!/usr/bin/env python3
import numpy as np
import adios2

def process_var(varname):
    print("--------------------------")
    print(f"Process variable {varname}")
    v = f.inquire_variable(varname)
    # b = f.all_blocks_info(varname)
    for step in range(v.steps()):
        v.set_step_selection([step, 1])
        bs = f.engine.blocks_info(varname, step)
        for bid in range(len(bs)):   # could use here len(b[step])
            v.set_step_selection([step, 1])
            v.set_block_selection(bid)
            data = f.read(v)
            print(f"step {step} block {bid} len = {len(data)}")


with adios2.FileReader("localArray.bp") as f:
    process_var('v0')
    process_var('v1')
    process_var('v2')
    process_var('v3')

@pnorbert pnorbert requested a review from lizdulac August 29, 2024 20:31
@pnorbert pnorbert requested a review from vicentebolea as a code owner August 29, 2024 20:31
@vicentebolea vicentebolea removed their request for review August 29, 2024 20:43
@@ -315,7 +315,7 @@ def _(self, name, content, shape=[], start=[], count=[], operations=None):
if shape != [] and not variable.single_value():
variable.set_shape(shape)

if start != [] and count != []:
if start != [] or count != []:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought start and count need to match in length

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For local arrays, start is empty, just like shape.

@pnorbert pnorbert requested review from anagainaru and removed request for lizdulac August 30, 2024 13:03
…election if either 'start' or 'count' is provided (fix for local arrays).
@pnorbert pnorbert force-pushed the fix-python-block-info branch from b5c1e6c to f0f6f52 Compare August 30, 2024 13:05
@pnorbert pnorbert merged commit fa5d3aa into ornladios:master Aug 30, 2024
37 of 38 checks passed
@pnorbert pnorbert deleted the fix-python-block-info branch August 30, 2024 15:02
@pnorbert pnorbert added this to the v2.10.2 milestone Oct 1, 2024
vicentebolea pushed a commit to vicentebolea/ADIOS2 that referenced this pull request Oct 23, 2024
* Fix in python API: a. BlocksInfo(name, step)  b. stream.write() sets selection if either 'start' or 'count' is provided (fix for local arrays).

* format

* black

(cherry picked from commit fa5d3aa)
vicentebolea pushed a commit to vicentebolea/ADIOS2 that referenced this pull request Oct 23, 2024
* Fix in python API: a. BlocksInfo(name, step)  b. stream.write() sets selection if either 'start' or 'count' is provided (fix for local arrays).

* format

* black

(cherry picked from commit fa5d3aa)
vicentebolea pushed a commit to vicentebolea/ADIOS2 that referenced this pull request Oct 23, 2024
* Fix in python API: a. BlocksInfo(name, step)  b. stream.write() sets selection if either 'start' or 'count' is provided (fix for local arrays).

* format

* black

(cherry picked from commit fa5d3aa)
vicentebolea pushed a commit to vicentebolea/ADIOS2 that referenced this pull request Oct 23, 2024
* Fix in python API: a. BlocksInfo(name, step)  b. stream.write() sets selection if either 'start' or 'count' is provided (fix for local arrays).

* format

* black

(cherry picked from commit fa5d3aa)
vicentebolea pushed a commit to vicentebolea/ADIOS2 that referenced this pull request Oct 24, 2024
* Fix in python API: a. BlocksInfo(name, step)  b. stream.write() sets selection if either 'start' or 'count' is provided (fix for local arrays).

* format

* black

(cherry picked from commit fa5d3aa)
vicentebolea pushed a commit to vicentebolea/ADIOS2 that referenced this pull request Oct 24, 2024
* Fix in python API: a. BlocksInfo(name, step)  b. stream.write() sets selection if either 'start' or 'count' is provided (fix for local arrays).

* format

* black

(cherry picked from commit fa5d3aa)
vicentebolea pushed a commit to vicentebolea/ADIOS2 that referenced this pull request Oct 24, 2024
* Fix in python API: a. BlocksInfo(name, step)  b. stream.write() sets selection if either 'start' or 'count' is provided (fix for local arrays).

* format

* black

(cherry picked from commit fa5d3aa)
vicentebolea added a commit that referenced this pull request Oct 26, 2024
* release_210: (33 commits)
  Bump version to v2.10.2
  Merge pull request #4379 from vicentebolea/increace-cmake-version-dep-blosc
  Fix missing int64_t definition (#4365)
  Merge pull request #4346 from anagainaru/memSelLayout
  Merge pull request #4343 from anagainaru/gpuSpan
  Compress the lossless part of MGARD data with Zstd. It was uncompressed until now. (#4339)
  Fix an issue when reading blocks in streaming mode, when some producers did not write any block to output. (#4332)
  Fix in python API:  (#4331)
  Merge pull request #4303 from anagainaru/build-frontier
  Merge pull request #4293 from anagainaru/check_parser_lexer
  Derived variables of type StoreData should not record the expression string (#4247)
  Update ReadMe.md (#4243)
  Switch from ROCM_VERSION_MAJOR to HIP_VERSION_MAJOR (#4222)
  Merge pull request #4207 from anagainaru/hide-symbols
  Patch for when both xrootd and derived variables are activated (#4200)
  Add defines for BP3, BP4 and BP5 (#4191)
  Overload NdCopy instead of adding a new argument for the 2.10 release
  Revert "Backport #3759 to Release 2.10 branch. (cherry-picked from master branch #4340) (#4341)"
  Revert "Merge pull request #4189 from pnorbert/campaign-s3-support"
  ci,python: enable python win2022 serial build
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants