Non block testing fix #363

TonyB9000 · 2025-02-21T16:10:00Z

Summary

Unifies the non-blocking zstash behavior between both "create" and "update" operations.

Addresses issue #361,

…ing additions for activity tracing

forsyth2

@TonyB9000 I left some initial review comments. I want to spend more time studying the code to understand how everything gets called/passed around though.

forsyth2 · 2025-02-21T21:46:45Z

zstash/create.py

@@ -92,7 +92,7 @@ def create():

    # Transfer to HPSS. Always keep a local copy.
    logger.debug(f"{ts_utc()}: calling hpss_put() for {get_db_filename(cache)}")
-    hpss_put(hpss, get_db_filename(cache), cache, keep=True)
+    hpss_put(hpss, get_db_filename(cache), cache, keep=args.keep)


This is specifically for archiving the database. I think we do want to always keep that, no?

Yes, I agree. That was a mistake. (But it always seems to remain in any case - a mystery)

forsyth2 · 2025-02-21T22:03:35Z

zstash/create.py

@@ -169,9 +169,8 @@ def setup_create() -> Tuple[str, argparse.Namespace]:
    # Now that we're inside a subcommand, ignore the first two argvs
    # (zstash create)
    args: argparse.Namespace = parser.parse_args(sys.argv[2:])
-    if args.hpss and args.hpss.lower() == "none":
+    if not args.hpss or args.hpss.lower() == "none":


Parentheses just for clarity: if (not args.hpss) or (args.hpss.lower() == "none"):

args.hpss args.hpss.lower() == "none" args.non_blocking original behavior new behavior change

T T T args.hpss = "none", args.keep = True args.hpss = "none", args.keep = True N/A

T T F args.hpss = "none" args.hpss = "none", args.keep = True Sets args.keep = True

T F T args.keep = True Nothing No longer sets args.keep = True

T F F Nothing Nothing N/A

F N/A T args.keep = True args.hpss = "none", args.keep = True Sets args.hpss = "none"

F N/A F Nothing args.hpss = "none", args.keep = True Sets args.hpss = "none", args.keep = True

Can you confirm these are the expected changes in behavior?

How did you arrive that the first two rows? Nothing in that code involves the status of "non-blocking".

Correct me if I'm wrong, but testing "if args.hpss" would only fail if the user included no "hpss" argument on the command line. That should be the same as "hpss=none" (unless some hidden config sets it elsewhere - I did not consider that).

In any case, (to my knowledge), the only time we intend to FORCE "keep" is when hpss=none. According the the "help" text, there is nothing that "non-blocking" (True or False) does to effect "keep".

Thus, rows 3 and 4 should not be seeing "keep = True" if the user did not specify keep.

I was looking at the combined behavior of

if args.hpss and args.hpss.lower() == "none": args.hpss = "none" if args.non_blocking: args.keep = True

becoming

if not args.hpss or args.hpss.lower() == "none": args.hpss = "none" args.keep = True

only fail if the user included no "hpss" argument on the command line.

Correct, and I don't think that is possible because we set it as required:

required.add_argument( "--hpss", type=str, help=( 'path to storage on HPSS. Set to "none" for local archiving. It also can be a Globus URL, ' 'globus://<GLOBUS_ENDPOINT_UUID>/<PATH>. Names "alcf" and "nersc" are recognized as referring to the ALCF HPSS ' "and NERSC HPSS endpoints, e.g. globus://nersc/~/my_archive." ), required=True, )

Thus, rows 3 and 4 should not be seeing "keep = True" if the user did not specify keep.

Ok, that makes sense.

forsyth2 · 2025-02-21T22:09:59Z

zstash/globus.py

@@ -157,6 +157,7 @@ def file_exists(name: str) -> bool:
            return True
    return False

+gv_push = 0


Why gv_push? A more descriptive name might be better. Maybe tar_file_count?

True, but it was just a way for me to track things. We could change it.

I wanted a variable to track "actual transfer submitted" (pushed), as opposed to just submitted to our globus_transfer() function, which may just add it to a pending transfer and return. I'll make it "gv_tarfiles_pushed".

forsyth2 · 2025-02-21T22:11:11Z

zstash/globus.py

@@ -215,7 +218,7 @@ def globus_transfer(  # noqa: C901
            fail_on_quota_errors=True,
        )
    transfer_data.add_item(src_path, dst_path)
-    transfer_data["label"] = subdir_label + " " + filename
+    transfer_data["label"] = label


Note to self: label is defined to be exactly the same thing above already.

forsyth2 · 2025-02-21T22:14:22Z

zstash/hpss.py

+                    for src_path in prev_transfers:
+                        os.remove(src_path)
+                    prev_transfers = curr_transfers
+                    curr_transfers = list()


You can just use = [] instead of = list().

I used to do that - but was cautioned against it (don't recall why). I'd be happy either way.

Hmm interesting, I wonder why. = [] definitely seems more "pythonic" to me, as is echoed on https://stackoverflow.com/questions/5790860/whats-the-difference-between-and-vs-list-and-dict.

forsyth2 · 2025-02-21T22:15:45Z

zstash/update.py

@@ -107,8 +114,17 @@ def setup_update() -> Tuple[argparse.Namespace, str]:
        help="Hard copy symlinks. This is useful for preventing broken links. Note that a broken link will result in a failed update.",
    )
    args: argparse.Namespace = parser.parse_args(sys.argv[2:])
-    if args.hpss and args.hpss.lower() == "none":
+
+    if not args.hpss or args.hpss.lower() == "none":


Parentheses, as in create, would be good: if (not args.hpss) or (args.hpss.lower()) == "none":

True. I was relying upon the default ("not" applies only the the very next argument). Also to the shortcut-pass where testing (A or B) never tests B when A is true, as it is unnecessary (useful when testing B might cause an exception.

I added the parentheses.

Also to the shortcut-pass where testing (A or B) never tests B when A is true, as it is unnecessary (useful when testing B might cause an exception.

Yes, the parentheses are only for human readers. They shouldn't affect the code at all.

forsyth2 · 2025-02-21T22:15:57Z

zstash/update.py

        args.hpss = "none"
+        args.keep - True


Ah! That will make a difference! :) Good catch!

TonyB9000 · 2025-02-21T23:25:10Z

@forsyth2 Allow me to make some changes to address the clear mistakes above. Should take just a moment.

forsyth2 · 2025-03-04T01:58:26Z

Allow me to make some changes to address the clear mistakes above. Should take just a moment.

@TonyB9000 Can you push those changes?

I've also reviewed the code logic; this looks good to me, aside from the already suggested changes.

Following the logic of the lists of transferred tars

hpss_utils.add_files -> hpss.hpss_put -> hpss.hpss_transfer:

        if transfer_type == "put":
            if not keep:
                if (scheme != "globus") or (
                    globus_status == "SUCCEEDED"
                ):
                    # Note: This is intended to fulfill the default removal of successfully-transfered
                    # tar files when keep=False, irrespective of non-blocking status
                    logger.info(f"{ts_utc()}: DEBUG: deleting transfered files {prev_transfers}")
                    for src_path in prev_transfers:
                        os.remove(src_path)
                    prev_transfers = curr_transfers
                    curr_transfers = list()

Globus succeeded. We don't have to worry about these tars anymore; they've been transferred.
Delete them and reset the lists.

Earlier in hpss.hpss_transfer, we saw:

curr_transfers.append(file_path)

which is how curr_transfers builds up the list of tars currently being transferred.

Following the logic of `gv_push`

In globus.globus_transfer:

        # DEBUG: review accumulated items in TransferData
        logger.info(f"{ts_utc()}: TransferData: accumulated items:")
        attribs = transfer_data.__dict__
        for item in attribs["data"]["DATA"]:
            if item["DATA_TYPE"] == "transfer_item":
                gv_push += 1
                print(f"   (routine)  PUSHING (#{gv_push}) STORED source item: {item['source_path']}", flush=True)

Increment for every transfer_item we encounter.

In globus.globus_finalize:

    if transfer_data:
        # DEBUG: review accumulated items in TransferData
        logger.info(f"{ts_utc()}: FINAL TransferData: accumulated items:")
        attribs = transfer_data.__dict__
        for item in attribs["data"]["DATA"]:
            if item["DATA_TYPE"] == "transfer_item":
                gv_push += 1
                print(f"    (finalize) PUSHING ({gv_push}) source item: {item['source_path']}", flush=True)

        # SUBMIT new transfer here
        logger.info(f"{ts_utc()}: DIVING: Submit Transfer for {transfer_data['label']}")

Again, increment for every transfer_item we encounter.

gv_push is only ever incremented, never reset to 0. From Tony:

I wanted a variable to track "actual transfer submitted" (pushed), as opposed to just submitted to our globus_transfer() function, which may just add it to a pending transfer and return.

So, gv_push simply counts the number of transfer_items encountered throughout the entire run.

forsyth2 · 2025-03-04T02:07:48Z

We'll also need to fix the pre-commit check before merging.

Anthony Bartoletti added 2 commits February 20, 2025 18:15

addressed non-blocking behavior for both create and update, many logg…

c1158ee

…ing additions for activity tracing

Reset maxsize to production value

86fbbd5

TonyB9000 requested a review from forsyth2 February 21, 2025 16:10

forsyth2 reviewed Feb 21, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Non block testing fix #363

Non block testing fix #363

TonyB9000 commented Feb 21, 2025

forsyth2 left a comment

forsyth2 Feb 21, 2025

TonyB9000 Feb 21, 2025

forsyth2 Feb 21, 2025

TonyB9000 Feb 21, 2025

forsyth2 Feb 21, 2025

forsyth2 Feb 21, 2025

forsyth2 Feb 21, 2025

TonyB9000 Feb 21, 2025

forsyth2 Feb 21, 2025

TonyB9000 Feb 21, 2025

forsyth2 Feb 21, 2025

TonyB9000 Feb 21, 2025

forsyth2 Feb 21, 2025

forsyth2 Feb 21, 2025

TonyB9000 Feb 22, 2025

forsyth2 Feb 24, 2025

forsyth2 Feb 21, 2025

TonyB9000 Feb 21, 2025

TonyB9000 commented Feb 21, 2025

forsyth2 commented Mar 4, 2025

forsyth2 commented Mar 4, 2025

`args.hpss`	`args.hpss.lower() == "none"`	`args.non_blocking`	original behavior	new behavior	change
T	T	T	`args.hpss = "none"`, `args.keep = True`	`args.hpss = "none"`, `args.keep = True`	N/A
T	T	F	`args.hpss = "none"`	`args.hpss = "none"`, `args.keep = True`	Sets `args.keep = True`
T	F	T	`args.keep = True`	Nothing	No longer sets `args.keep = True`
T	F	F	Nothing	Nothing	N/A
F	N/A	T	`args.keep = True`	`args.hpss = "none"`, `args.keep = True`	Sets `args.hpss = "none"`
F	N/A	F	Nothing	`args.hpss = "none"`, `args.keep = True`	Sets `args.hpss = "none"`, `args.keep = True`

Non block testing fix #363

Are you sure you want to change the base?

Non block testing fix #363

Conversation

TonyB9000 commented Feb 21, 2025

Summary

forsyth2 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TonyB9000 commented Feb 21, 2025

forsyth2 commented Mar 4, 2025

forsyth2 commented Mar 4, 2025