Latest az cli fails to run download-batch command #21966

taoisu · 2022-04-06T22:55:36Z

%23%23%23 This is autogenerated. Please review and update as needed.

%23%23 Describe the bug

Command Name
az storage blob download-batch

Errors:

The command failed with an unexpected error. Here is the traceback:
'BlobServiceClient' object has no attribute 'exists'
Traceback (most recent call last):
  File "/opt/az/lib/python3.8/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 658, in execute
    raise ex
  File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 721, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 713, in _run_job
    return cmd_copy.exception_handler(ex)
  File "/opt/az/lib/python3.8/site-packages/azure/cli/command_modules/storage/__init__.py", line 385, in new_handler
    first(ex)
  File "/opt/az/lib/python3.8/site-packages/azure/cli/command_modules/storage/__init__.py", line 385, in new_handler
    first(ex)
  File "/opt/az/lib/python3.8/site-packages/azure/cli/command_modules/storage/_exception_handler.py", line 17, in file_related_exception_handler
    raise ex
  File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 692, in _run_job
    result = cmd_copy(params)
  File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 328, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/opt/az/lib/python3.8/site-packages/azure/cli/command_modules/storage/operations/blob.py", line 363, in storage_blob_download_batch
    source_blobs = collect_blobs(client, source_container_name, pattern)
  File "/opt/az/lib/python3.8/site-packages/azure/cli/command_modules/storage/util.py", line 16, in collect_blobs
    return [name for (name, _) in collect_blob_objects(blob_service, container, pattern)]
  File "/opt/az/lib/python3.8/site-packages/azure/cli/command_modules/storage/util.py", line 16, in <listcomp>
    return [name for (name, _) in collect_blob_objects(blob_service, container, pattern)]
  File "/opt/az/lib/python3.8/site-packages/azure/cli/command_modules/storage/util.py", line 31, in collect_blob_objects
    if blob_service.exists(container, pattern):
AttributeError: 'BlobServiceClient' object has no attribute 'exists'

%23%23 To Reproduce:
Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

Put any pre-requisite steps here...
az storage blob download-batch -d {} -s {} --auth-mode {} --account-name {} --subscription {} --pattern {}

%23%23 Expected Behavior

%23%23 Environment Summary

Linux-5.4.0-1063-azure-x86_64-with-glibc2.27, Ubuntu 18.04.6 LTS
Python 3.8.12
Installer: DEB

azure-cli 2.35.0

Extensions:
azure-cli-ml 1.37.0

Dependencies:
msal 1.17.0
azure-mgmt-resource 20.0.0

%23%23 Additional Context

The text was updated successfully, but these errors were encountered:

yonzhan · 2022-04-07T00:21:33Z

storage

evelyn-ys · 2022-04-07T03:38:37Z

Can you share which pattern you are using?

taoisu · 2022-04-07T03:44:00Z

Something as simple as "model.onnx".

It doesn't matter as far as I can tell, any pattern would fail. This just started to happen today, I was using azure cli in a docker image, I built the image yesterday it was working just fine, today I re-built the docker image the error started to show up.

evelyn-ys · 2022-04-07T05:35:58Z

Actually --pattern only supports few cases. You can check the help message with CLI doc or az storage blob download-batch -h:

--pattern
The pattern used for globbing files or blobs in the source. The supported patterns are '*', '?', '[seq]', and '[!seq]'. For more information, please refer to https://docs.python.org/3.7/library/fnmatch.html.

evelyn-ys · 2022-04-07T05:44:38Z

If you are giving a blob name as --pattern, I'll suggest you to use az storage blob download.
It used to work with previous CLI using az storage blob download-batch but we have an bug dealing with full blob name in latest CLI. This will be fixed in next release. But this usage is not documented in our help message or doc, so it's not recommended.

taoisu · 2022-04-07T06:17:24Z

@evelyn-ys thanks for the investigation, but I'm afraid that is not the case for me. We've been using the same command for two years in one of our pipeline, and it was just broken today, so it is clear something is wrong in the az cli.

taoisu · 2022-04-07T06:19:20Z

az storage blob download-batch -d {} -s jobs --auth-mode key --account-name {} --subscription {} --pattern output/invoice/20220402/layoutlm-TT_MH5MSL1024_Tier1_FT2/layoutlm@2022-04-02_22-37-11/model_artifacts/model.onnx

This is the full command we're using.

taoisu · 2022-04-07T06:21:27Z

az storage blob download is not an option for us because we need --pattern specifically, one of the patterns is something like "*.bias"

taoisu · 2022-04-07T06:22:34Z

I'm not sure if you just need to fix the latest CLI, because I've also tried installing previous versions (which works in the past), and I'm seeing the exactly the same issue.

evelyn-ys · 2022-04-07T06:29:44Z

When installing previous versions, please make sure all the dependencies of latest version are uninstalled. I'm afraid there's some storage dependency left.

I'm not sure if you just need to fix the latest CLI, because I've also tried installing previous versions (which works in the past), and I'm seeing the exactly the same issue.

evelyn-ys · 2022-04-07T06:34:33Z

az storage blob download-batch -d {} -s jobs --auth-mode key --account-name {} --subscription {} --pattern output/invoice/20220402/layoutlm-TT_MH5MSL1024_Tier1_FT2/layoutlm@2022-04-02_22-37-11/model_artifacts/model.onnx

This is the full command we're using.

For patterns not following the four cases(*, ?, [seq], [!seq]), we used to take it as a full blob name. And the latest CLI does have a bug here.

evelyn-ys · 2022-04-07T08:09:48Z

@venkymns for specific blob file, you can use az storage blob download as workaround

taoisu · 2022-04-07T23:44:23Z

When installing previous versions, please make sure all the dependencies of latest version are uninstalled. I'm afraid there's some storage dependency left.

I'm not sure if you just need to fix the latest CLI, because I've also tried installing previous versions (which works in the past), and I'm seeing the exactly the same issue.

What is interesting is that even if I install azure-cli=2.29.2-1~bionic freshly inside the docker image this exact same error would occur, any ideas why?

taoisu · 2022-04-07T23:46:11Z

az storage blob download-batch -d {} -s jobs --auth-mode key --account-name {} --subscription {} --pattern output/invoice/20220402/layoutlm-TT_MH5MSL1024_Tier1_FT2/layoutlm@2022-04-02_22-37-11/model_artifacts/model.onnx
This is the full command we're using.
For patterns not following the four cases(*, ?, [seq], [!seq]), we used to take it as a full blob name. And the latest CLI does have a bug here.

I can understand that if the pattern is not working I might end up downloading the wrong files, but the callstack tells me that seems the API has been deleted or some wrong version of dependencies are installed. Are you sure this is related to the bug you are talking about?

evelyn-ys · 2022-04-08T02:13:14Z

I can understand that if the pattern is not working I might end up downloading the wrong files, but the callstack tells me that seems the API has been deleted or some wrong version of dependencies are installed. Are you sure this is related to the bug you are talking about?

We upgraded the dependency Python SDK and the APIs have changed a lot. For patterns not following *, ?, [seq], [!seq], the API is unchanged so making the command fail

…hen `--pattern` is blob name (#22072)

taoisu · 2022-10-11T08:05:33Z

I use something like “model.onnx”, but it doesn’t matter as far as I know, any pattern fails for me. And it just happens today after I rebuilt the docker image which reinstalls the azure cli with latest version. Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Yishi Wang ***@***.***> Sent: Wednesday, April 6, 2022 8:38:47 PM To: Azure/azure-cli ***@***.***> Cc: Xiangyu Guo ***@***.***>; Author ***@***.***> Subject: Re: [Azure/azure-cli] Latest az cli fails to run download-batch command (Issue #21966) Can you share which pattern you are using? — Reply to this email directly, view it on GitHub<#21966 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHE74KRU4LCMKWJAUL4UWELVDZKEPANCNFSM5SXQ7C4Q>. You are receiving this because you authored the thread.Message ID: ***@***.***>

ghost added customer-reported Issues that are reported by GitHub users external to the Azure organization. Storage az storage labels Apr 6, 2022

ghost added this to the Backlog milestone Apr 6, 2022

ghost assigned evelyn-ys Apr 6, 2022

ghost added Auto-Assign Auto assign by bot CXP Attention This issue is handled by CXP team. Machine Learning az ml Batch az batch labels Apr 6, 2022

yonzhan removed Batch az batch Machine Learning az ml CXP Attention This issue is handled by CXP team. labels Apr 7, 2022

evelyn-ys mentioned this issue Apr 15, 2022

[Storage] Fix #21966: az storage blob download-batch: Fix failure when --pattern is blob name #22072

Merged

3 tasks

evelyn-ys closed this as completed in #22072 Apr 15, 2022

evelyn-ys added a commit that referenced this issue Apr 15, 2022

[Storage] Fix #21966: az storage blob download-batch: Fix failure w…

806b7ab

…hen `--pattern` is blob name (#22072)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Latest az cli fails to run download-batch command #21966

Latest az cli fails to run download-batch command #21966

taoisu commented Apr 6, 2022

yonzhan commented Apr 7, 2022

evelyn-ys commented Apr 7, 2022

taoisu commented Apr 7, 2022

evelyn-ys commented Apr 7, 2022

evelyn-ys commented Apr 7, 2022

taoisu commented Apr 7, 2022

taoisu commented Apr 7, 2022

taoisu commented Apr 7, 2022

taoisu commented Apr 7, 2022

evelyn-ys commented Apr 7, 2022

evelyn-ys commented Apr 7, 2022

evelyn-ys commented Apr 7, 2022

taoisu commented Apr 7, 2022

taoisu commented Apr 7, 2022

evelyn-ys commented Apr 8, 2022

taoisu commented Oct 11, 2022 via email

Latest az cli fails to run download-batch command #21966

Latest az cli fails to run download-batch command #21966

Comments

taoisu commented Apr 6, 2022

yonzhan commented Apr 7, 2022

evelyn-ys commented Apr 7, 2022

taoisu commented Apr 7, 2022

evelyn-ys commented Apr 7, 2022

evelyn-ys commented Apr 7, 2022

taoisu commented Apr 7, 2022

taoisu commented Apr 7, 2022

taoisu commented Apr 7, 2022

taoisu commented Apr 7, 2022

evelyn-ys commented Apr 7, 2022

evelyn-ys commented Apr 7, 2022

evelyn-ys commented Apr 7, 2022

taoisu commented Apr 7, 2022

taoisu commented Apr 7, 2022

evelyn-ys commented Apr 8, 2022

taoisu commented Oct 11, 2022 via email