Dataprotection ModuleNotFoundError 'azure.mgmt.resourcegraph' #4562

HansZimmer5000 · 2022-03-23T09:47:51Z

I understand that of course I could just install this missing module via pip3 as a hotfix. But, I also think this is not how this extension should behave in the first place.
Is there maybe something missing during installation of the extension? Didn't experience any ModuleNotFoundErrors before.
Find below the 'az feedback --verbose' output with the list of my installed extensions, I manually noted which of them is a preview version.

Extension name (the extension in question)

dataprotection

Description of issue (in as much detail as possible)

ModuleNotFoundError: No module named 'azure.mgmt.resourcegraph'
command: az dataprotection backup-instance --help

This is autogenerated. Please review and update as needed.

Describe the bug

az feedback --verbose

Command Name
az dataprotection backup-instance

Errors:

The command failed with an unexpected error. Here is the traceback:
No module named 'azure.mgmt.resourcegraph'
Traceback (most recent call last):
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 583, in execute
    parsed_args = self.parser.parse_args(args)
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/knack/parser.py", line 261, in parse_args
    return super().parse_args(args)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1821, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/azure/cli/core/parser.py", line 284, in parse_known_args
    self._namespace, self._raw_arguments = super().parse_known_args(args=args, namespace=namespace)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1854, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 2045, in _parse_known_args
    positionals_end_index = consume_positionals(start_index)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 2022, in consume_positionals
    take_action(action, args)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1931, in take_action
    action(self, namespace, argument_values, option_string)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1210, in __call__
    subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/azure/cli/core/parser.py", line 284, in parse_known_args
    self._namespace, self._raw_arguments = super().parse_known_args(args=args, namespace=namespace)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1854, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 2045, in _parse_known_args
    positionals_end_index = consume_positionals(start_index)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 2022, in consume_positionals
    take_action(action, args)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1931, in take_action
    action(self, namespace, argument_values, option_string)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1210, in __call__
    subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/azure/cli/core/parser.py", line 284, in parse_known_args
    self._namespace, self._raw_arguments = super().parse_known_args(args=args, namespace=namespace)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1854, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 2063, in _parse_known_args
    start_index = consume_optional(start_index)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 2003, in consume_optional
    take_action(action, args, option_string)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1931, in take_action
    action(self, namespace, argument_values, option_string)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 1095, in __call__
    parser.print_help()
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/argparse.py", line 2551, in print_help
    self._print_message(self.format_help(), file)
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/azure/cli/core/parser.py", line 195, in format_help
    super(AzCliCommandParser, self).format_help()
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/knack/parser.py", line 248, in format_help
    self.cli_help.show_help(self.prog.split()[0],
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/azure/cli/core/_help.py", line 165, in show_help
    else self.group_help_cls(self, delimiters, parser)
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/knack/help.py", line 253, in __init__
    child.load(options)
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/azure/cli/core/_help.py", line 303, in load
    loader.versioned_load(self, options)
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/azure/cli/core/_help_loaders.py", line 154, in versioned_load
    super(CliHelpFile, help_obj).load(parser)  # pylint:disable=bad-super-call
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/knack/help.py", line 198, in load
    description = getattr(options, 'description', None)
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/knack/parser.py", line 241, in __getattribute__
    self.description = self._description() \
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 133, in description_loader
    op = self.get_op_handler(self.op_path)
  File "/usr/local/Cellar/azure-cli/2.32.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 59, in get_op_handler
    handler = import_module(mod_to_import)
  File "/usr/local/Cellar/python@3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/NJ62KD/.azure/cliextensions/dataprotection/azext_dataprotection/custom.py", line 20, in <module>
    raise e
  File "/Users/NJ62KD/.azure/cliextensions/dataprotection/azext_dataprotection/custom.py", line 15, in <module>
    from .manual.custom import *  # noqa: F403
  File "/Users/NJ62KD/.azure/cliextensions/dataprotection/azext_dataprotection/manual/custom.py", line 17, in <module>
    from azure.mgmt.resourcegraph.models import \
ModuleNotFoundError: No module named 'azure.mgmt.resourcegraph'

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

install extension via first use (implicit) or via az extension add -n dataprotection (explicit). Both ways led to the same outcome.
az dataprotection backup-instance -h or any az dataprotection backup-instance subcommand.

Expected Behavior

Shows help output.

Environment Summary

macOS-12.2.1-x86_64-i386-64bit, Darwin 21.3.0
Python 3.10.1
Installer: HOMEBREW

azure-cli 2.32.0 *

Extensions:
front-door 1.0.16
alertsmanagement 0.1.1
logic 0.1.5 (manual note: preview)
resource-graph 2.1.0
sentinel 0.1.1
ssh 1.0.0 (manual note: preview)
dataprotection 0.2.0
application-insights 0.1.14 (manual note: preview)

Dependencies:
msal 1.16.0
azure-mgmt-resource 20.0.0

Additional Context

The text was updated successfully, but these errors were encountered:

HansZimmer5000 · 2022-03-23T10:03:56Z

To my surprise hotfix via pip and pip3 install azure-mgmt-resourcegraph did not fix the issue above with az.

With python3 I canimport azure.mgmt.resourcegraph without any problems.

yonzhan · 2022-03-23T10:11:05Z

route to CXP team

navba-MSFT · 2022-03-24T05:42:47Z

@HansZimmer5000 Thanks for reaching out to us and proving the current update. Please let us know if you are still facing any issues while running the az dataprotection backup-instance now after importing the above module using Python3. Awaiting your reply.

navba-MSFT · 2022-03-29T06:32:27Z

@HansZimmer5000 I wanted to do quick follow-up to check if you had a chance to look at my above comment. Please let me know if you are still facing the same issue. Awaiting your reply.

HansZimmer5000 · 2022-03-29T12:22:30Z

Good day @navba-MSFT !
Thanks for the response, please excuse my late answer.
What do you mean by:

after importing the above module using Python3

If you mean the pip3 install I mentioned above, that does not change the issue.

jiasli · 2022-03-31T04:52:15Z

Let me clarify 2 things:

python3's import won't work because Azure CLI runs in an isolated virtual env, meaning it doesn't share system's python packages.
Each python3's execution is a separate process, so even you can do import with python3, this import statement only affects the current python3 process, so it doesn't affect other python3 executions like az dataprotection backup-instance.

I am also able to reproduce this issue on Windows:

> az extension add -n dataprotection
> az dataprotection backup-instance -h
...
  File "C:\Users\name\.azure\cliextensions\dataprotection\azext_dataprotection\manual\custom.py", line 17, in <module>
    from azure.mgmt.resourcegraph.models import \
ModuleNotFoundError: No module named 'azure.mgmt.resourcegraph'

jiasli · 2022-03-31T05:28:58Z

Cause

Even though Azure/azure-cli#13163 added the support to load azure.mgmt from extension installation folder:

https://github.com/Azure/azure-cli/blob/e2f55ec7de8ff0b19c05e082d408992ac1a1139c/src/azure-cli-core/azure/cli/core/extension/operations.py#L501-L521

def add_extension_to_path(extension_name, ext_dir=None):
    ext_dir = ext_dir or get_extension(extension_name).path
    sys.path.append(ext_dir)
    # If this path update should have made a new "azure" module available,
    # extend the existing module with its path. This allows extensions to
    # include (or depend on) Azure SDK modules that are not yet part of
    # the CLI. This applies to both the "azure" and "azure.mgmt" namespaces,
    # but ensures that modules installed by the CLI take priority.
    azure_dir = os.path.join(ext_dir, "azure")
    if os.path.isdir(azure_dir):
        import azure
        azure.__path__.append(azure_dir)
        azure_mgmt_dir = os.path.join(azure_dir, "mgmt")
        if os.path.isdir(azure_mgmt_dir):
            try:
                # Should have been imported already, so this will be quick
                import azure.mgmt
            except ImportError:
                pass
            else:
                azure.mgmt.__path__.append(azure_mgmt_dir)

it is not used in core's extension loading process (but in other modules like network):

https://github.com/Azure/azure-cli/blob/af0c73a37ca3f540cb0b3f0b18e0f362932ab191/src/azure-cli-core/azure/cli/core/__init__.py#L351

                    ext_dir = ext.path or get_extension_path(ext_name)
                    sys.path.append(ext_dir)

We can show what azure really is at this time:

import azure
print(azure.__path__)

['d:\\cli\\azure-cli\\src\\azure-cli-telemetry\\azure', 'D:\\cli\\py310\\lib\\site-packages\\azure', 'd:\\cli\\azure-cli\\src\\azure-cli-core\\azure', 'd:\\cli\\azure-cli\\src\\azure-cli\\azure', 'd:\\cli\\azure-cli\\src\\azure-cli-testsdk\\azure']

Because azure has already been loaded, adding ext_dir won't affect existing azure module.

Solution

The common practice is to put SDKs in a vendored_sdks folder, instead of installing from PyPI.

To my surprise, this extension was added by #3459 (Jun 3, 2021) but this bug is never discovered.

jiasli · 2022-03-31T09:37:54Z

This problem also happens in a virtual env, but not Azure CLI installed with MSI or DEB:

I tested with a sample script:

import sys

import azure
print(azure, azure.__path__)

sys.path.append(r'C:\Users\username\.azure\cliextensions\dataprotection')

import azure
print(azure, azure.__path__)

import azure.mgmt
print(azure.mgmt, azure.mgmt.__path__)

import azure.mgmt.resourcegraph
print(azure.mgmt.resourcegraph, azure.mgmt.resourcegraph.__path__)

System python:

> & "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe" D:\cli\testproj\main.py
<module 'azure' (namespace)> _NamespacePath(['C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2\\lib\\site-packages\\azure'])
<module 'azure' (namespace)> _NamespacePath(['C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2\\lib\\site-packages\\azure'])
<module 'azure.mgmt' (namespace)> _NamespacePath(['C:\\Program Files (x86)\\Microsoft SDKs\\Azure\\CLI2\\lib\\site-packages\\azure\\mgmt', 'C:\\Users\\username\\.azure\\cliextensions\\dataprotection\\azure\\mgmt'])
<module 'azure.mgmt.resourcegraph' from 'C:\\Users\\username\\.azure\\cliextensions\\dataprotection\\azure\\mgmt\\resourcegraph\\__init__.py'> ['C:\\Users\\username\\.azure\\cliextensions\\dataprotection\\azure\\mgmt\\resourcegraph']

Virtual env python:

> & "D:\cli\py310\Scripts\python.exe" D:\cli\testproj\main.py
<module 'azure' from 'd:\\cli\\azure-cli\\src\\azure-cli-telemetry\\azure\\__init__.py'> ['d:\\cli\\azure-cli\\src\\azure-cli-telemetry\\azure', 'D:\\cli\\py310\\lib\\site-packages\\azure', 'd:\\cli\\azure-cli\\src\\azure-cli-core\\azure', 'd:\\cli\\azure-cli\\src\\azure-cli\\azure', 'd:\\cli\\azure-cli\\src\\azure-cli-testsdk\\azure']
<module 'azure' from 'd:\\cli\\azure-cli\\src\\azure-cli-telemetry\\azure\\__init__.py'> ['d:\\cli\\azure-cli\\src\\azure-cli-telemetry\\azure', 'D:\\cli\\py310\\lib\\site-packages\\azure', 'd:\\cli\\azure-cli\\src\\azure-cli-core\\azure', 'd:\\cli\\azure-cli\\src\\azure-cli\\azure', 'd:\\cli\\azure-cli\\src\\azure-cli-testsdk\\azure']
<module 'azure.mgmt' (<_frozen_importlib_external._NamespaceLoader object at 0x0000025B0E2DA7A0>)> _NamespacePath(['D:\\cli\\py310\\lib\\site-packages\\azure\\mgmt'])
Traceback (most recent call last):
  File "D:\cli\testproj\main.py", line 14, in <module>
    import azure.mgmt.resourcegraph
ModuleNotFoundError: No module named 'azure.mgmt.resourcegraph'

In a virtual env which installs Azure CLI from source code, under azure and azure/mgmt folders, there are __init__.py files containing one line:

__path__ = __import__("pkgutil").extend_path(__path__, __name__)

These files make azure and azure.mgmt normal (non-namespace) packages, preventing Python from looking up additional library folders.

navba-MSFT · 2022-04-06T06:57:40Z

@HansZimmer5000 Apologies for the late reply. Could you please refer this comment and follow the workaround suggested there? Please let me know if you face any issues. Awaiting your reply.

navba-MSFT · 2022-04-11T04:46:45Z

@HansZimmer5000 Could you please let me know if you have any follow-up question on this ?

navba-MSFT · 2022-04-14T06:08:46Z

@HansZimmer5000 Please let me know if you had a chance to look at the above comments. Awaiting your reply.

bebound · 2022-04-14T09:11:59Z

@sambitratha Could you please vendor azure.mgmt.resourcegraph in dataprotection extension?

navba-MSFT · 2022-04-18T03:28:26Z

@HansZimmer5000 We will now proceed with closure of this github issue. Once you have an update and need any further assistance on this issue, please feel free to reopen this thread. We would be happy to help.

HansZimmer5000 · 2022-04-25T07:17:19Z

Solution

The common practice is to put SDKs in a vendored_sdks folder, instead of installing from PyPI.

Good day @jiasli !

Please excuse my late response. Thank you very much for the detailed debugging.
What are the steps to realize your solution? Tried to figure it out myself, sadly I don't have more time for tinkering.
Tried to remove azure('brew remove az' + rm -rf ~/.azure) and install it again (brew install az + 'az extension add --name dataprotection') with now change.

fyi: In python3 your import azure.mgmt.resourcegraph and print(azure.mgmt.resourcegraph, azure.mgmt.resourcegraph.__path__) showed no error for me.

bebound · 2023-10-26T05:55:25Z

The __init__.py in source code make it's possible to load the azure folder in extension folder if azure-cli is installed in develop mode(azdev setup).
Although we encourage extension to use vendered_sdks instead of azure-xxx in dependency, I still prefer to use the modern find_namespace_package way to create namespace package and remove __init__.py.

yonzhan added Data Protection and removed question The issue doesn't require a change to the product in order to be resolved. Most issues start as that VM SSH Alerts Management labels Mar 23, 2022

navba-MSFT self-assigned this Mar 24, 2022

navba-MSFT added the needs-author-feedback More information is needed from author to address the issue. label Mar 24, 2022

ghost added needs-team-attention This issue needs attention from Azure service team or SDK team and removed needs-author-feedback More information is needed from author to address the issue. labels Mar 29, 2022

This was referenced Mar 30, 2022

{CI} Update issue and pr template Azure/azure-cli#21860

Merged

{CI} Update issue and pr template #4599

Merged

navba-MSFT added needs-author-feedback More information is needed from author to address the issue. and removed needs-team-attention This issue needs attention from Azure service team or SDK team labels Apr 6, 2022

navba-MSFT closed this as completed Apr 14, 2022

navba-MSFT reopened this Apr 14, 2022

navba-MSFT closed this as completed Apr 18, 2022

navba-MSFT added the question The issue doesn't require a change to the product in order to be resolved. Most issues start as that label Jun 27, 2022

bebound mentioned this issue Nov 1, 2023

Removed unused files in repo Azure/azure-cli#27732

Open

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dataprotection ModuleNotFoundError 'azure.mgmt.resourcegraph' #4562

Dataprotection ModuleNotFoundError 'azure.mgmt.resourcegraph' #4562

HansZimmer5000 commented Mar 23, 2022 •

edited

Loading

HansZimmer5000 commented Mar 23, 2022 •

edited

Loading

yonzhan commented Mar 23, 2022

navba-MSFT commented Mar 24, 2022

navba-MSFT commented Mar 29, 2022

HansZimmer5000 commented Mar 29, 2022 •

edited

Loading

jiasli commented Mar 31, 2022

jiasli commented Mar 31, 2022

jiasli commented Mar 31, 2022

navba-MSFT commented Apr 6, 2022

navba-MSFT commented Apr 11, 2022

navba-MSFT commented Apr 14, 2022

bebound commented Apr 14, 2022

navba-MSFT commented Apr 18, 2022

HansZimmer5000 commented Apr 25, 2022 •

edited

Loading

Solution

bebound commented Oct 26, 2023 •

edited

Loading

Dataprotection ModuleNotFoundError 'azure.mgmt.resourcegraph' #4562

Dataprotection ModuleNotFoundError 'azure.mgmt.resourcegraph' #4562

Comments

HansZimmer5000 commented Mar 23, 2022 • edited Loading

Extension name (the extension in question)

Description of issue (in as much detail as possible)

This is autogenerated. Please review and update as needed.

Describe the bug

az feedback --verbose

To Reproduce:

Expected Behavior

Environment Summary

Additional Context

HansZimmer5000 commented Mar 23, 2022 • edited Loading

yonzhan commented Mar 23, 2022

navba-MSFT commented Mar 24, 2022

navba-MSFT commented Mar 29, 2022

HansZimmer5000 commented Mar 29, 2022 • edited Loading

jiasli commented Mar 31, 2022

jiasli commented Mar 31, 2022

Cause

Solution

jiasli commented Mar 31, 2022

navba-MSFT commented Apr 6, 2022

navba-MSFT commented Apr 11, 2022

navba-MSFT commented Apr 14, 2022

bebound commented Apr 14, 2022

navba-MSFT commented Apr 18, 2022

HansZimmer5000 commented Apr 25, 2022 • edited Loading

Solution

bebound commented Oct 26, 2023 • edited Loading

HansZimmer5000 commented Mar 23, 2022 •

edited

Loading

HansZimmer5000 commented Mar 23, 2022 •

edited

Loading

HansZimmer5000 commented Mar 29, 2022 •

edited

Loading

HansZimmer5000 commented Apr 25, 2022 •

edited

Loading

bebound commented Oct 26, 2023 •

edited

Loading