Network security group with flow logs setup cause "Parameter 'FlowLog.target_resource_id' can not be None" error

[ecchong]

SUMMARY

When network security group is setup with flow logs, ansible failed with error

Parameter 'FlowLog.target_resource_id' can not be None

It is also reported in #583

ISSUE TYPE

• Bug Report

COMPONENT NAME

ANSIBLE VERSION

ansible 2.9.23
  config file = /home/echong/Azure_Infra/ansible.cfg
  configured module search path = ['/home/echong/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/echong/venv-azure/lib64/python3.6/site-packages/ansible
  executable location = /home/echong/venv-azure/bin/ansible
  python version = 3.6.8 (default, Mar 18 2021, 08:58:41) [GCC 8.4.1 20200928 (Red Hat 8.4.1-1)]

COLLECTION VERSION

1.9.0

CONFIGURATION

None

OS / ENVIRONMENT

RHEL 8.4

STEPS TO REPRODUCE

---
- hosts: localhost
  connection: local
  gather_facts: no
  collections:
  - azure.azcollection
  tasks:
  - name: Get all network security groups
    azure_rm_securitygroup_info:
      resource_group: ansible-group

EXPECTED RESULTS

Listing network security groups. It works when there is no flow logs created for the NSG.

ACTUAL RESULTS

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: msrest.exceptions.ValidationError: Parameter 'FlowLog.target_resource_id' can not be None.
fatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):
  File \"/home/echong/.ansible/tmp/ansible-tmp-1631333616.259789-308142-134151056145727/AnsiballZ_azure_rm_securitygroup_info.py\", line 102, in <module>
    _ansiballz_main()
  File \"/home/echong/.ansible/tmp/ansible-tmp-1631333616.259789-308142-134151056145727/AnsiballZ_azure_rm_securitygroup_info.py\", line 94, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File \"/home/echong/.ansible/tmp/ansible-tmp-1631333616.259789-308142-134151056145727/AnsiballZ_azure_rm_securitygroup_info.py\", line 40, in invoke_module
    runpy.run_module(mod_name='ansible_collections.azure.azcollection.plugins.modules.azure_rm_securitygroup_info', init_globals=None, run_name='__main__', alter_sys=True)
  File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code
    exec(code, run_globals)
  File \"/tmp/ansible_azure_rm_securitygroup_info_payload_rhpm8tz_/ansible_azure_rm_securitygroup_info_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_securitygroup_info.py\", line 316, in <module>
  File \"/tmp/ansible_azure_rm_securitygroup_info_payload_rhpm8tz_/ansible_azure_rm_securitygroup_info_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_securitygroup_info.py\", line 312, in main
  File \"/tmp/ansible_azure_rm_securitygroup_info_payload_rhpm8tz_/ansible_azure_rm_securitygroup_info_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_securitygroup_info.py\", line 254, in __init__
  File \"/tmp/ansible_azure_rm_securitygroup_info_payload_rhpm8tz_/ansible_azure_rm_securitygroup_info_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py\", line 469, in __init__
  File \"/tmp/ansible_azure_rm_securitygroup_info_payload_rhpm8tz_/ansible_azure_rm_securitygroup_info_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_securitygroup_info.py\", line 268, in exec_module
  File \"/tmp/ansible_azure_rm_securitygroup_info_payload_rhpm8tz_/ansible_azure_rm_securitygroup_info_payload.zip/ansible_collections/azure/azcollection/plugins/modules/azure_rm_securitygroup_info.py\", line 305, in list_items
  File \"/tmp/ansible_azure_rm_securitygroup_info_payload_rhpm8tz_/ansible_azure_rm_securitygroup_info_payload.zip/ansible_collections/azure/azcollection/plugins/module_utils/azure_rm_common.py\", line 630, in serialize_obj
  File \"/home/echong/venv-azure/lib64/python3.6/site-packages/msrest/serialization.py\", line 628, in body
    raise errors[0]
  File \"/home/echong/venv-azure/lib64/python3.6/site-packages/msrest/serialization.py\", line 250, in validate
    Serializer.validate(value, debug_name, **self._validation.get(attr_name, {}))
  File \"/home/echong/venv-azure/lib64/python3.6/site-packages/msrest/serialization.py\", line 730, in validate
    raise ValidationError(\"required\", name, True)
msrest.exceptions.ValidationError: Parameter 'FlowLog.target_resource_id' can not be None.
", "module_stdout": "", "msg": "MODULE FAILURE
See stdout/stderr for the exact error", "rc": 1}

[iscai-msft]

hey @ecchong, it seems that probably whoever is creating the ansible SDK is not passing target_resource_id to create a FlowLog, @Fred-sun are you the person to assign the issue to?

[ecchong]

@iscai-msft Not sure, but I think the Ansible project will be the place to raise the issue. However, I think the problem is between the azcollection module and azure core itself.

[iscai-msft]

I still think the issue is that the ansible SDK is not passing all required parameters to the generated python SDK, not an issue in azure-core / ansible. The generated python SDK requires that the target_resource_id is passed when creating a FlowLog object, and it doesn't seem this is happening. cc @Fred-sun

[Fred-sun]

@iscai-msft I'm sorry to reply you so late. I was busy upgrading SDK recently and ignored these questions. Please forgive me. Thank you very much!

[Fred-sun]

@iscai-msft @ecchong Thank you for reporting this problem, yes, the network security group should have a parameter associated with NSG flow logs, this parameter is not supported for the moment, we will add it later. I just tested this，no matter managed flow logs attach or not, it did not get an error when retrieving resource group information . Thank you very much!

[l3ender]

@Fred-sun Could you provide a little more detail on your findings? We are facing this issue as well and I am willing to work on a PR, but I'm struggling to understand why simply retrieving NSGs fails for this scenario.

Thank you!

[l3ender]

I've opened #726 to resolve the issue. I am able to verify the change in our environment: an NSG with flow logs can be retrieved without the error.

The change aligns the output of azure_rm_securitygroup_info to match that of azure_rm_securitygroup. The issue was occurring after the NSG had been loaded from the server and Ansible is trying to convert the output object. The update will be a breaking change for the info module as the output properties are changing.

@Fred-sun Please review...thank you!