MacOS - SSL verification fails with: unable to get local issuer certificate #154

kbreit · 2020-10-20T19:08:31Z

SUMMARY

I am having problems where the vmware_rest modules won't run on my system. It works on someone else's system and I'd have to test it on another machine of mine.

ISSUE TYPE

Bug Report

COMPONENT NAME

vcenter_vm_info

ANSIBLE VERSION

(hubs-infrastructure) anthos-node-host [master●] % ansible --version
ansible 2.9.13
  config file = None
  configured module search path = ['/Users/kbreit/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/kbreit/.pyenv/versions/3.7.7/envs/hubs-infrastructure/lib/python3.7/site-packages/ansible
  executable location = /Users/kbreit/.pyenv/versions/hubs-infrastructure/bin/ansible
  python version = 3.7.7 (default, Apr 17 2020, 09:55:29) [Clang 11.0.0 (clang-1100.0.33.16)]

CONFIGURATION

DEFAULT_KEEP_REMOTE_FILES(env: ANSIBLE_KEEP_REMOTE_FILES) = True
DEFAULT_VAULT_PASSWORD_FILE(env: ANSIBLE_VAULT_PASSWORD_FILE) = /Users/kbreit/pass.txt

OS / ENVIRONMENT

macOS 10.15, latest

STEPS TO REPRODUCE

    - name: Get all VMs
      vmware.vmware_rest.vcenter_vm_info:
        vcenter_hostname: hostnamehere
        vcenter_username: admin@vsphere.local
        vcenter_password: mypass
        vcenter_validate_certs: no
      register: existing_vms
      until: existing_vms is not failed
      delegate_to: localhost

EXPECTED RESULTS

It should work and return output

ACTUAL RESULTS

Using module file /Users/kbreit/.ansible/collections/ansible_collections/vmware/vmware_rest/plugins/modules/vcenter_vm_info.py
<localhost> PUT /Users/kbreit/.ansible/tmp/ansible-local-10358qp094r_6/tmpjnz0p7he TO /Users/kbreit/.ansible/tmp/ansible-tmp-1603220877.142858-10389-225793521646905/AnsiballZ_vcenter_vm_info.py
<localhost> EXEC /bin/sh -c 'chmod u+x /Users/kbreit/.ansible/tmp/ansible-tmp-1603220877.142858-10389-225793521646905/ /Users/kbreit/.ansible/tmp/ansible-tmp-1603220877.142858-10389-225793521646905/AnsiballZ_vcenter_vm_info.py && sleep 0'
<localhost> EXEC /bin/sh -c '/Users/kbreit/.pyenv/versions/3.7.7/envs/hubs-infrastructure/bin/python3.7 /Users/kbreit/.ansible/tmp/ansible-tmp-1603220877.142858-10389-225793521646905/AnsiballZ_vcenter_vm_info.py && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/Users/kbreit/.ansible/tmp/ansible-tmp-1603220877.142858-10389-225793521646905/AnsiballZ_vcenter_vm_info.py", line 247, in <module>
    _ansiballz_main()
  File "/Users/kbreit/.ansible/tmp/ansible-tmp-1603220877.142858-10389-225793521646905/AnsiballZ_vcenter_vm_info.py", line 237, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/Users/kbreit/.ansible/tmp/ansible-tmp-1603220877.142858-10389-225793521646905/AnsiballZ_vcenter_vm_info.py", line 108, in invoke_module
    runpy.run_module(mod_name='ansible_collections.vmware.vmware_rest.plugins.modules.vcenter_vm_info', init_globals=None, run_name='__main__', alter_sys=True)
  File "/Users/kbreit/.pyenv/versions/3.7.7/lib/python3.7/runpy.py", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/Users/kbreit/.pyenv/versions/3.7.7/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/Users/kbreit/.pyenv/versions/3.7.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/var/folders/t5/gx_568vj4w958qgxx_hhklvc0000gn/T/ansible_vmware.vmware_rest.vcenter_vm_info_payload_c9k54e1w/ansible_vmware.vmware_rest.vcenter_vm_info_payload.zip/ansible_collections/vmware/vmware_rest/plugins/modules/vcenter_vm_info.py", line 425, in <module>
  File "/Users/kbreit/.pyenv/versions/3.7.7/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/var/folders/t5/gx_568vj4w958qgxx_hhklvc0000gn/T/ansible_vmware.vmware_rest.vcenter_vm_info_payload_c9k54e1w/ansible_vmware.vmware_rest.vcenter_vm_info_payload.zip/ansible_collections/vmware/vmware_rest/plugins/modules/vcenter_vm_info.py", line 366, in main
  File "/var/folders/t5/gx_568vj4w958qgxx_hhklvc0000gn/T/ansible_vmware.vmware_rest.vcenter_vm_info_payload_c9k54e1w/ansible_vmware.vmware_rest.vcenter_vm_info_payload.zip/ansible_collections/cloud/common/plugins/module_utils/turbo/module.py", line 28, in __init__
  File "/var/folders/t5/gx_568vj4w958qgxx_hhklvc0000gn/T/ansible_vmware.vmware_rest.vcenter_vm_info_payload_c9k54e1w/ansible_vmware.vmware_rest.vcenter_vm_info_payload.zip/ansible_collections/cloud/common/plugins/module_utils/turbo/module.py", line 70, in run_on_daemon
  File "/var/folders/t5/gx_568vj4w958qgxx_hhklvc0000gn/T/ansible_vmware.vmware_rest.vcenter_vm_info_payload_c9k54e1w/ansible_vmware.vmware_rest.vcenter_vm_info_payload.zip/ansible_collections/cloud/common/plugins/module_utils/turbo/module.py", line 61, in connect
FileNotFoundError: [Errno 2] No such file or directory
fatal: [vmware]: FAILED! => {
    "attempts": 3,
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/Users/kbreit/.ansible/tmp/ansible-tmp-1603220877.142858-10389-225793521646905/AnsiballZ_vcenter_vm_info.py\", line 247, in <module>\n    _ansiballz_main()\n  File \"/Users/kbreit/.ansible/tmp/ansible-tmp-1603220877.142858-10389-225793521646905/AnsiballZ_vcenter_vm_info.py\", line 237, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/Users/kbreit/.ansible/tmp/ansible-tmp-1603220877.142858-10389-225793521646905/AnsiballZ_vcenter_vm_info.py\", line 108, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.vmware.vmware_rest.plugins.modules.vcenter_vm_info', init_globals=None, run_name='__main__', alter_sys=True)\n  File \"/Users/kbreit/.pyenv/versions/3.7.7/lib/python3.7/runpy.py\", line 205, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/Users/kbreit/.pyenv/versions/3.7.7/lib/python3.7/runpy.py\", line 96, in _run_module_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/Users/kbreit/.pyenv/versions/3.7.7/lib/python3.7/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/var/folders/t5/gx_568vj4w958qgxx_hhklvc0000gn/T/ansible_vmware.vmware_rest.vcenter_vm_info_payload_c9k54e1w/ansible_vmware.vmware_rest.vcenter_vm_info_payload.zip/ansible_collections/vmware/vmware_rest/plugins/modules/vcenter_vm_info.py\", line 425, in <module>\n  File \"/Users/kbreit/.pyenv/versions/3.7.7/lib/python3.7/asyncio/base_events.py\", line 587, in run_until_complete\n    return future.result()\n  File \"/var/folders/t5/gx_568vj4w958qgxx_hhklvc0000gn/T/ansible_vmware.vmware_rest.vcenter_vm_info_payload_c9k54e1w/ansible_vmware.vmware_rest.vcenter_vm_info_payload.zip/ansible_collections/vmware/vmware_rest/plugins/modules/vcenter_vm_info.py\", line 366, in main\n  File \"/var/folders/t5/gx_568vj4w958qgxx_hhklvc0000gn/T/ansible_vmware.vmware_rest.vcenter_vm_info_payload_c9k54e1w/ansible_vmware.vmware_rest.vcenter_vm_info_payload.zip/ansible_collections/cloud/common/plugins/module_utils/turbo/module.py\", line 28, in __init__\n  File \"/var/folders/t5/gx_568vj4w958qgxx_hhklvc0000gn/T/ansible_vmware.vmware_rest.vcenter_vm_info_payload_c9k54e1w/ansible_vmware.vmware_rest.vcenter_vm_info_payload.zip/ansible_collections/cloud/common/plugins/module_utils/turbo/module.py\", line 70, in run_on_daemon\n  File \"/var/folders/t5/gx_568vj4w958qgxx_hhklvc0000gn/T/ansible_vmware.vmware_rest.vcenter_vm_info_payload_c9k54e1w/ansible_vmware.vmware_rest.vcenter_vm_info_payload.zip/ansible_collections/cloud/common/plugins/module_utils/turbo/module.py\", line 61, in connect\nFileNotFoundError: [Errno 2] No such file or directory\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

The text was updated successfully, but these errors were encountered:

Akasurde · 2020-10-21T04:10:27Z

@kbreit Thanks for reporting this issue. This is likely caused by asyncio.fork() in macOS.
Could you please use ansible-collections/cloud.common#16 and let us know if it still the issue?

Thanks.

needs_info

kbreit · 2020-10-21T13:43:09Z

@Akasurde That helped quite a bit. I'm now receiving another error. Do you happen to know how to resolve this properly?

fatal: [vmware]: FAILED! => {"attempts": 3, "changed": false, "msg": "Authentication failure: Cannot connect to host e7vmw0vic05.datalinklabs.local:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)')]"}

goneri · 2020-10-21T16:48:32Z

This may not be SSL related, are you sure the node can reach e7vmw0vic05.datalinklabs.local?

goneri · 2020-10-21T16:57:11Z

Actually the unable to get local issuer certificate message makes me think it's not a connectivity issue.

I used Homebrew to install Python and it works fine. If you use the one from xcode, it may be more something like: https://qiita.com/0_a_e/items/d0c7287ea71b555f9ffd

cd "/Applications/Python 3.8/"
"./Install Certificates.command"

kbreit · 2020-10-21T16:59:41Z

I use pyenv. Do you know where that is?

goneri · 2020-10-21T17:03:13Z

Yes, but unless you used a package manager like Brew to pull Python, you probably use the Python from /Applications/Python 3.8/ (or some other similar path) underneath.

kbreit · 2020-10-21T17:10:46Z

/Applications % ls *Python*
zsh: no matches found: *Python*

kbreit · 2020-10-21T17:11:55Z

My pyenv virtual environment is stored in ~/.pyenv/versions/hubs-infrastructure/bin but I don't see Certificates.command.

goneri · 2020-10-21T17:21:20Z

My pyenv virtual environment is stored in ~/.pyenv/versions/hubs-infrastructure/bin but I don't see Certificates.command.

Sorry, this is way above my knowledge of MacOS.

goneri · 2020-10-22T14:28:30Z

Hi @kbreit,

Could you try the following patch?

diff --git a/module_utils/vmware_rest.py b/module_utils/vmware_rest.py
index 515c138..b1bc03d 100644
--- a/module_utils/vmware_rest.py
+++ b/module_utils/vmware_rest.py
@@ -51,7 +51,11 @@ async def open_session(
     if validate_certs:
         connector = aiohttp.TCPConnector(limit=20)
     else:
-        connector = aiohttp.TCPConnector(limit=20, ssl=False)
+        ssl = importlib.import_module("ssl")
+        ssl_context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
+        ssl_context.verify_mode = ssl.CERT_NONE
+        ssl_context.check_hostname = False
+        connector = aiohttp.TCPConnector(limit=20, ssl=ssl_context)
     async with aiohttp.ClientSession(
         connector=connector, connector_owner=False, trace_configs=trace_configs
     ) as session:

kbreit · 2020-10-22T14:41:32Z

@goneri I'm still receiving the error. If you'd like, we can organize some time to do some screen sharing or IRC talk to troubleshoot.

goneri · 2020-10-22T16:09:40Z

@kbreit Sure, good idea. You can ping me on #ansible-vmware anytime.

goneri · 2020-10-22T18:14:37Z

@kbreit can you try this change instead:

diff --git a/plugins/module_utils/vmware_rest.py b/plugins/module_utils/vmware_rest.py
index 515c138..ba88f47 100644
--- a/plugins/module_utils/vmware_rest.py
+++ b/plugins/module_utils/vmware_rest.py
@@ -48,10 +48,17 @@ async def open_session(
         trace_configs = []
 
     auth = aiohttp.BasicAuth(vcenter_username, vcenter_password)
-    if validate_certs:
-        connector = aiohttp.TCPConnector(limit=20)
-    else:
-        connector = aiohttp.TCPConnector(limit=20, ssl=False)
+    ssl = importlib.import_module("ssl")
+    ssl_context = ssl.create_default_context()
+    try:
+        certifi = importlib.import_module("certifi")
+        ssl_context.load_verify_locations(certifi.where())
+    except ModuleNotFoundError:
+        pass
+    if validate_certs is False:
+        ssl_context.check_hostname = False
+        ssl_context.verify_mode = ssl.CERT_NONE
+    connector = aiohttp.TCPConnector(limit=20, ssl=ssl_context)
     async with aiohttp.ClientSession(
         connector=connector, connector_owner=False, trace_configs=trace_configs
     ) as session:

kbreit · 2020-10-22T18:30:18Z

Unfortunately the same error.

A parameter with a `False` value can be a legit parameter that we want to pass to the module. See: ansible-collections/vmware.vmware_rest#154

goneri · 2020-10-22T20:30:16Z

I managed to reproduce the problem. Can you try with this extra fix https://github.com/ansible-collections/cloud.common/pull/20/files

The file to modify is here on your system: ~/.ansible/collections/ansible_collections/cloud/common/plugins/module_utils/turbo/module.py

Akasurde · 2020-10-23T08:09:35Z

@goneri @kbreit I tested ansible-collections/cloud.common#20 on macOS and works for me.

goneri · 2020-10-23T14:28:47Z

Ok, great. This was the main problem. vcenter_validate_certs was silently ignored.

A parameter with a `False` value can be a legit parameter that we want to pass to the module. See: ansible-collections/vmware.vmware_rest#154

goneri · 2020-12-02T14:45:17Z

I close the issue since the problem has been addressed.

Akasurde added bug This issue/PR relates to a bug has_pr labels Oct 21, 2020

goneri changed the title ~~REST modules will not run~~ MacOS - SSL verification fails with: unable to get local issuer certificate Oct 22, 2020

goneri added a commit to goneri/cloud.common that referenced this issue Oct 22, 2020

don't ignore parameter if value == False

34fb902

A parameter with a `False` value can be a legit parameter that we want to pass to the module. See: ansible-collections/vmware.vmware_rest#154

goneri mentioned this issue Oct 22, 2020

don't ignore parameter if value == False ansible-collections/cloud.common#20

Merged

goneri added a commit to goneri/cloud.common that referenced this issue Oct 23, 2020

don't ignore parameter if value == False

a7d2361

A parameter with a `False` value can be a legit parameter that we want to pass to the module. See: ansible-collections/vmware.vmware_rest#154

goneri closed this as completed Dec 2, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MacOS - SSL verification fails with: unable to get local issuer certificate #154

MacOS - SSL verification fails with: unable to get local issuer certificate #154

kbreit commented Oct 20, 2020

Akasurde commented Oct 21, 2020

kbreit commented Oct 21, 2020

goneri commented Oct 21, 2020

goneri commented Oct 21, 2020

kbreit commented Oct 21, 2020

goneri commented Oct 21, 2020

kbreit commented Oct 21, 2020

kbreit commented Oct 21, 2020

goneri commented Oct 21, 2020

goneri commented Oct 22, 2020

kbreit commented Oct 22, 2020

goneri commented Oct 22, 2020

goneri commented Oct 22, 2020

kbreit commented Oct 22, 2020

goneri commented Oct 22, 2020

Akasurde commented Oct 23, 2020

goneri commented Oct 23, 2020

goneri commented Dec 2, 2020

MacOS - SSL verification fails with: unable to get local issuer certificate #154

MacOS - SSL verification fails with: unable to get local issuer certificate #154

Comments

kbreit commented Oct 20, 2020

SUMMARY

ISSUE TYPE

COMPONENT NAME

ANSIBLE VERSION

CONFIGURATION

OS / ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

Akasurde commented Oct 21, 2020

kbreit commented Oct 21, 2020

goneri commented Oct 21, 2020

goneri commented Oct 21, 2020

kbreit commented Oct 21, 2020

goneri commented Oct 21, 2020

kbreit commented Oct 21, 2020

kbreit commented Oct 21, 2020

goneri commented Oct 21, 2020

goneri commented Oct 22, 2020

kbreit commented Oct 22, 2020

goneri commented Oct 22, 2020

goneri commented Oct 22, 2020

kbreit commented Oct 22, 2020

goneri commented Oct 22, 2020

Akasurde commented Oct 23, 2020

goneri commented Oct 23, 2020

goneri commented Dec 2, 2020