Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

requests.exceptions.SSLError: HTTPSConnectionPool(host='our-domain', port=443): Max retries exceeded with url: / (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) #1548

Closed
Lukikrew opened this issue Jul 6, 2022 · 15 comments · Fixed by #1942

Comments

@Lukikrew
Copy link

Lukikrew commented Jul 6, 2022

Hello, we where able to compile the app with cx_freeze successfully and it opens also on all test devices. However, as soon as we try to activate the license, we receive this error:

requests.exceptions.SSLError: HTTPSConnectionPool(host='our-domain', port=443): Max retries exceeded with url: / (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available."))
That's our setup.py, any idea what we can try to solve this?

import sys
import requests
from cx_Freeze import setup, Executable


build_options = {'packages': ['requests', 'ssl'],
                "include_files": [(requests.certs.where(), 'cacert.pem')],
                 "zip_exclude_packages": []}

bdist_mac_options = {'iconfile': 'icons/icon.icns',
                     'bundle_name': 'App'}

base = 'Win32GUI' if sys.platform=='win32' else None

exe = Executable(script = 'app.py', base=base, target_name = 'App', icon = 'icons/icon.icns')

setup(name='App',
      version = '1.6.0',
      description = 'App',
      options = {
            'build_exe': build_options,
            'bdist_mac': bdist_mac_options},
      executables = [exe])

It's also weird that it works on my local device after compiling, and on some test devices. But most of the time the test users receive the SSL Error.

@marcelotduarte
Copy link
Owner

Are you using a pre-compiled cx_Freeze (binary from pypi, v6.11.1 for instance) ou 6.12 development version or are you compiling by sources?
When compiling by sources, it searches for local libraries, so some test machines have the library and other do not.
Try to make a minimum example to reproduce the behavior, maybe adapting a sample from requests and I will try to identify the issue.

@louistiti
Copy link

louistiti commented Oct 3, 2022

Hello @marcelotduarte, thanks for maintaining this awesome tool! I have tried Pyinstaller but I feel that it is not straight forward as cx_Freeze and it gets quickly complicated when dealing with many deps, where cx_Freeze makes it way easier (and it feels lighter and faster IMO).

I'm about to finish up a PoC for my open-source project. I have some Python files that use the requests package. But when I use a "frozen" environment with cx_Freeze, I receive the same error as @Lukikrew:

HTTPSConnectionPool(host='jsonplaceholder.typicode.com', port=443): Max retries exceeded with url: /todos/1 (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available."))

Here is about my env:

  • Python version: 3.9.10
  • Pipenv version: 2022.9.4
  • OS: macOS Monterey - Version 12.4 (macosx-12.4-x86_64)

Here is how to reproduce the error in a simple way:

  1. In an isolated folder, create the test.py file:
import requests

try:
    r = requests.get('https://jsonplaceholder.typicode.com/todos/1')
    print(r.json())
except requests.exceptions.RequestException as e:
    print(e)
  1. Create a new env with Pipenv:
pipenv --three
  1. Install the requests package:
pipenv install requests
  1. Install cx_Freeze:
pipenv install --dev cx_Freeze
  1. Check your virtual env path:
pipenv --venv
  1. In my case, it is ~/.local/share/virtualenvs/Workspace-C02APTuR/bin/cxfreeze so I'm gonna use this for the following command to build:
~/.local/share/virtualenvs/Workspace-C02APTuR/bin/cxfreeze --compress test.py
  1. Run:
./build/exe.macosx-12.4-x86_64-3.9/test

The error appears on that level.

Extra Info

Whereas when I run it directly from my virtual env, it works correctly:

pipenv run python test.py

# Result
{'userId': 1, 'id': 1, 'title': 'delectus aut autem', 'completed': False}

Also, when I run the following to see if the SSL module is found in the virtual env, it returns the version without any error:

pipenv run python -c "import ssl; print(ssl.OPENSSL_VERSION)"

# Result
OpenSSL 3.0.5 5 Jul 2022

I also tried to install from a development build by installing in this way:

pipenv install ./cx_Freeze-6.12.0.dev1-cp39-cp39-macosx_10_9_x86_64.whl

But the result is the same.

By trying with the urllib3 package directly (which is what requests actually uses), the error also happens, meaning that the error occurs at the urllib3 level.

Please let me know if you need further information, I will be happy to provide!

@louistiti
Copy link

It seems to be working when using cx_Freeze==6.10. Meaning something in the 6.11 release makes cx_Freeze incompatible with urllib3 (and all packages relying on it such as requests). Perhaps only for macOS? 🤔

I'm not sure what breaks it from the changelog, can anyone help to point this out?

@marcelotduarte
Copy link
Owner

I'm not sure what breaks it from the changelog, can anyone help to point this out?

Earlier I asked:

Are you using a pre-compiled cx_Freeze (binary from pypi, v6.11.1 for instance) ou 6.12 development version or are you compiling by sources?

My assumption was that the PR #1505 is breaking something:

When compiling by sources, it searches for local libraries, so some test machines have the library and other do not.

Try using:
pip install --no-binary=cx_Freeze cx_Freeze

@louistiti
Copy link

louistiti commented Oct 4, 2022

Yes it seems to make the job. I'm using Pipenv, so I'm doing the following command to install packages from the Pipfile:

PIP_NO_BINARY=cx_Freeze pipenv install

@marcelotduarte
Copy link
Owner

marcelotduarte commented Oct 13, 2022

cx_Freeze 6.12.0 has just been released.
Can you check if this has been resolved?

@adamcharnock
Copy link

adamcharnock commented Nov 26, 2022

This works for me just fine if I downgrade to cx-Freeze==6.11.1.

But cx-Freeze==6.12.0 and cx-Freeze==6.13.1 give the error described above:

Traceback (most recent call last):
  File "/Users/adam/Library/Caches/pypoetry/virtualenvs/bni-CHj4EBEx-py3.11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 692, in urlopen
    conn = self._get_conn(timeout=pool_timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/adam/Library/Caches/pypoetry/virtualenvs/bni-CHj4EBEx-py3.11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 281, in _get_conn
    return conn or self._new_conn()
                   ^^^^^^^^^^^^^^^^
  File "/Users/adam/Library/Caches/pypoetry/virtualenvs/bni-CHj4EBEx-py3.11/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1011, in _new_conn
    raise SSLError(
urllib3.exceptions.SSLError: Can't connect to HTTPS URL because the SSL module is not available.

However, if I use pip3 install --no-binary=cx_Freeze "cx-freeze==6.xx.xx" then it works on all three versions I list above

@marcelotduarte
Copy link
Owner

@louistiti @adamcharnock Can you test using certifi?

import requests
import certifi

try:
    r = requests.get('https://jsonplaceholder.typicode.com/todos/1', cert=certifi.where())
    print(r.json())
except requests.exceptions.RequestException as e:
    print(e)

A second test is:

import urllib.request

resp = urllib.request.urlopen('https://wikipedia.org')

Does it fail?
And:

import urllib.request
import certifi

resp = urllib.request.urlopen('https://wikipedia.org', cafile=certifi.where())

Does it work?

@adamcharnock
Copy link

Sorry for the slow reply. I've just come back to this because it has stopped working again. I'm digging into this now, but to answer your questions, all three produce errors. The three errors (respectively) are:

HTTPSConnectionPool(host='jsonplaceholder.typicode.com', port=443): Max retries exceeded with url: /todos/1 (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available."))
Error while running command: URLError: <urlopen error unknown url type: https>
  File "/Users/github/actions-runner/_work/infrastructure/infrastructure/python/bn/bni/commands/self/test.py", line 23, in command_check_ssl2
    resp = urllib.request.urlopen('https://wikipedia.org')
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.0/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.0/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 519, in open
    response = self._open(req, data)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.0/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 541, in _open
    return self._call_chain(self.handle_open, 'unknown',
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.0/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.0/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1419, in unknown_open
    raise URLError('unknown url type: %s' % type)
urllib.error.URLError: <urlopen error unknown url type: https>
  File "/Users/github/actions-runner/_work/infrastructure/infrastructure/python/bn/bni/commands/self/test.py", line 31, in command_check_ssl3
    resp = urllib.request.urlopen('https://wikipedia.org', cafile=certifi.where())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.0/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 201, in urlopen
    raise ValueError('SSL support not available')
ValueError: SSL support not available

Also running a test for a simple import ssl:

  File "/Users/github/actions-runner/_work/infrastructure/infrastructure/python/bn/bni/commands/self/test.py", line 36, in command_check_ssl4
    import ssl
  File "/opt/homebrew/Cellar/python@3.11/3.11.0/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 100, in <module>
    import _ssl             # if we can't import it, let the error propagate
    ^^^^^^^^^^^
ImportError: dlopen(/Users/github/actions-runner/_work/infrastructure/infrastructure/build/bni_0.1-1_darwin_arm64/lib/_ssl.cpython-311-darwin.so, 0x0002): Library not loaded: @loader_path/../lib/libssl.1.1.dylib
  Referenced from: <82966696-D797-3F45-9A10-32A4AD4F9D1E> /Users/github/actions-runner/_work/infrastructure/infrastructure/build/bni_0.1-1_darwin_arm64/lib/_ssl.cpython-311-darwin.so
  Reason: tried: '/Users/github/actions-runner/_work/infrastructure/infrastructure/build/bni_0.1-1_darwin_arm64/lib/../lib/libssl.1.1.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS@loader_path/../lib/libssl.1.1.dylib' (no such file), '/Users/github/actions-runner/_work/infrastructure/infrastructure/build/bni_0.1-1_darwin_arm64/lib/../lib/libssl.1.1.dylib' (no such file), '/usr/local/lib/libssl.1.1.dylib' (no such file), '/usr/lib/libssl.1.1.dylib' (no such file, not in dyld cache)

Which seems like a clue.

Also, FYI:

$ otool -L ../../../build/bni_0.1-1_darwin_arm64/bni
../../../build/bni_0.1-1_darwin_arm64/bni:
	@loader_path/lib/Python (compatibility version 3.11.0, current version 3.11.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1856.105.0)

Will continue looking into this momentarily.

@adamcharnock
Copy link

Right, so this file doesn't exist

build/bni_0.1-1_darwin_arm64/lib/../lib/libssl.1.1.dylib
build/bni_0.1-1_darwin_arm64/lib/libssl.1.1.dylib (normalised path)

But that files does exist here:

build/bni_0.1-1_darwin_arm64/libssl.1.1.dylib

In fact, there are a few dylib files in there: libcrypto.1.1.dylib, libncursesw.5.dylib, libpanelw.5.dylib, libssl.1.1.dylib, libtcl8.6.dylib, libtk8.6.dylib

So I'm guessing the build process isn't putting the .dylib files into the lib/ directory for some reason.

@adamcharnock
Copy link

adamcharnock commented Jan 10, 2023

Ok, I've put in place a bit of a hack in the build process to just see if I can get to a solution. I just move all the .dylib files to the lib/ directory:

    for dylib_file in build_directory.glob("*.dylib"):
        dylib_file.rename(dylib_file.parent / "lib" / dylib_file.name)

Which seems to help. Tests 3 and 4 (test 4 is import ssl) both work. Tests 1 and 2 provide respectively:

HTTPSConnectionPool(host='jsonplaceholder.typicode.com', port=443): Max retries exceeded with url: /todos/1 (Caused by SSLError(SSLError(9, '[SSL] PEM lib (_ssl.c:3895)')))
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)>

So, progress.


Update: It actually seems to work now. In test one, changing this:

r = requests.get('https://jsonplaceholder.typicode.com/todos/1', cert=certifi.where())

to this:

r = requests.get('https://jsonplaceholder.typicode.com/todos/1')

resulted in test 1 now working. I guess requests knows where to find its own certs.

@tatablack
Copy link

tatablack commented Jan 18, 2023

Same problem here (MacOS 13.1, cx_Freeze 6.13.1 installed via pip).

@adamcharnock's hack works for me too, and since it took me a while to figure out how to actually put it in place 😅 (I have zero experience with setuptools), I thought it worth adding it here:

from pathlib import Path

from cx_Freeze import setup, build_exe

class DyLibFixerCommand(build_exe):
    def run(self):
        super().run()

        for dylib_file in Path(self.build_exe).glob("*.dylib"):
            dylib_file.rename(dylib_file.parent / "lib" / dylib_file.name)


COMMAND_CLASS = {"build_exe": DyLibFixerCommand}

setup(
    ...  # Your configuration here
    cmdclass=COMMAND_CLASS,
)

@marcelotduarte
Copy link
Owner

Sorry for the delay in looking at this. I don't work with Mac, and usually, a contributor looked at this part, but he migrated to Windows.
Testing using CI and using what @adamcharnock found, I developed a patch. Please test and give me feedback.

You can test the patch in the latest development build:
pip install --upgrade --pre --extra-index-url https://marcelotduarte.github.io/packages/ cx_Freeze

@tatablack
Copy link

Ah, thanks, but unfortunately I don't work on that project anymore, and I don't have access to the source code.
Hopefully one of the others in this thread will be able to verify.

Thanks anyway, and I completely understand the delay! 🙇🏻

@marcelotduarte
Copy link
Owner

Release 6.15.3 is out!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants