GH-84850: Remove `urllib.request.URLopener` and `FancyURLopener` #125739

barneygale · 2024-10-20T00:06:58Z

These classes have been deprecated since Python 3.3. They originate in Python 2's urllib, whereas most of the rest of this module comes from urllib2.

Issue: remove deprecated urllib.request.URLopener/FancyURLopener #84850

📚 Documentation preview 📚: https://cpython-previews--125739.org.readthedocs.build/

These classes have been deprecated since Python 3.3.

Lib/urllib/request.py

vstinner · 2024-10-20T13:50:45Z

A code search on PyPI top 8,000 projects (at 2024-10-16) on URLopener finds 21 projects:

arelle_release (2.31.6)
bandit (1.7.10)
cinemagoer (2023.5.1)
eventlet (0.37.0)
fissix (24.4.24)
flake8_tidy_imports (4.10.0)
future (1.0.0)
jplephem (2.22)
m2crypto (0.42.0)
newrelic (10.2.0)
pscript (0.7.7)
py-backwards (0.7)
pygments (2.18.0)
reportlab (4.2.5)
ruff (0.6.9)
tf_keras (2.17.0)
tf_keras_nightly-2.19.0.dev2024101609
wagon (1.0.1)
wandb (0.18.3)
wxPython (4.2.2)
yt (4.3.1)

I didn't check the code to see if imports are conditionals on Python <= 3.3 or not. I ignored six.py copies and .pyi files.

PYPI-2024-10-16/arelle_release-2.31.6.tar.gz: arelle_release-2.31.6/arelle/WebCache.py: class WebCacheUrlOpener(request.FancyURLopener):
PYPI-2024-10-16/bandit-1.7.10.tar.gz: bandit-1.7.10/bandit/blacklists/calls.py: |      |                     | - urllib.URLopener                 |           |
PYPI-2024-10-16/bandit-1.7.10.tar.gz: bandit-1.7.10/bandit/blacklists/calls.py: |      |                     | - urllib.request.URLopener         |           |
PYPI-2024-10-16/bandit-1.7.10.tar.gz: bandit-1.7.10/bandit/blacklists/calls.py: |      |                     | - urllib.FancyURLopener            |           |
PYPI-2024-10-16/bandit-1.7.10.tar.gz: bandit-1.7.10/bandit/blacklists/calls.py: |      |                     | - urllib.request.FancyURLopener    |           |
PYPI-2024-10-16/bandit-1.7.10.tar.gz: bandit-1.7.10/bandit/blacklists/calls.py: |      |                     |   .URLopener                       |           |
PYPI-2024-10-16/bandit-1.7.10.tar.gz: bandit-1.7.10/bandit/blacklists/calls.py: |      |                     |   .FancyURLopener                  |           |
PYPI-2024-10-16/bandit-1.7.10.tar.gz: bandit-1.7.10/bandit/blacklists/calls.py: "urllib.request.URLopener",
PYPI-2024-10-16/bandit-1.7.10.tar.gz: bandit-1.7.10/bandit/blacklists/calls.py: "urllib.request.FancyURLopener",
PYPI-2024-10-16/bandit-1.7.10.tar.gz: bandit-1.7.10/examples/urlopen.py: opener = urllib.request.URLopener()
PYPI-2024-10-16/bandit-1.7.10.tar.gz: bandit-1.7.10/examples/urlopen.py: opener = urllib.request.FancyURLopener()
PYPI-2024-10-16/cinemagoer-2023.5.1.tar.gz: cinemagoer-2023.5.1/imdb/parser/http/__init__.py: class IMDbURLopener:
PYPI-2024-10-16/cinemagoer-2023.5.1.tar.gz: cinemagoer-2023.5.1/imdb/parser/http/__init__.py: self.urlOpener = IMDbURLopener(*arguments, **keywords)
PYPI-2024-10-16/cinemagoer-2023.5.1.tar.gz: cinemagoer-2023.5.1/tests/conftest.py: from imdb.parser.http import IMDbURLopener
PYPI-2024-10-16/cinemagoer-2023.5.1.tar.gz: cinemagoer-2023.5.1/tests/conftest.py: retrieve_unicode_orig = IMDbURLopener.retrieve_unicode
PYPI-2024-10-16/cinemagoer-2023.5.1.tar.gz: cinemagoer-2023.5.1/tests/conftest.py: IMDbURLopener.retrieve_unicode = retrieve_unicode_cached
PYPI-2024-10-16/cinemagoer-2023.5.1.tar.gz: cinemagoer-2023.5.1/tests/conftest.py: IMDbURLopener.retrieve_unicode = retrieve_unicode_orig
PYPI-2024-10-16/eventlet-0.37.0.tar.gz: eventlet-0.37.0/eventlet/green/urllib/request.py: URLopener.open_ftp = patcher.patch_function(URLopener.open_ftp, *to_patch_in_functions)
PYPI-2024-10-16/fissix-24.4.24.tar.gz: fissix-24.4.24/fissix/fixes/fix_urllib.py: "URLopener",
PYPI-2024-10-16/fissix-24.4.24.tar.gz: fissix-24.4.24/fissix/fixes/fix_urllib.py: "FancyURLopener",
PYPI-2024-10-16/flake8_tidy_imports-4.10.0.tar.gz: flake8_tidy_imports-4.10.0/src/flake8_tidy_imports/__init__.py: "urllib.FancyURLopener": (
PYPI-2024-10-16/flake8_tidy_imports-4.10.0.tar.gz: flake8_tidy_imports-4.10.0/src/flake8_tidy_imports/__init__.py: "urllib.URLopener": (
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/src/future/moves/urllib/request.py: #                     URLopener,
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/src/future/moves/urllib/request.py: #                     FancyURLopener,
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/src/libpasteurize/fixes/fix_imports2.py: u'ContentTooShortError', u'FancyURLopener',u'URLopener',
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib.py: # Shortcut for testing FancyURLopener
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib.py: opener = urllib_request.FancyURLopener(proxies=proxies)
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib.py: ('FancyURLopener style of invoking requests is deprecated.',
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib.py: opener = urllib_request.FancyURLopener()
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib.py: def test_URLopener_deprecation(self):
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib.py: urllib_request.URLopener()
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib.py: class URLopener_Tests(unittest.TestCase):
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib.py: """Testcase to test the open method of URLopener class."""
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib.py: class DummyURLopener(urllib_request.URLopener):
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib.py: ('DummyURLopener style of invoking requests is deprecated.',
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib.py: self.assertEqual(DummyURLopener().open(
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib.py: self.assertEqual(DummyURLopener().open(
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib_toplevel.py: # Shortcut for testing FancyURLopener
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib_toplevel.py: opener = urllib_request.FancyURLopener(proxies=proxies)
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib_toplevel.py: ('FancyURLopener style of invoking requests is deprecated.',
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib_toplevel.py: opener = urllib_request.FancyURLopener()
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib_toplevel.py: def test_URLopener_deprecation(self):
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib_toplevel.py: urllib_request.URLopener()
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib_toplevel.py: class URLopener_Tests(unittest.TestCase):
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib_toplevel.py: """Testcase to test the open method of URLopener class."""
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib_toplevel.py: class DummyURLopener(urllib_request.URLopener):
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib_toplevel.py: ('DummyURLopener style of invoking requests is deprecated.',
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib_toplevel.py: self.assertEqual(DummyURLopener().open(
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllib_toplevel.py: self.assertEqual(DummyURLopener().open(
PYPI-2024-10-16/future-1.0.0.tar.gz: future-1.0.0/tests/test_future/test_urllibnet.py: open_url = urllib_request.FancyURLopener().open(URL)
PYPI-2024-10-16/jplephem-2.22.tar.gz: jplephem-2.22/jplephem/excerpter.py: from urllib.request import URLopener
PYPI-2024-10-16/jplephem-2.22.tar.gz: jplephem-2.22/jplephem/excerpter.py: from urllib import URLopener
PYPI-2024-10-16/jplephem-2.22.tar.gz: jplephem-2.22/jplephem/excerpter.py: self.opener = URLopener()
PYPI-2024-10-16/m2crypto-0.42.0.tar.gz: m2crypto-0.42.0/src/M2Crypto/m2urllib.py: URLopener.open_https = open_https
PYPI-2024-10-16/newrelic-10.2.0.tar.gz: newrelic-10.2.0/newrelic/hooks/external_urllib.py: if hasattr(module, 'URLopener'):
PYPI-2024-10-16/newrelic-10.2.0.tar.gz: newrelic-10.2.0/newrelic/hooks/external_urllib.py: wrap_function_wrapper(module, 'URLopener.open',
PYPI-2024-10-16/newrelic-10.2.0.tar.gz: newrelic-10.2.0/tests/external_httplib/test_urllib.py: opener = urllib.URLopener()
PYPI-2024-10-16/newrelic-10.2.0.tar.gz: newrelic-10.2.0/tests/external_httplib/test_urllib.py: opener = urllib.URLopener()
PYPI-2024-10-16/newrelic-10.2.0.tar.gz: newrelic-10.2.0/tests/external_httplib/test_urllib.py: opener = urllib.URLopener()
PYPI-2024-10-16/newrelic-10.2.0.tar.gz: newrelic-10.2.0/tests/external_httplib/test_urllib.py: opener = urllib.URLopener()
PYPI-2024-10-16/newrelic-10.2.0.tar.gz: newrelic-10.2.0/tests/external_httplib/test_urllib.py: opener = urllib.URLopener()
PYPI-2024-10-16/newrelic-10.2.0.tar.gz: newrelic-10.2.0/tests/external_httplib/test_urllib.py: opener = urllib.URLopener()
PYPI-2024-10-16/pscript-0.7.7.tar.gz: pscript-0.7.7/translate_to_legacy.py: 'ContentTooShortError', 'FancyURLopener', 'URLopener',
PYPI-2024-10-16/py-backwards-0.7.tar.gz: py-backwards-0.7/py_backwards/transformers/six_moves.py: MovedAttribute("URLopener", "urllib", "urllib.request"),
PYPI-2024-10-16/py-backwards-0.7.tar.gz: py-backwards-0.7/py_backwards/transformers/six_moves.py: MovedAttribute("FancyURLopener", "urllib", "urllib.request"),
PYPI-2024-10-16/pygments-2.18.0.tar.gz: pygments-2.18.0/pygments/lexers/_sourcemod_builtins.py: from urllib.request import FancyURLopener
PYPI-2024-10-16/pygments-2.18.0.tar.gz: pygments-2.18.0/pygments/lexers/_sourcemod_builtins.py: class Opener(FancyURLopener):
PYPI-2024-10-16/reportlab-4.2.5.tar.gz: reportlab-4.2.5/src/reportlab/lib/utils.py: #copied here from urllib.URLopener.open_data because
PYPI-2024-10-16/ruff-0.6.9.tar.gz: ruff-0.6.9/crates/ruff_linter/src/rules/flake8_bandit/rules/suspicious_function_call.rs: // URLOpen (`URLopener`, `FancyURLopener`)
PYPI-2024-10-16/ruff-0.6.9.tar.gz: ruff-0.6.9/crates/ruff_linter/src/rules/flake8_bandit/rules/suspicious_function_call.rs: ["urllib", "request", "URLopener" | "FancyURLopener"] |
PYPI-2024-10-16/tf_keras-2.17.0.tar.gz: tf_keras-2.17.0/tf_keras/src/utils/data_utils.py: Under Python 2, `urlretrieve` relies on `FancyURLopener` from legacy
PYPI-2024-10-16/tf_keras_nightly-2.19.0.dev2024101609.tar.gz: tf_keras_nightly-2.19.0.dev2024101609/tf_keras/src/utils/data_utils.py: Under Python 2, `urlretrieve` relies on `FancyURLopener` from legacy
PYPI-2024-10-16/wagon-1.0.1.tar.gz: wagon-1.0.1/wagon.py: from urllib.request import URLopener
PYPI-2024-10-16/wagon-1.0.1.tar.gz: wagon-1.0.1/wagon.py: from urllib import URLopener
PYPI-2024-10-16/wagon-1.0.1.tar.gz: wagon-1.0.1/wagon.py: f = URLopener()
PYPI-2024-10-16/wandb-0.18.3.tar.gz: wandb-0.18.3/wandb/vendor/pygments/lexers/_sourcemod_builtins.py: from urllib import FancyURLopener
PYPI-2024-10-16/wandb-0.18.3.tar.gz: wandb-0.18.3/wandb/vendor/pygments/lexers/_sourcemod_builtins.py: from urllib.request import FancyURLopener
PYPI-2024-10-16/wandb-0.18.3.tar.gz: wandb-0.18.3/wandb/vendor/pygments/lexers/_sourcemod_builtins.py: class Opener(FancyURLopener):
PYPI-2024-10-16/wxPython-4.2.2.tar.gz: wxPython-4.2.2/wx/lib/pubsub/py2and3.py: MovedAttribute("URLopener", "urllib", "urllib.request"),
PYPI-2024-10-16/wxPython-4.2.2.tar.gz: wxPython-4.2.2/wx/lib/pubsub/py2and3.py: MovedAttribute("FancyURLopener", "urllib", "urllib.request"),
PYPI-2024-10-16/yt-4.3.1.tar.gz: yt-4.3.1/yt/funcs.py: class MyURLopener(urllib.request.FancyURLopener):
PYPI-2024-10-16/yt-4.3.1.tar.gz: yt-4.3.1/yt/funcs.py: fn, h = MyURLopener().retrieve(url, filename)

cc @hugovk

barneygale · 2024-10-21T13:50:11Z

Thanks Victor. From that list, I think about 6 packages are using URLopener in anger, while the remainder are naming coincidences, linter rules, or vendoring of one of affected packages.

barneygale · 2024-10-22T18:09:30Z

I believe only four packages would break for users if we removed these classes: eventlet, jplephem, wagon, and yt. It looks like eventlet is best fixed with a Python version conditional, so we can't fix that til this patch lands. I've opened issues for the other packages:

I'll work on PRs if the package authors seem interested.

barneygale · 2024-10-25T17:18:43Z

I've submitted PRs against the three projects, one of which has been accepted already. They're all simple changes - in Python 2 terms, we're upgrading from urllib to urllib2.

Given that 3.14 is still around a year away, and only 3 projects in 8,000 are affected and possible to remedy at this stage, I submit that we don't need to wait for the other two PRs. Thoughts?

hugovk · 2024-10-25T17:36:45Z

Given that 3.14 is still around a year away, and only 3 projects in 8,000 are affected and possible to remedy at this stage, I submit that we don't need to wait for the other two PRs. Thoughts?

Agreed, this is definitely on the low side.

Lib/test/test_urllib.py

Doc/whatsnew/3.14.rst

python#125739)

pythonGH-84850: Remove urllib.request.URLopener and FancyURLopener

2b3f775

These classes have been deprecated since Python 3.3.

bedevere-app bot mentioned this pull request Oct 20, 2024

remove deprecated urllib.request.URLopener/FancyURLopener #84850

Closed

bedevere-app bot added the awaiting core review label Oct 20, 2024

Typo fix

6544b89

barneygale requested a review from vstinner October 20, 2024 04:10

vstinner reviewed Oct 20, 2024

View reviewed changes

Lib/urllib/request.py Show resolved Hide resolved

Delay import warnings

062116e

barneygale commented Oct 25, 2024

View reviewed changes

Lib/test/test_urllib.py Show resolved Hide resolved

Remove unused helper class

041529e

barneygale requested a review from vstinner October 29, 2024 16:07

Merge branch 'main' into pythongh-84850

ea17ffc

barneygale mentioned this pull request Oct 30, 2024

GH-125866: RFC8089 file URIs in urllib.request #126148

Closed

hugovk approved these changes Nov 17, 2024

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting core review labels Nov 17, 2024

vstinner reviewed Nov 18, 2024

View reviewed changes

Doc/whatsnew/3.14.rst Show resolved Hide resolved

hugovk merged commit 4d77197 into python:main Nov 19, 2024
40 checks passed

bedevere-app bot removed the awaiting merge label Nov 19, 2024

ebonnal pushed a commit to ebonnal/cpython that referenced this pull request Jan 12, 2025

pythonGH-84850: Remove urllib.request.URLopener and FancyURLopener (

21980f6

python#125739)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-84850: Remove `urllib.request.URLopener` and `FancyURLopener` #125739

GH-84850: Remove `urllib.request.URLopener` and `FancyURLopener` #125739

barneygale commented Oct 20, 2024 •

edited

Loading

vstinner commented Oct 20, 2024 •

edited

Loading

barneygale commented Oct 21, 2024

barneygale commented Oct 22, 2024 •

edited

Loading

barneygale commented Oct 25, 2024

hugovk commented Oct 25, 2024

GH-84850: Remove urllib.request.URLopener and FancyURLopener #125739

GH-84850: Remove urllib.request.URLopener and FancyURLopener #125739

Conversation

barneygale commented Oct 20, 2024 • edited Loading

vstinner commented Oct 20, 2024 • edited Loading

barneygale commented Oct 21, 2024

barneygale commented Oct 22, 2024 • edited Loading

barneygale commented Oct 25, 2024

hugovk commented Oct 25, 2024

GH-84850: Remove `urllib.request.URLopener` and `FancyURLopener` #125739

GH-84850: Remove `urllib.request.URLopener` and `FancyURLopener` #125739

barneygale commented Oct 20, 2024 •

edited

Loading

vstinner commented Oct 20, 2024 •

edited

Loading

barneygale commented Oct 22, 2024 •

edited

Loading