Proposal: add new builder that checks if hardcoded URLs can be replaced with crossrefs

CHANGES

@@ -1063,6 +1063,8 @@ Features added
 * #9075: autodoc: Add a config variable :confval:`autodoc_typehints_format`
   to suppress the leading module names of typehints of function signatures (ex.
   ``io.StringIO`` -> ``StringIO``)
+* #9626: intersphinx: Emit warning if a hardcoded link is replaceable
+  by an intersphinx cross-reference, suggesting a replacement.
 * #9831: Autosummary now documents only the members specified in a module's
   ``__all__`` attribute if :confval:`autosummary_ignore_module_all` is set to
   ``False``. The default behaviour is unchanged. Autogen also now supports

doc/conf.py

@@ -116,8 +116,8 @@
 
 intersphinx_mapping = {
     'python': ('https://docs.python.org/3/', None),
-    'requests': ('https://requests.readthedocs.io/en/latest/', None),
     'readthedocs': ('https://docs.readthedocs.io/en/stable', None),
+    'requests': ('https://requests.readthedocs.io/en/latest/', None),
 }
 
 # Sphinx document translation with sphinx gettext feature uses these settings:

doc/development/tutorials/helloworld.rst

@@ -129,7 +129,7 @@ Using the extension
 The extension has to be declared in your :file:`conf.py` file to make Sphinx
 aware of it. There are two steps necessary here:
 
-#. Add the :file:`_ext` directory to the `Python path`_ using
+#. Add the :file:`_ext` directory to the :std:envvar:`PYTHONPATH` using
    ``sys.path.append``. This should be placed at the top of the file.
 
 #. Update or create the :confval:`extensions` list and add the extension file
@@ -149,8 +149,8 @@ For example:
 .. tip::
 
    We're not distributing this extension as a `Python package`_, we need to
-   modify the `Python path`_ so Sphinx can find our extension. This is why we
-   need the call to ``sys.path.append``.
+   modify the :std:envvar:`PYTHONPATH` so Sphinx can find our extension.
+   This is why we need the call to ``sys.path.append``.
 
 You can now use the extension in a file. For example:
 
@@ -186,4 +186,3 @@ For a more advanced example, refer to :doc:`todo`.
 .. _docutils nodes: https://docutils.sourceforge.io/docs/ref/doctree.html
 .. _PyPI: https://pypi.org/
 .. _Python package: https://packaging.python.org/
-.. _Python path: https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH

doc/development/tutorials/todo.rst

@@ -291,7 +291,7 @@ Using the extension
 As before, we need to enable the extension by declaring it in our
 :file:`conf.py` file. There are two steps necessary here:
 
-#. Add the :file:`_ext` directory to the `Python path`_ using
+#. Add the :file:`_ext` directory to the :std:envvar:`PYTHONPATH` using
    ``sys.path.append``. This should be placed at the top of the file.
 
 #. Update or create the :confval:`extensions` list and add the extension file
@@ -363,5 +363,4 @@ For more information, refer to the `docutils`_ documentation and
 
 
 .. _docutils: https://docutils.sourceforge.io/docs/
-.. _Python path: https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPATH
 .. _docutils documentation: https://docutils.sourceforge.io/docs/ref/rst/directives.html

doc/faq.rst

@@ -48,10 +48,11 @@ Using Sphinx with...
 --------------------
 
 Read the Docs
-    `Read the Docs <https://readthedocs.org>`_ is a documentation hosting
+    :std:doc:`readthedocs:index` is a documentation hosting
     service based around Sphinx.  They will host sphinx documentation, along
     with supporting a number of other features including version support, PDF
-    generation, and more. The `Getting Started`_ guide is a good place to start.
+    generation, and more. The :std:ref:`Getting Started
+    <readthedocs:tutorial/index:getting started>` guide is a good place to start.
 
 Epydoc
    There's a third-party extension providing an `api role`_ which refers to
@@ -145,7 +146,6 @@ Google Search
 
    3. Add ``searchbox.html`` to the :confval:`html_sidebars` configuration value.
 
-.. _Getting Started: https://docs.readthedocs.io/en/stable/intro/getting-started-with-sphinx.html
 .. _api role: https://git.savannah.gnu.org/cgit/kenozooid.git/tree/doc/extapi.py
 .. _xhtml to reST: https://docutils.sourceforge.io/sandbox/xhtml2rest/xhtml2rest.py
 

doc/man/sphinx-apidoc.rst

@@ -16,10 +16,8 @@ package in the style of other automatic API documentation tools.
 
 *MODULE_PATH* is the path to a Python package to document, and *OUTPUT_PATH* is
 the directory where the generated sources are placed. Any *EXCLUDE_PATTERN*\s
-given are `fnmatch-style`_ file and/or directory patterns that will be excluded
-from generation.
-
-.. _fnmatch-style: https://docs.python.org/3/library/fnmatch.html
+given are :std:doc:`python:library/fnmatch`\-style file and/or directory patterns
+that will be excluded from generation.
 
 .. warning::
 
@@ -167,5 +165,3 @@ See also
 --------
 
 :manpage:`sphinx-build(1)`, :manpage:`sphinx-autogen(1)`
-
-.. _fnmatch: https://docs.python.org/3/library/fnmatch.html

doc/usage/configuration.rst

@@ -608,13 +608,11 @@ General configuration
 
    .. versionadded:: 1.5
 
-   .. tip:: Sphinx uses requests_ as a HTTP library internally.
+   .. tip:: Sphinx uses :std:doc:`requests:index` as a HTTP library internally.
             Therefore, Sphinx refers a certification file on the
             directory pointed ``REQUESTS_CA_BUNDLE`` environment
             variable if ``tls_cacerts`` not set.
 
-            .. _requests: https://requests.readthedocs.io/en/master/
-
 .. confval:: today
              today_fmt
 

doc/usage/extensions/coverage.rst

@@ -24,7 +24,7 @@ should check:
 
 .. confval:: coverage_ignore_pyobjects
 
-   List of `Python regular expressions`_.
+   List of :std:doc:`Python regular expressions <python:library/re>`.
 
    If any of these regular expressions matches any part of the full import path
    of a Python object, that Python object is excluded from the documentation

doc/usage/restructuredtext/directives.rst

@@ -1055,8 +1055,7 @@ Including content based on tags
    are not available there.
 
    All tags must follow the standard Python identifier syntax as set out in
-   the `Identifiers and keywords
-   <https://docs.python.org/3/reference/lexical_analysis.html#identifiers>`_
+   the :std:ref:`identifiers`
    documentation.  That is, a tag expression may only consist of tags that
    conform to the syntax of Python variables.  In ASCII, this consists of the
    uppercase and lowercase letters ``A`` through ``Z``, the underscore ``_``

sphinx/ext/intersphinx.py

@@ -25,7 +25,7 @@
 import sys
 import time
 from os import path
-from typing import TYPE_CHECKING, cast
+from typing import IO, TYPE_CHECKING, Any, cast
 from urllib.parse import urlsplit, urlunsplit
 
 from docutils import nodes
@@ -36,15 +36,15 @@
 from sphinx.builders.html import INVENTORY_FILENAME
 from sphinx.errors import ExtensionError
 from sphinx.locale import _, __
-from sphinx.transforms.post_transforms import ReferencesResolver
+from sphinx.transforms.post_transforms import ReferencesResolver, SphinxPostTransform
 from sphinx.util import logging, requests
 from sphinx.util.docutils import CustomReSTDispatcher, SphinxRole
 from sphinx.util.inventory import InventoryFile
 
 if TYPE_CHECKING:
-    from collections.abc import Iterable
+    from collections.abc import Iterable, Iterator
     from types import ModuleType
-    from typing import IO, Any, Union
+    from typing import Union
 
     from docutils.nodes import Node, TextElement, system_message
     from docutils.utils import Reporter
@@ -98,6 +98,77 @@ def clear(self) -> None:
         self.env.intersphinx_named_inventory.clear()  # type: ignore[attr-defined]
 
 
+class ExternalLinksChecker(SphinxPostTransform):
+    """
+    For each external link, check if it can be replaced by a crossreference.
+
+    We treat each ``reference`` node without ``internal`` attribute as an external link.
+    """
+
+    default_priority = 500
+
+    def run(self, **kwargs: Any) -> None:
+        for refnode in self.document.findall(nodes.reference):
+            self.check_uri(refnode)
+
+    def check_uri(self, refnode: nodes.reference) -> None:
+        """
+        If the URI in ``refnode`` has a replacement in an ``intersphinx`` inventory,
+        emit a warning with a replacement suggestion.
+        """
+        if 'internal' in refnode or 'refuri' not in refnode:
+            return
+
+        uri = refnode['refuri']
+        cache = getattr(self.app.env, 'intersphinx_cache', {})
+
+        for inventory_uri, (inventory_name, _size, _inventory) in cache.items():
+            if uri.startswith(inventory_uri):
+                # build a replacement suggestion
+                replacements = find_replacements(self.app, uri)
+                try:
+                    suggestion = f'try using {next(replacements)!r} instead'
+                    logger.warning(
+                        __(
+                            'hardcoded link %r could be replaced by '
+                            'a cross-reference to %r inventory (%s)',
+                        ),
+                        uri,
+                        inventory_name,
+                        suggestion,
+                        location=refnode,
+                    )
+                except StopIteration:
+                    pass
+
+
+def find_replacements(app: Sphinx, uri: str) -> Iterator[str]:
+    """
+    Try finding a crossreference to replace hardcoded ``uri``.
+
+    This is straightforward: search the available inventories
+    for an entry that points to the given ``uri`` and build
+    a ReST markup that should replace ``uri`` with a crossref.
+    """
+    cache = getattr(app.env, 'intersphinx_cache', {})
+
+    for inventory_uri, (_inventory_name, _size, inventory) in cache.items():
+        if uri.startswith(inventory_uri):
+            for key, entries in inventory.items():
+                domain_name, directive_type = key.split(':')
+                for target, (
+                    _project_name,
+                    _version,
+                    target_uri,
+                    _display_name,
+                ) in entries.items():
+                    if uri == target_uri:
+                        for domain in app.env.domains.values():
+                            if domain_name == domain.name:
+                                role = domain.role_for_objtype(directive_type) or 'any'
+                                yield f':{domain_name}:{role}:`{target}`'
+
+
 def _strip_basic_auth(url: str) -> str:
     """Returns *url* with basic auth credentials removed. Also returns the
     basic auth username and password if they're present in *url*.
@@ -247,7 +318,7 @@ def fetch_inventory_group(
         else:
             issues = '\n'.join([f[0] % f[1:] for f in failures])
             logger.warning(__("failed to reach any of the inventories "
-                              "with the following issues:") + "\n" + issues)
+                           "with the following issues:") + "\n" + issues)
 
 
 def load_mappings(app: Sphinx) -> None:
@@ -691,6 +762,8 @@ def setup(app: Sphinx) -> dict[str, Any]:
     app.connect('source-read', install_dispatcher)
     app.connect('missing-reference', missing_reference)
     app.add_post_transform(IntersphinxRoleResolver)
+    app.add_post_transform(ExternalLinksChecker)
+
     return {
         'version': sphinx.__display_version__,
         'env_version': 1,
@@ -737,6 +810,7 @@ class MockApp:
 
 if __name__ == '__main__':
     import logging as _logging
+
     _logging.basicConfig()
 
     raise SystemExit(inspect_main(sys.argv[1:]))

tests/roots/test-ext-intersphinx-hardcoded-urls/conf.py

@@ -0,0 +1,2 @@
+extensions = ['sphinx.ext.intersphinx']
+intersphinx_mapping = {'python': ('https://example.com', 'inventory.inv')}

tests/roots/test-ext-intersphinx-hardcoded-urls/index.rst

@@ -0,0 +1,37 @@
+test-ext-intersphinx-hardcoded-urls
+===================================
+
+.. Links generated by intersphinx crossrefs should not raise any warnings.
+.. Only hardcoded URLs are affected.
+
+:py:mod:`module1`
+
+.. hardcoded replaceable link
+
+https://example.com/foo.html#module-module1
+
+`inline replaceable link <https://example.com/foo.html#module-module1>`_
+
+`replaceable link`_
+
+.. hardcoded non-replaceable link
+
+https://example.com/spam.html#eggs
+
+`inline non-replaceable link <https://example.com/spam.html#eggs>`_
+
+`non-replaceable link`_
+
+.. hardcoded external link
+
+https://spam.eggs
+
+`inline external link <https://spam.eggs>`_
+
+`external link`_
+
+.. hyperlinks
+
+.. _replaceable link: https://example.com/foo.html#module-module1
+.. _non-replaceable link: https://example.com/spam.html#eggs
+.. _external link: https://spam.eggs

tests/roots/test-ext-intersphinx-hardcoded-urls/inventory.inv

@@ -0,0 +1,5 @@
+# Sphinx inventory version 2
+# Project: foo
+# Version: 2.0
+# The remainder of this file is compressed with zlib.
+x����N�0��y�Hp��;�A�i<C�&�V�?�I����N�nC�ĩ����ږ]Ѓ5�F�r�_�!��)9�T2�&f��J��X���&����oj���C�X��&���&��F��ڜJ�w�#򃦏UV��[�(.lQ�Ť��:����V#%�B�Oou��3
��Ȱ�2�[�525�q�����r��uW]�×�w:*��P�>3�zf�I�L�8���цe?eAB����VN�n��96"	L�3ك1�`JeC���~�v[�|��xV._�s4�r4��{8�A�$�q�]�^�A��(T�8��v���/�;4'