Sphinx 7.2 breaks Matplotlib's directive with Jax's arrays #11652

jeertmans · 2023-08-28T19:09:38Z

Describe the bug

Hello,

When using the usual make html, I faced the following issue (minimized):

...
TypeError: concatenate requires ndarray or scalar arguments, got <class 'jaxlib.xla_extension.ArrayImpl'> at position 0.
/export/home/eertmans/repositories/DiffeRT/DiffeRT2d/differt2d/scene.py:docstring of differt2d.scene.Scene.square_scene:20: WARNING: Exception occurred in plotting differt2d-scene-3
 from /export/home/eertmans/repositories/DiffeRT/DiffeRT2d/docs/source/reference/differt2d.scene.rst:
Traceback (most recent call last):
  File "/home/eertmans/.cache/pypoetry/virtualenvs/differt2d-4IUpg1Dz-py3.10/lib/python3.10/site-packages/matplotlib/sphinxext/plot_directive.py", line 481, in _run_code
    exec(code, ns)
  File "<string>", line 6, in <module>
  File "/export/home/eertmans/repositories/DiffeRT/DiffeRT2d/differt2d/scene.py", line 602, in plot
    [
  File "/export/home/eertmans/repositories/DiffeRT/DiffeRT2d/differt2d/scene.py", line 603, in <listcomp>
    emitter.plot(ax, *emitters_args, *args, **emitters_kwargs, **kwargs)
  File "/export/home/eertmans/repositories/DiffeRT/DiffeRT2d/differt2d/geometry.py", line 254, in plot
    [self.point[0]],
TypeError: 'ArrayImpl' object is not subscriptable
/export/home/eertmans/repositories/DiffeRT/DiffeRT2d/differt2d/scene.py:docstring of differt2d.scene.Scene.square_scene_with_obstacle:24: WARNING: Exception occurred in plotting differt2d-scene-4
 from /export/home/eertmans/repositories/DiffeRT/DiffeRT2d/docs/source/reference/differt2d.scene.rst:
Traceback (most recent call last):
  File "/home/eertmans/.cache/pypoetry/virtualenvs/differt2d-4IUpg1Dz-py3.10/lib/python3.10/site-packages/matplotlib/sphinxext/plot_directive.py", line 481, in _run_code
    exec(code, ns)
  File "<string>", line 6, in <module>
  File "/export/home/eertmans/repositories/DiffeRT/DiffeRT2d/differt2d/scene.py", line 602, in plot
    [
  File "/export/home/eertmans/repositories/DiffeRT/DiffeRT2d/differt2d/scene.py", line 603, in <listcomp>
    emitter.plot(ax, *emitters_args, *args, **emitters_kwargs, **kwargs)
  File "/export/home/eertmans/repositories/DiffeRT/DiffeRT2d/differt2d/geometry.py", line 254, in plot
    [self.point[0]],
TypeError: 'ArrayImpl' object is not subscriptable
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
                  copying downloadable files... [  8%] ../build/plot_directive/reference/differt2d-geo
copying downloadable files... [100%] ../build/plot_directive/reference/differt2d-scene-4.py
copying static files... done
copying extra files... done
done
writing output... [100%] reference/index
generating indices... genindex py-modindex done
highlighting module code... [100%] differt2d.utils
writing additional pages... search done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 13 warnings.

The HTML pages are in build/html.

As you can see, they say that ArrayImpl is not indexable. But self.point (and others) should not be a ArrayImpl, but rather some concrete array value. When running pytest and other commands, I get the expected output. When building with Sphinx<7.2, it works as expected. I don't know what triggers this ArrayImpl type to be used, and this may be a side effect of using Jax.

This issue is faced when combining Sphinx, jax and matplotlib's Sphinx directive altogether. The error was quite hard to pinpoint, and it is still quite vague to me, even after taking a look at the 7.2 release notes.

I think the issue comes from the Sphinx side, because changing the version requirements to Sphinx<7.2 fixed it all. Using pip freeze, there was no other relevant package that could cause this issue (to my understanding). This is why I post the issue in this repository, but please tell me if I just rather put it elsewhere.

How to Reproduce

This is not a real MWE, but the issue happened on my repo (https://github.com/jeertmans/DiffeRT2d), which you can see the different between this commit (not working) to this commit (working).

Build logs are available here for the failing version:
logs_281.zip

Environment Information

Platform:              linux; (Linux-6.2.0-26-generic-x86_64-with-glibc2.35)
Python version:        3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0])
Python implementation: CPython
Sphinx version:        7.2.0
Docutils version:      0.20.1
Jinja2 version:        3.1.2
Pygments version:      2.16.1

Sphinx extensions

extensions = [
      # Built-in
      "sphinx.ext.autodoc",
      "sphinx.ext.extlinks",
      "sphinx.ext.intersphinx",
      "sphinx.ext.mathjax",
      "sphinx.ext.viewcode",
      # Additional
      "myst_parser",
      "matplotlib.sphinxext.plot_directive",  # Relevant one
      "sphinxext.opengraph",
      "sphinx_copybutton",
      "sphinx_autodoc_typehints",
]

Additional context

No response

The text was updated successfully, but these errors were encountered:

AA-Turner · 2023-08-28T19:36:51Z

Hi Jérome (@jeertmans),

Please would you be able to paste the full log mentioned in the failure message, and ideally also the untruncated -T output?

Additionally, please may you try to reduce the set of extensions as much as possible -- certainly removing myst_parser, and ideally as much as possible. If this is an autodoc failure, ideally just a single .. auto<blah>:: directive in an index.rst file would be very helpful, otherwise knowing which directive caused the failure etc, please.

Thanks,
Adam

jeertmans · 2023-08-29T14:20:32Z

Ok so I managed to have a MWE. The error is not the same, but it also occurs when upgrading to Sphinx 7.2.

pyproject.toml

[tool.poetry]
name = "sphinx-11652-mwe"
version = "0.1.0"
description = ""
authors = ["Jérome Eertmans <jeertmans@icloud.com>"]
readme = "README.md"
packages = [{include = "sphinx_11652_mwe"}]

[tool.poetry.dependencies]
python = "^3.10"
sphinx = "<7.2" # Works
# sphinx = ">=7.2" # Fails
jax = {extras = ["cpu"], version = "^0.4.14"}
matplotlib = "^3.7.2"
chex = "^0.1.82"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

sphinx_11652_mwe/__init__.py

from __future__ import annotations

import jax.numpy as jnp
from chex import dataclass
from jax import Array


@dataclass
class Foo:
    """
    .. plot::

        import matplotlib.pyplot as plt
        from sphinx_11652_mwe import Foo

        n = 5

        f = Foo.from_size(n)

        plt.plot(f.x[n:], f.x[:n])
        plt.show()
    """
    x: Array

    @classmethod
    def from_size(cls, n: int) -> Foo:
        return cls(x=jnp.arange(2 * n))

source/conf.py

project = 'test'
copyright = '2023, NA'
author = 'NA'

extensions = [
    'sphinx.ext.autodoc',
    'matplotlib.sphinxext.plot_directive'
]

templates_path = ['_templates']
exclude_patterns = []

html_theme = 'alabaster'
html_static_path = ['_static']

source/index.rst

Welcome to test's documentation!
================================

.. automodule:: sphinx_11652_mwe
   :members:

Output from failing build:

Full traceback

WARNING: autodoc: failed to import module 'sphinx_11652_mwe'; the following exception was raised:
Traceback (most recent call last):
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/sphinx/ext/autodoc/importer.py", line 64, in import_module
    return importlib.import_module(modname)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/export/home/eertmans/Desktop/sphinx-11652-mwe/sphinx_11652_mwe/__init__.py", line 3, in <module>
    import jax.numpy as jnp
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/jax/__init__.py", line 169, in <module>
    from jax import scipy as scipy
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/jax/scipy/__init__.py", line 24, in <module>
    from jax.scipy import signal as signal
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/jax/scipy/signal.py", line 18, in <module>
    from jax._src.scipy.signal import (
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/jax/_src/scipy/signal.py", line 23, in <module>
    import scipy.signal as osp_signal
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/scipy/signal/__init__.py", line 323, in <module>
    from ._filter_design import *
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/scipy/signal/_filter_design.py", line 16, in <module>
    from scipy import special, optimize, fft as sp_fft
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/scipy/__init__.py", line 211, in __getattr__
    return _importlib.import_module(f'scipy.{name}')
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/scipy/optimize/__init__.py", line 424, in <module>
    from ._direct_py import direct
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/scipy/optimize/_direct_py.py", line 13, in <module>
    from _typeshed import NoneType
ModuleNotFoundError: No module named '_typeshed'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/sphinx/ext/autodoc/importer.py", line 88, in import_object
    module = import_module(modname, warningiserror=warningiserror)
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/sphinx/ext/autodoc/importer.py", line 68, in import_module
    raise ImportError(exc, traceback.format_exc()) from exc
ImportError: (ModuleNotFoundError("No module named '_typeshed'"), 'Traceback (most recent call last):\n  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/sphinx/ext/autodoc/importer.py", line 64, in import_module\n    return importlib.import_module(modname)\n  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module\n    return _bootstrap._gcd_import(name[level:], package, level)\n  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import\n  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load\n  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked\n  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked\n  File "<frozen importlib._bootstrap_external>", line 883, in exec_module\n  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed\n  File "/export/home/eertmans/Desktop/sphinx-11652-mwe/sphinx_11652_mwe/__init__.py", line 3, in <module>\n    import jax.numpy as jnp\n  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/jax/__init__.py", line 169, in <module>\n    from jax import scipy as scipy\n  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/jax/scipy/__init__.py", line 24, in <module>\n    from jax.scipy import signal as signal\n  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/jax/scipy/signal.py", line 18, in <module>\n    from jax._src.scipy.signal import (\n  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/jax/_src/scipy/signal.py", line 23, in <module>\n    import scipy.signal as osp_signal\n  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/scipy/signal/__init__.py", line 323, in <module>\n    from ._filter_design import *\n  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/scipy/signal/_filter_design.py", line 16, in <module>\n    from scipy import special, optimize, fft as sp_fft\n  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/scipy/__init__.py", line 211, in __getattr__\n    return _importlib.import_module(f\'scipy.{name}\')\n  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module\n    return _bootstrap._gcd_import(name[level:], package, level)\n  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/scipy/optimize/__init__.py", line 424, in <module>\n    from ._direct_py import direct\n  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/scipy/optimize/_direct_py.py", line 13, in <module>\n    from _typeshed import NoneType\nModuleNotFoundError: No module named \'_typeshed\'\n')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/sphinx/ext/autodoc/importer.py", line 64, in import_module
    return importlib.import_module(modname)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/export/home/eertmans/Desktop/sphinx-11652-mwe/sphinx_11652_mwe/__init__.py", line 4, in <module>�[91mWARNING: autodoc: failed to import module 'sphinx_11652_mwe'; the following exception was raised:
Traceback (most recent call last):
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/sphinx/ext/autodoc/importer.py", line 64, in import_module
    return importlib.import_module(modname)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/export/home/eertmans/Desktop/sphinx-11652-mwe/sphinx_11652_mwe/__init__.py", line 3, in <module>
    import jax.numpy as jnp
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/jax/__init__.py", line 169, in <module>
    from jax import scipy as scipy
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/jax/scipy/__init__.py", line 24, in <module>
    from jax.scipy import signal as signal
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/jax/scipy/signal.py", line 18, in <module>
    from jax._src.scipy.signal import (
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/jax/_src/scipy/signal.py", line 23, in <module>
    import scipy.signal as osp_signal
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/scipy/signal/__init__.py", line 323, in <module>
    from ._filter_design import *
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/scipy/signal/_filter_design.py", line 16, in <module>
    from scipy import special, optimize, fft as sp_fft
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/scipy/__init__.py", line 211, in __getattr__
    return _importlib.import_module(f'scipy.{name}')
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/scipy/optimize/__init__.py", line 424, in <module>
    from ._direct_py import direct
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/scipy/optimize/_direct_py.py", line 13, in <module>
    from _typeshed import NoneType
ModuleNotFoundError: No module named '_typeshed'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/sphinx/ext/autodoc/importer.py", line 88, in import_object
    module = import_module(modname, warningiserror=warningiserror)
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/sphinx/ext/autodoc/importer.py", line 68, in import_module
    raise ImportError(exc, traceback.format_exc()) from exc
ImportError: (ModuleNotFoundError("No module named '_typeshed'"), 'Traceback (most recent call last):\n  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/sphinx/ext/autodoc/importer.py", line 64, in import_module\n    return importlib.import_module(modname)\n  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module\n    return _bootstrap._gcd_import(name[level:], package, level)\n  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import\n  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load\n  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked\n  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked\n  File "<frozen importlib._bootstrap_external>", line 883, in exec_module\n  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed\n  File "/export/home/eertmans/Desktop/sphinx-11652-mwe/sphinx_11652_mwe/__init__.py", line 3, in <module>\n    import jax.numpy as jnp\n  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/jax/__init__.py", line 169, in <module>\n    from jax import scipy as scipy\n  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/jax/scipy/__init__.py", line 24, in <module>\n    from jax.scipy import signal as signal\n  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/jax/scipy/signal.py", line 18, in <module>\n    from jax._src.scipy.signal import (\n  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/jax/_src/scipy/signal.py", line 23, in <module>\n    import scipy.signal as osp_signal\n  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/scipy/signal/__init__.py", line 323, in <module>\n    from ._filter_design import *\n  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/scipy/signal/_filter_design.py", line 16, in <module>\n    from scipy import special, optimize, fft as sp_fft\n  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/scipy/__init__.py", line 211, in __getattr__\n    return _importlib.import_module(f\'scipy.{name}\')\n  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module\n    return _bootstrap._gcd_import(name[level:], package, level)\n  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/scipy/optimize/__init__.py", line 424, in <module>\n    from ._direct_py import direct\n  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/scipy/optimize/_direct_py.py", line 13, in <module>\n    from _typeshed import NoneType\nModuleNotFoundError: No module named \'_typeshed\'\n')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/sphinx/ext/autodoc/importer.py", line 64, in import_module
    return importlib.import_module(modname)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/export/home/eertmans/Desktop/sphinx-11652-mwe/sphinx_11652_mwe/__init__.py", line 4, in <module>
    from chex import dataclass
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/chex/__init__.py", line 17, in <module>
    from chex._src.asserts import assert_axis_dimension
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/chex/_src/asserts.py", line 27, in <module>
    from chex._src import asserts_internal as _ai
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/chex/_src/asserts_internal.py", line 34, in <module>
    from chex._src import pytypes
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/chex/_src/pytypes.py", line 53, in <module>
    Shape = jax.core.Shape
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/jax/_src/deprecations.py", line 53, in getattr
    raise AttributeError(f"module {module!r} has no attribute {name!r}")
AttributeError: module 'jax' has no attribute 'core'
�[39;49;00m
    from chex import dataclass
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/chex/__init__.py", line 17, in <module>
    from chex._src.asserts import assert_axis_dimension
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/chex/_src/asserts.py", line 27, in <module>
    from chex._src import asserts_internal as _ai
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/chex/_src/asserts_internal.py", line 34, in <module>
    from chex._src import pytypes
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/chex/_src/pytypes.py", line 53, in <module>
    Shape = jax.core.Shape
  File "/home/eertmans/.cache/pypoetry/virtualenvs/sphinx-11652-mwe-IiPyajJx-py3.10/lib/python3.10/site-packages/jax/_src/deprecations.py", line 53, in getattr
    raise AttributeError(f"module {module!r} has no attribute {name!r}")
AttributeError: module 'jax' has no attribute 'core'

I have attached the project files if you prefer.
sphinx-11652-mwe.zip

picnixz · 2023-08-29T14:39:25Z

Ah it's maybe because of some TYPE_CHECKING guard I think. It appears to me that the original exception is the fact that _typeshed cannot be found and this appears to come from scipy.

jeertmans · 2023-08-29T20:53:19Z

That's what I thought, too, but what change in 7.2 could have cause that?

AA-Turner · 2023-08-30T00:07:59Z

@jeertmans -- I'm not able to reproduce the error you're seeing. I get:

/mnt/s/Development/sphinx-issue-11652/sphinx_11652_mwe.py:docstring of sphinx_11652_mwe.Foo:1: WARNING: Exception occurred in plotting index-1
 from /mnt/s/Development/sphinx-issue-11652/index.rst:
Traceback (most recent call last):
  File "/mnt/s/Development/sphinx-issue-11652/.venv/lib/python3.11/site-packages/matplotlib/sphinxext/plot_directive.py", line 481, in _run_code
    exec(code, ns)
  File "<string>", line 8, in <module>
TypeError: 'ArrayImpl' object is not subscriptable

with:

Heading
=======

.. automodule:: sphinx_11652_mwe
   :members:

import os, sys
sys.path.insert(0, os.path.abspath('.'))

extensions = [
    'sphinx.ext.autodoc',
    'matplotlib.sphinxext.plot_directive'
]

exclude_patterns = [
    '.venv',
]

html_theme = 'alabaster'

from __future__ import annotations

import jax.numpy as jnp
from chex import dataclass
from jax import Array

@dataclass
class Foo:
    """
    .. plot::

        import matplotlib.pyplot as plt
        from sphinx_11652_mwe import Foo

        f = Foo.from_size(5)

        plt.plot(f.x[5:], f.x[:5])
        plt.show()
    """
    x: Array

    @classmethod
    def from_size(cls, n: int) -> Foo:
        return cls(x=jnp.arange(2 * n))

Are you able to check without Poetry please? I ran: python3.11 -m venv .venv and then pip install sphinx jax[cpu] chex matplotlib in the activated venv (through WSL). I then ran Sphinx with sphinx-build -M html . _build -aET.

A

jeertmans · 2023-08-30T08:39:38Z

I get the exact same error message as you when I do not use poetry. But what is also weird is that when I pip freeze and compare both environments (Poetry and -m venv), I get the exact same output.

Anyway, having ArrayImpl instead of a concrete array is still an issue...

picnixz · 2023-08-30T09:05:12Z

Not entirely sure, but I'm wondering how matplotlib actually executes the code. Maybe it is an issue with what the local namespace consists of? Also, could you like print the type of f.x and check what happens? (I never used the plot directive) and check the output of jnp.arange(...) in the classmethod itself.

Finally, is the chex.dataclass possibly responsible for this? can you use a real dataclass instead and real functions to create your dataclass (still with this Array annotation)?

jeertmans · 2023-08-30T10:29:47Z

I guess so..

Printing f.x (by raising a ValueError, because Matplotlib captures the output otherwise):

/home/jeertmans/Downloads/sphinx-11652-mwe/sphinx_11652_mwe/__init__.py:docstring of sphinx_11652_mwe.Foo:1: WARNING: Exception occurred in plotting index-1
 from /home/jeertmans/Downloads/sphinx-11652-mwe/source/index.rst:
Traceback (most recent call last):
  File "/home/jeertmans/Downloads/sphinx-11652-mwe/venv/lib/python3.10/site-packages/matplotlib/sphinxext/plot_directive.py", line 481, in _run_code
    exec(code, ns)                                  
  File "<string>", line 8, in <module>
ValueError: <jaxlib.xla_extension.ArrayImpl object at 0x562f77bb2070>

Changing chex.dataclass to dataclasses.dataclass outputs the same.

picnixz · 2023-08-30T10:43:11Z

What about jnp.arange? By the way, maybe it's an issue on jax then? Or matplotlib and its compatibility with jax (can you bisect the (Sphinx) commit responsibile for the issue?)

jeertmans · 2023-08-30T11:37:11Z

This faulty (but quite lengthy) commit is: ad61e41.

jeertmans · 2023-08-30T11:44:03Z

More precisely, the error can be fixed with this checkout:
git checkout 80ec9798612073be699c3fee04a1d6738303dc14 sphinx/ext/autodoc/importer.py

jeertmans · 2023-08-30T11:48:42Z

It's quite unexpected, to me, that the autodoc importer would set TYPE_CHECKING to True. This is what, I suppose, causes the error because jax is imported thinking the library is loaded for type checking, not actual computation purposes. Maybe that also their fault to have a different behavior when using TYPE_CHECKING too... I might create an issue on their repo!

AA-Turner · 2023-08-30T19:07:39Z

It's intentional, as there are often e.g. typing or collections.abc imports in the TYPE_CHECKING block, which are needed for documentation. However, it should be that failure to import with TYPE_CHECKING=True falls back to TYPE_CHECKING=False.

AA-Turner · 2023-08-30T21:29:31Z

Running autodoc on this works:

import numpy as np

def from_size(n: int) -> np.ndarray:
    return np.arange(2 * n)

But fails with Jax:

import jax.numpy as jnp
from jax import Array


def from_size(n: int) -> Array:
    return jnp.arange(2 * n)

I suspect therefore the problem is with Jax -- they'll also break with runtime type checkers (e.g. pyanalyze), I believe.

A

jeertmans · 2023-08-31T05:56:57Z

But here we are also talking about import within docstrings.

I totally get that importing with type checking enabled is importing when running auto doc on a module for example. But why importing the content of the docstring also with type checking set to True?
Usually, code in dosctring are code examples, not type documentation

picnixz · 2023-09-06T06:18:03Z

@AA-Turner by looking at the different arguments in different issues, I think we should not set TYPE_CHECKING=True and rather improve the ModuleAnalyzer, possibly by taking some ideas from MyPy or pylint for instance.

While it does indeed reduce the capability of Sphinx to discover new modules, I think it's more likely that we would hit more and more issues if we were to blindly set TYPE_CHECKING=True. One argument in favor of not doing this is that the PEP (600 something iirc) mentions that the value should be assumed False at runtime but true for static type checkers. In particular, these tools do not need to assume that the code within it successfully executes (e.g., importing from a .pyi file).

IMO, we should make the feature experimental and disabled by default while highlighting its limitations. Alternatively, we could suggest "default" type aliases instead or an enhanced mechanism for type aliases.

For my personal projects, I either use import collections.abc or from collections.abc import ..., so I think we could create presets that assume that collections.abc.* and members from collections.abc are implicitly imported. Same for typing and possibly typing_extensions. That way, people do not need to specify them in the autodoc_type_aliases map.

This would only work for projects that only use "standard" types. For others, I think they should manually specify the type aliases to use. I n addition, I think we could enhance the current autodoc_type_aliases map by allowing to define type aliases per module being documented. That way, I could still have from p import User and from q import User in two different files and specify that the first (resp. second) one maps User to p.User (resp. q.User).

Ideally, we should rather develop our static analysis tool rather than relying on dynamic analysis since we could on techniques from other tools and avoid runtime surprises.

picnixz · 2023-09-19T15:15:16Z

@jeertmans With #11679 being merged, I think you shouldn't have issues, so we may close the issue unless the issue persists.

jeertmans · 2023-09-19T16:16:44Z

Is it already part of any release @picnixz?

AA-Turner · 2023-09-19T16:28:32Z

7.2.6

jeertmans · 2023-09-21T07:53:23Z

Ok, I can confirm that it works under 7.2.6, thanks for the patch!

AA-Turner added type:bug awaiting:response Waiting for a response from the author of this issue labels Aug 28, 2023

This was referenced Aug 31, 2023

Setting typing.TYPE_CHECKING=True breaks Jax's API jax-ml/jax#17385

Closed

BUG: no module named 'numpy._typing._ufunc' numpy/numpy#24604

Closed

This was referenced Sep 7, 2023

Sphinx 7.2.5 breaks with PyO3: "modules may only be initialized once per interpreter process" #11662

Closed

from __future__ import annotations and autodoc_typehints = 'description' breaks type hint resolution #11674

Open

picnixz added extensions:autodoc and removed awaiting:response Waiting for a response from the author of this issue labels Sep 19, 2023

This was referenced Sep 20, 2023

[tracker] Improvement of autodoc_type_aliases #11687

Open

[tracker] Improvement of ModuleAnalyzer #11688

Open

jeertmans closed this as completed Sep 21, 2023

github-actions bot locked as resolved and limited conversation to collaborators Oct 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sphinx 7.2 breaks Matplotlib's directive with Jax's arrays #11652

Sphinx 7.2 breaks Matplotlib's directive with Jax's arrays #11652

jeertmans commented Aug 28, 2023

AA-Turner commented Aug 28, 2023

jeertmans commented Aug 29, 2023

picnixz commented Aug 29, 2023 •

edited

Loading

jeertmans commented Aug 29, 2023

AA-Turner commented Aug 30, 2023 •

edited

Loading

jeertmans commented Aug 30, 2023

picnixz commented Aug 30, 2023 •

edited

Loading

jeertmans commented Aug 30, 2023

picnixz commented Aug 30, 2023 •

edited

Loading

jeertmans commented Aug 30, 2023

jeertmans commented Aug 30, 2023

jeertmans commented Aug 30, 2023

AA-Turner commented Aug 30, 2023

AA-Turner commented Aug 30, 2023

jeertmans commented Aug 31, 2023

picnixz commented Sep 6, 2023

picnixz commented Sep 19, 2023

jeertmans commented Sep 19, 2023

AA-Turner commented Sep 19, 2023

jeertmans commented Sep 21, 2023

Sphinx 7.2 breaks Matplotlib's directive with Jax's arrays #11652

Sphinx 7.2 breaks Matplotlib's directive with Jax's arrays #11652

Comments

jeertmans commented Aug 28, 2023

Describe the bug

How to Reproduce

Environment Information

Sphinx extensions

Additional context

AA-Turner commented Aug 28, 2023

jeertmans commented Aug 29, 2023

picnixz commented Aug 29, 2023 • edited Loading

jeertmans commented Aug 29, 2023

AA-Turner commented Aug 30, 2023 • edited Loading

jeertmans commented Aug 30, 2023

picnixz commented Aug 30, 2023 • edited Loading

jeertmans commented Aug 30, 2023

picnixz commented Aug 30, 2023 • edited Loading

jeertmans commented Aug 30, 2023

jeertmans commented Aug 30, 2023

jeertmans commented Aug 30, 2023

AA-Turner commented Aug 30, 2023

AA-Turner commented Aug 30, 2023

jeertmans commented Aug 31, 2023

picnixz commented Sep 6, 2023

picnixz commented Sep 19, 2023

jeertmans commented Sep 19, 2023

AA-Turner commented Sep 19, 2023

jeertmans commented Sep 21, 2023

picnixz commented Aug 29, 2023 •

edited

Loading

AA-Turner commented Aug 30, 2023 •

edited

Loading

picnixz commented Aug 30, 2023 •

edited

Loading

picnixz commented Aug 30, 2023 •

edited

Loading