Autodoc has trouble with type hinting generics and class name forward references

[adamjstewart]

Describe the bug

My library contains class methods with class name forward references like:

from typing import List

class Foo:
    def foo(self) -> List["Foo"]:
        return [Foo()]

Starting with Python 3.9, we're now able to use type hinting generics in standard collections (PEP 585):

class Foo:
    def foo(self) -> list["Foo"]:
        return [Foo()]

However, autodoc does not like this.

How to Reproduce

With the following conf.py:

nitpicky = True
extensions = ["sphinx.ext.autodoc"]
autodoc_typehints = "description"

and the following index.rst:

.. automodule:: main
   :members:

and the following main.py:

from typing import List


class Foo:
    """Foo"""
    def foo(self) -> "Foo":  # works
        """:returns: stuff"""
        return Foo()


class Bar:
    """Bar"""
    def bar(self) -> List["Bar"]:  # works
        """:returns: stuff"""
        return [Bar()]


class Baz:
    """Baz"""
    def baz(self) -> list["Baz"]:  # fails
        """:returns: stuff"""
        return [Baz()]

we can reproduce the issue:

$ sphinx-build . _build
...
main.py:docstring of main.Baz.baz:1: WARNING: py:class reference target not found: 'Baz'

The Foo and Bar methods work fine, but the Baz method does not. The only way I've managed to get Baz working is via:

from __future__ import annotations

class Baz:
    def baz(self) -> list[Baz]:
        return [Baz()]

Environment Information

Platform:              darwin; (macOS-13.3.1-arm64-arm-64bit)
Python version:        3.10.10 (main, Mar 31 2023, 13:44:49) [Clang 14.0.3 (clang-1403.0.22.14.1)])
Python implementation: CPython
Sphinx version:        6.1.3
Docutils version:      0.19
Jinja2 version:        3.1.2
Pygments version:      2.13.0

Sphinx extensions

sphinx.ext.autodoc

Additional context

Also reproduced with Python 3.9, Sphinx 5.2/5.3.

[picnixz]

While I was trying to investigate this, I see that not using autodoc_typehints = 'description' leads to

docstring of class.Bar.bar:1: WARNING: py:obj reference target not found: typing.List[~class.Bar]

However, the value of autodoc_typehints should not affect how the type annotations are parsed. I will investigate furthermore and try to understand what really happened. By the way, using from __future__ import annotations does not work with me, unless you replace list["Baz"] by list[Baz] (no quotes).

What I found so far:

autodoc_typehints	from __future__ import annotations	annotation	working
signature	no	list["Baz"]	no
signature	no	list[Baz]	no
signature	no	List["Baz"]	yes
signature	no	List[Baz]	no
signature	yes	list["Baz"]	no
signature	yes	list[Baz]	yes
signature	yes	List["Baz"]	yes
signature	yes	List[Baz]	yes
description	no	list["Baz"]	no
description	no	list[Baz]	no
description	no	List["Baz"]	yes
description	no	List[Baz]	no
description	yes	list["Baz"]	no
description	yes	list[Baz]	yes
description	yes	List["Baz"]	yes
description	yes	List[Baz]	yes

I will try investigating. The only difference I have with your build is the version of docutils. By the way, here is the script for generating the results:

Generating script

#!/usr/bin/env bash

mkdir -p gh-11327
cd gh-11327

cat <<-EOF > conf.py
import os

project = 'Foo'
copyright = '2023, picnixz'
author = 'picnixz'

nitpicky = True
extensions = ['sphinx.ext.autodoc']
autodoc_typehints = 'description'

master_doc = 'index'
include_patterns = ['index.rst']
html_theme = 'alabaster'
EOF

cat <<-EOF > index.rst
Project
=======

.. automodule:: main
   :members:
EOF

cat <<-EOF > main.py_t
FUTURE_ANNOTATION
from typing import List

class Baz:
  """docstring"""
  def baz(self) -> RETURN_ANNOTATION:
    """:return: docstring"""
    return [Baz()]
EOF

RESULTS=''

for autodoc_typehint in 'signature' 'description' ; do
  for future_annotation in 'no' 'yes' ; do
    for return_annotation in 'list["Baz"]' 'list[Baz]' 'List["Baz"]' 'List[Baz]' ; do
      if [[ $future_annotation =~ 'no' ]] ; then
        sed -E '/FUTURE_ANNOTATION/d' main.py_t > main.py
      else
        sed -E 's/FUTURE_ANNOTATION/from __future__ import annotations/' main.py_t > main.py
      fi
      sed -i'.bak' -E 's/RETURN_ANNOTATION/'"${return_annotation}"'/' main.py
      export AUTODOC_TYPEHINT=$autodoc_typehint
      python3.10 -m sphinx -M html . _build -q -E -W &>/dev/null
      [[ $? -eq 0 ]] && { status='yes' ; } || { status='no'; }
      printf 'autodoc_typehint=%s, future_annotation=%s, return_annotation=%s, working=%s\n' \
        "${autodoc_typehint}" "${future_annotation}" "${return_annotation}" "$status"
read -r -d '' row <<- EOM
    <tr>
      <td><code>${autodoc_typehint}</code></td>
      <td>${future_annotation}</td>
      <td><code>${return_annotation}</code></td>
      <td>$status</td>
    </tr>
EOM
read -r -d '' RESULTS <<-EOM
    $RESULTS
    $row
EOM
    done
  done
done

cat <<-EOF > table.html
<table>
  <thead>
    <th><code>autodoc_typehints</code></th>
    <th><code>from __future__ import annotations</code></th>
    <th>annotation</th>
    <th>working</th>
  </thead>
  <tbody>
    $RESULTS
  </tbody>
</table>
EOF

Environment

Platform:              linux; (Linux-5.3.18-lp152.106-default-x86_64-with-glibc2.26)
Python version:        3.10.3 (main, Jan 31 2023, 10:47:25) [GCC 7.5.0])
Python implementation: CPython
Sphinx version:        6.1.3
Docutils version:      0.18.1
Jinja2 version:        3.1.2
Pygments version:      2.14.0

[picnixz]

Important update

According to Python official documentation on ForwardRef:

PEP 585 generic types such as list["SomeClass"] will not be implicitly transformed into list[ForwardRef("SomeClass")] and thus will not automatically resolve to list[SomeClass].

This is the reason why list["Baz"] fails to resolve as hoped but this is the normal behaviour. I can try implementing something in that direction, but it might not be the best idea (any thoughts on that @AA-Turner or @tk0miya ?).

I have created a branch which can be used as a playground for anyone who wants to work in that direction (https://github.com/picnixz/sphinx/tree/fix/11327-autodoc-forward-ref) by adding some testing roots.

In the meantime, I would either suggest using from __future__ import annotations together with list[Baz] since this works with both autodoc_typehints configuration value. However, it remains a mystery why autodoc_typehints='description' and autodoc_typehints='signature' give different results.

---

Possibly related to: #10605

[adamjstewart]

By the way, using from __future__ import annotations does not work with me, unless you replace list["Baz"] by list[Baz] (no quotes).

Yes, this is what I meant, apologies for the typo.