GH-103699: Add `__orig_bases__` to various typing classes #103698

adriangb · 2023-04-22T22:31:09Z

As discussed at the typing summit, non generic TypedDict's completely loose all inheritance information. Funny enough, generic TypedDict's actually preserve it via __orig_bases__ because Generic does that. So this just brings non-generic TypedDicts to be the same.

Issue: Add __orig_bases__ to non-generic TypedDict #103699

AlexWaygood

Thank you! Will wait for a second thumbs-up from a typing maintainer before merging, but LGTM

Lib/test/test_typing.py

JelleZijlstra · 2023-04-23T00:00:03Z

Lib/test/test_typing.py

@@ -7126,6 +7126,9 @@ class TD(TypedDict):
        self.assertIs(type(a), dict)
        self.assertEqual(a, {'a': 1})

+    def test_orig_bases(self):
+        self.assertEqual(ChildTotalMovie.__orig_bases__, (ParentNontotalMovie,))


Could you add some more tests? Suggestions: a TypedDict with no bases, one with multiple bases, a generic TypedDict.

__orig_bases__ on generic TypedDicts are already tested quite extensively, e.g.

cpython/Lib/test/test_typing.py

Line 7017 in 4037df1

self.assertEqual(A.__orig_bases__, (TypedDict, Generic[T]))

Tests for TypedDicts with no bases and multiple bases sound worth adding, though

I added quite a few :)

Should I remove the checks on generic TypedDicts then? Or move them all here?

Should I remove the checks on generic TypedDicts then? Or move them all here?

I think it's probably fine as is, actually. A little bit of duplication in tests isn't a massive issue, and it makes sense to both:

1). Test __orig_bases__ in the test method for testing generic TypedDicts
2). Test generic TypedDicts in the test method for testing __orig_bases__

JelleZijlstra

Looks good, thanks!

I might want documentation for this attribute, but we have another PR open to add typing.get_original_bases, so we can cover TypedDicts there too.

AlexWaygood · 2023-04-23T00:32:53Z

Hmm, one edge case that isn't currently covered in the tests is call-based TypedDicts. I'm not sure if this behaviour is correct:

>>> from typing import TypedDict
>>> class Foo(TypedDict): ...
...
>>> Foo2 = TypedDict("Foo2", {})
>>> Foo.__orig_bases__
(<function TypedDict at 0x000002796B986410>,)
>>> Foo2.__orig_bases__
()

Here's the current behaviour with call-based NamedTuples, which is the closest stdlib analogue:

>>> from typing import NamedTuple
>>> class Bar(NamedTuple): ...
...
>>> Bar2 = NamedTuple("Bar2", {})
>>> Bar.__orig_bases__
(<function NamedTuple at 0x000002796B986200>,)
>>> Bar2.__orig_bases__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: type object 'Bar2' has no attribute '__orig_bases__'

AlexWaygood · 2023-04-23T00:40:50Z

We could match the behaviour of NamedTuple here (adding __orig_bases__ only for class-based TypedDicts -- which I think is probably the correct thing to do?) by applying this diff to your PR branch:

diff --git a/Lib/typing.py b/Lib/typing.py
index 8408edc49a..660e89cb30 100644
--- a/Lib/typing.py
+++ b/Lib/typing.py
@@ -3107,7 +3107,9 @@ class body be required.
         # Setting correct module is necessary to make typed dict classes pickleable.
         ns['__module__'] = module

-    return _TypedDictMeta(typename, (), ns, total=total)
+    td = _TypedDictMeta(typename, (), ns, total=total)
+    del td.__orig_bases__
+    return td

 _TypedDict = type.__new__(_TypedDictMeta, 'TypedDict', (), {})
 TypedDict.__mro_entries__ = lambda bases: (_TypedDict,)

adriangb · 2023-04-23T00:43:56Z

I'm happy to do that, but I think the new behavior is better. Even better would be if the call approaches matched the class-based approaches, but certainly the attribute always existing is better than it only existing sometimes, right?

JelleZijlstra · 2023-04-23T04:14:18Z

I think the conclusion from the in-person discussion was that we should make it return (TypedDict,) for call-based TypedDicts, right?

AlexWaygood · 2023-04-23T04:26:22Z

I think the conclusion from the in-person discussion was that we should make it return (TypedDict,) for call-based TypedDicts, right?

Yes, that sounds good. But we should fix it for call-based NamedTuples as well, so that it's consistent between call-based NamedTuples and call-based TypedDicts. (That could be done in a separate PR, but I also wouldn't mind doing it in the same PR.)

adriangb · 2023-04-23T04:49:49Z

Yup that’s what I recall as well.

Alex you did bring up the point about inline TypedDicts (which is being discussed in the mailing list and it seems like has a good chance of moving to a PEP), and how it might not make sense to have TypedDict in the bases for inline TypedDicts. Philosophically, I agree with that. But in practice, for the things that are going to be digging into the bases of a TypedDict, it is not a problem to have or not have TypedDict itself in the bases. So I think we should move forward with this keeping it as is to minimize changes and deal with inline TypedDicts if/when they show up since even if they behave different it wouldn’t really be a huge deal (and we can probably just get them to behave the same if we want to).

AlexWaygood · 2023-04-23T04:51:38Z

Alex you did bring up the point about inline TypedDicts

I think @hauntsaninja brought up the point about inline TypedDicts, but I agree with your conclusion here!

JelleZijlstra · 2023-04-23T04:52:14Z

I don't know exactly how inline TypedDicts will work (for one, they won't have a name); that'll be up to Nikita to specify in the PEP. But yeah, we can discuss that when the time comes to actually implement it.

AlexWaygood · 2023-04-23T04:57:53Z

Okay, applying this patch to your PR branch appears to make call-based TypedDicts and NamedTuples consistent with class-based TypedDicts and NamedTuples when it comes to __orig_bases__:

diff --git a/Lib/typing.py b/Lib/typing.py
index 8408edc49a..354bc80eb3 100644
--- a/Lib/typing.py
+++ b/Lib/typing.py
@@ -2962,7 +2962,9 @@ class Employee(NamedTuple):
     elif kwargs:
         raise TypeError("Either list of fields or keywords"
                         " can be provided to NamedTuple, not both")
-    return _make_nmtuple(typename, fields, module=_caller())
+    nt = _make_nmtuple(typename, fields, module=_caller())
+    nt.__orig_bases__ = (NamedTuple,)
+    return nt

 _NamedTuple = type.__new__(NamedTupleMeta, 'NamedTuple', (), {})

@@ -3107,7 +3109,9 @@ class body be required.
         # Setting correct module is necessary to make typed dict classes pickleable.
         ns['__module__'] = module

-    return _TypedDictMeta(typename, (), ns, total=total)
+    td = _TypedDictMeta(typename, (), ns, total=total)
+    td.__orig_bases__ = (TypedDict,)
+    return td

 _TypedDict = type.__new__(_TypedDictMeta, 'TypedDict', (), {})
 TypedDict.__mro_entries__ = lambda bases: (_TypedDict,)

(Needs tests as well)

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

AlexWaygood · 2023-04-23T05:03:03Z

Okay, applying this patch to your PR branch appears to make call-based TypedDicts and NamedTuples consistent with class-based TypedDicts and NamedTuples when it comes to __orig_bases__:

This, uh, would introduce a discrepancy between call-based collections.namedtuple and call-based typing.NamedTuple:

>>> import collections as c, typing as t
>>> class Foo(c.namedtuple('Foo', {})): ...
...
>>> class Bar(t.NamedTuple('Bar', {})): ...
...
>>> Foo.__orig_bases__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: type object 'Foo' has no attribute '__orig_bases__'
>>> Bar.__orig_bases__
(<function NamedTuple at 0x000001426A124730>,)

But I think we can live with that...

Misc/NEWS.d/next/Library/2023-04-22-22-37-39.gh-issue-103699.NizCjc.rst

…izCjc.rst Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

adriangb · 2023-04-23T05:21:10Z

I added tests for call-based TypedDict and various NamedTuple scenarios in 631d428, your patch works great, thank you. I previously pushed 7224ddd but then remembered that NamedTuple inheritance is not really a thing.

But I think we can live with that...

I agree, especially given that (1) it can probably be fixed there as well and (2) I think we all agree that the new behavior is better.

AlexWaygood

LGTM.

It surprises me that Bar.__orig_bases__ in this case is (NamedTuple,), since the "original bases" of Bar are not (NamedTuple,) (they're (Foo,)):

>>> from typing import NamedTuple
>>> class Foo(NamedTuple): ...
...
>>> class Bar(Foo): ...
...
>>> Bar.__orig_bases__
(<function NamedTuple at 0x0000016E1F4ACEA0>,)

However, this behaviour is perfectly consistent with how inheriting from Generic[T] works:

>>> from typing import Generic, TypeVar
>>> T = TypeVar("T")
>>> class Baz(Generic[T]): ...
...
>>> class Eggs(Baz): ...
...
>>> Eggs.__orig_bases__
(typing.Generic[~T],)

So I think it's fine.

Misc/NEWS.d/next/Library/2023-04-22-22-37-39.gh-issue-103699.NizCjc.rst

…izCjc.rst Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>

adriangb requested review from gvanrossum, Fidget-Spinner, JelleZijlstra and AlexWaygood as code owners April 22, 2023 22:31

bedevere-bot added the awaiting review label Apr 22, 2023

adriangb changed the title ~~Add to non-generic TypedDict~~ Add __orig_bases__ to non-generic TypedDict Apr 22, 2023

Add __orig_bases__ to non-generic TypedDict

7a81c98

adriangb force-pushed the add-orig-bases-typeddict branch from dba5772 to 7a81c98 Compare April 22, 2023 22:31

adriangb changed the title ~~Add __orig_bases__ to non-generic TypedDict~~ GH-103699: Add __orig_bases__ to non-generic TypedDict Apr 22, 2023

bedevere-bot mentioned this pull request Apr 22, 2023

Add __orig_bases__ to non-generic TypedDict #103699

Closed

blurb-it bot and others added 2 commits April 22, 2023 22:37

📜🤖 Added by blurb_it.

270cadf

Use double backticks

29b643d

AlexWaygood approved these changes Apr 22, 2023

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting review labels Apr 22, 2023

AlexWaygood added type-feature A feature request or enhancement topic-typing 3.12 bugs and security fixes stdlib Python modules in the Lib dir labels Apr 22, 2023

AlexWaygood reviewed Apr 22, 2023

View reviewed changes

Lib/test/test_typing.py Outdated Show resolved Hide resolved

gvanrossum approved these changes Apr 22, 2023

View reviewed changes

remove whitespace

4037df1

AlexWaygood added the 🤖 automerge label Apr 22, 2023

JelleZijlstra reviewed Apr 23, 2023

View reviewed changes

AlexWaygood removed the 🤖 automerge label Apr 23, 2023

adriangb added 2 commits April 22, 2023 18:17

add more test cases

19569f9

one more test

b777373

JelleZijlstra approved these changes Apr 23, 2023

View reviewed changes

AlexWaygood added the 🤖 automerge label Apr 23, 2023

AlexWaygood removed the 🤖 automerge label Apr 23, 2023

Make call-based NamedTuples behave like call-based TypedDicts

4c9ee8c

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

adriangb added 2 commits April 22, 2023 23:07

Add (failing) tests

7224ddd

More tests

631d428

AlexWaygood reviewed Apr 23, 2023

View reviewed changes

Misc/NEWS.d/next/Library/2023-04-22-22-37-39.gh-issue-103699.NizCjc.rst Outdated Show resolved Hide resolved

Update Misc/NEWS.d/next/Library/2023-04-22-22-37-39.gh-issue-103699.N…

58701ac

…izCjc.rst Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

AlexWaygood changed the title ~~GH-103699: Add __orig_bases__ to non-generic TypedDict~~ GH-103699: Add __orig_bases__ to various typing classes Apr 23, 2023

AlexWaygood approved these changes Apr 23, 2023

View reviewed changes

AlexWaygood requested a review from JelleZijlstra April 23, 2023 05:48

AlexWaygood added skip news and removed skip news labels Apr 23, 2023

JelleZijlstra reviewed Apr 23, 2023

View reviewed changes

Misc/NEWS.d/next/Library/2023-04-22-22-37-39.gh-issue-103699.NizCjc.rst Outdated Show resolved Hide resolved

Update Misc/NEWS.d/next/Library/2023-04-22-22-37-39.gh-issue-103699.N…

3fe75eb

…izCjc.rst Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>

JelleZijlstra approved these changes Apr 23, 2023

View reviewed changes

AlexWaygood merged commit 0056701 into python:main Apr 23, 2023

bedevere-bot removed the awaiting merge label Apr 23, 2023

adriangb mentioned this pull request Apr 23, 2023

Add __orig_bases__ to TypedDict python/typing_extensions#150

Merged

AlexWaygood added a commit to Gobot1234/cpython that referenced this pull request Apr 23, 2023

More tests and docs following python#103698

cb21a65

adriangb mentioned this pull request Jul 2, 2023

Fix validate_call schema generation and doc error for TypedDict pydantic/pydantic#6370

Merged

adriangb mentioned this pull request Jul 13, 2023

Contradictory typing.TypedDict >=3.11 or >=3.12 support pydantic/pydantic#6645

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-103699: Add `__orig_bases__` to various typing classes #103698

GH-103699: Add `__orig_bases__` to various typing classes #103698

adriangb commented Apr 22, 2023 •

edited by bedevere-bot

Loading

AlexWaygood left a comment

JelleZijlstra Apr 23, 2023

AlexWaygood Apr 23, 2023 •

edited

Loading

adriangb Apr 23, 2023

adriangb Apr 23, 2023

AlexWaygood Apr 23, 2023

JelleZijlstra left a comment •

edited

Loading

AlexWaygood commented Apr 23, 2023 •

edited

Loading

AlexWaygood commented Apr 23, 2023 •

edited

Loading

adriangb commented Apr 23, 2023

JelleZijlstra commented Apr 23, 2023

AlexWaygood commented Apr 23, 2023 •

edited

Loading

adriangb commented Apr 23, 2023

AlexWaygood commented Apr 23, 2023

JelleZijlstra commented Apr 23, 2023

AlexWaygood commented Apr 23, 2023 •

edited

Loading

AlexWaygood commented Apr 23, 2023

adriangb commented Apr 23, 2023 •

edited

Loading

AlexWaygood left a comment •

edited

Loading

GH-103699: Add __orig_bases__ to various typing classes #103698

GH-103699: Add __orig_bases__ to various typing classes #103698

Conversation

adriangb commented Apr 22, 2023 • edited by bedevere-bot Loading

AlexWaygood left a comment

Choose a reason for hiding this comment

JelleZijlstra Apr 23, 2023

Choose a reason for hiding this comment

AlexWaygood Apr 23, 2023 • edited Loading

Choose a reason for hiding this comment

adriangb Apr 23, 2023

Choose a reason for hiding this comment

adriangb Apr 23, 2023

Choose a reason for hiding this comment

AlexWaygood Apr 23, 2023

Choose a reason for hiding this comment

JelleZijlstra left a comment • edited Loading

Choose a reason for hiding this comment

AlexWaygood commented Apr 23, 2023 • edited Loading

AlexWaygood commented Apr 23, 2023 • edited Loading

adriangb commented Apr 23, 2023

JelleZijlstra commented Apr 23, 2023

AlexWaygood commented Apr 23, 2023 • edited Loading

adriangb commented Apr 23, 2023

AlexWaygood commented Apr 23, 2023

JelleZijlstra commented Apr 23, 2023

AlexWaygood commented Apr 23, 2023 • edited Loading

AlexWaygood commented Apr 23, 2023

adriangb commented Apr 23, 2023 • edited Loading

AlexWaygood left a comment • edited Loading

Choose a reason for hiding this comment

GH-103699: Add `__orig_bases__` to various typing classes #103698

GH-103699: Add `__orig_bases__` to various typing classes #103698

adriangb commented Apr 22, 2023 •

edited by bedevere-bot

Loading

AlexWaygood Apr 23, 2023 •

edited

Loading

JelleZijlstra left a comment •

edited

Loading

AlexWaygood commented Apr 23, 2023 •

edited

Loading

AlexWaygood commented Apr 23, 2023 •

edited

Loading

AlexWaygood commented Apr 23, 2023 •

edited

Loading

AlexWaygood commented Apr 23, 2023 •

edited

Loading

adriangb commented Apr 23, 2023 •

edited

Loading

AlexWaygood left a comment •

edited

Loading