Feature request: support TypedDict completions

[pappasam]

Jedi now supports dictionary completions: #951

Unfortunately, for those of us trying to get the most out of TypedDict, the latest master does not support completions for TypedDicts. If we can get the dictionary keys of a TypedDict to autocomplete, I believe users would start using TypedDict a lot more.

# python 3.8
from typing import TypedDict

class MyDict(TypedDict):
    hello: str
    world: int

def hello(x: MyDict):
    z = x["  # it'd be awesome if this autocompleted "hello" and "world"

As a stretch goal, we could also consider completing dictionary keys as the dictionary is being created.

# python 3.8
from typing import TypedDict

class MyDict(TypedDict):
    hello: str
    world: int
h: MyDict = {
    "   # it'd be awesome if this autocompleted "hello" and "world"
}

Happy to start chipping away at this if you can point me in a profitable direction. And great to see that we're getting dictionary key completion!!!

[davidhalter]

I think while the second one would be nice, that's quite a bit of additional code just for completions, so not saying to not do that, it's just probably less realistic.

However the first one looks like we should definitely support it!

If you look into it, one good example of this is probably the Django migration: https://github.com/davidhalter/jedi/pull/1467/files. Or the Enum stuff, see

jedi/jedi/plugins/stdlib.py

Lines 785 to 816 in 47e2cf9

	if metaclass.py__name__() == 'EnumMeta' \
	and metaclass.get_root_context().py__name__() == 'enum':
	filter_ = ParserTreeFilter(parent_context=cls.as_context())
	return [DictFilter({
	name.string_name: EnumInstance(cls, name).name for name in filter_.values()
	})]
	return func(cls, metaclasses)
	return wrapper
	class EnumInstance(LazyValueWrapper):
	def __init__(self, cls, name):
	self.inference_state = cls.inference_state
	self._cls = cls # Corresponds to super().__self__
	self._name = name
	self.tree_node = self._name.tree_name
	@safe_property
	def name(self):
	return ValueName(self, self._name.tree_name)
	def _get_wrapped_value(self):
	value, = self._cls.execute_with_values()
	return value
	def get_filters(self, origin_scope=None):
	yield DictFilter(dict(
	name=compiled.create_simple_object(self.inference_state, self._name.string_name).name,
	value=self._name,
	))
	for f in self._get_wrapped_value().get_filters():
	yield f

.

In Python 3.8, the TypedDict is a _TypedDictMeta, so that could work pretty well. The problem is probably that there's a stub in typeshed that takes preference over the actual TypeDict implementation. To work around this, you can probably just replace TypedDict: object with

class _TypedDictMeta(type): 
class TypedDict(dict, metaclass=_TypedDictMeta): ...

in jedi/third_party/typeshed/stdlib/3/typing.pyi. This is probably also we can contribute to typeshed, because the current object definition is definitely less helpful than the TypeDict that inherits from dict. I'm not sure if that's good enough, because we might also need to add:

for c in self._wrapped_name.infer():
    yield c

to jedi.inference.gradual.typing.TypingModuleName (search for TypeDict), because the TypeDict might be completely ignored for now (for no reasons at all, but there's a TODO).

Also your class will probably look something like this:

class TypeDictInstance(LazyValueWrapper):
    def __init__(self, cls): ...

    def get_key_values(self):
        return ...

Please feel free to ask questions :) The primitives in Jedi are "filters" for finding stuff, "contexts" for being in somewhere (e.g. a FunctionContext), "values" (e.g. a Function) and "names" (e.g. a function name). Your entry point is stdlibs get_metaclass_filters and then you can write pretty similar code as the EnumMeta stuff.

[davidhalter]

I just realized that this is not really how it works. What I described works for attributes, but not for dict keys/values. That's a bit unfortunate. I have to think a bit about this

[pappasam]

Good to know, thanks for the response! I'll remain waiting as you ponder.

[davidhalter]

All right. This took me quite a bit longer than I anticipated. Took me about 5-7 hours to figure out how to do this. Now it feels a bit hacky, but I guess it's still the right way. TypedDict itself is a big hack that breaks pretty much all rules of traditional meta classes.

I have pushed a typeddict branch that you can checkout and work on. There's already a few tests. Some of them are passing. Your main task is to get those working. Everything after that is probably pretty easy. Work has to be done mostly in inference/gradual/typing.py. You have to use self._definition_class.get_filters()[...].all() to get the names and then deal with them.

Also you will probably need to add a few tests to cover inheritance/multiple inheritance, but that can wait a bit.

[pappasam]

@davidhalter awesome, I'm excited to start working on this! Will try dedicate some significant time to it throughout this week.

[pappasam]

Ok, I'm a bit lost here. I've played around with class TypedDict(LazyValueWrapper): and haven't been able to get self._definition_class.get_filters() to return any useful values. Not sure whether I should be editing code in TypedDict, TypedDictClass, both, or neither.

Also, in case this helps at all, outside of Jedi, the only way I've been able to see the "attributes" defined within a typed dict is with typing.get_type_hints. For example:

from typing import TypedDict, get_type_hints

class Hello(TypedDict):
    world: str
    number: int

print(get_type_hints(Hello))

It results in:

$ python script.py
{'world': <class 'str'>, 'number': <class 'int'>}

I haven't found another way to find these "attributes", if you can call them that.

[davidhalter]

Jedi doesn't execute anything, it parses using parso, so there's no reason to use get_type_hints, because that would require us to actually import the file (and struggle if the file had errors).

You can get the annotation dict in Jedi in certain ways. But usually filters are better to actually get what you want. However, you're certainly right that you don't anything meaningful out of the filters I proposed to you. I mentioned get_filters().all(), when it's actually get_filters().values(). But that itself is not enough. It would only be enough if all variables were a ClassVar`.

So the way to get what you want at that point is to use print([f.values(from_instance=True) for f in self._definition_class.get_filters(is_instance=True)]). Now if you want a specific name, you can also use get_filters(is_instance=True).get('foo').

Sorry, I really forgot that annotations in classes are actually instance attributes.

[pappasam]

Ok, I've got basic completion to work with your suggestions: https://github.com/pappasam/jedi/blob/typeddict/jedi/inference/gradual/typing.py#L395

An initial problem: given the mplementation, unlike regular dictionaries, Jedi doesn't know the type of the dictionary values coming from TypedDict:

Any advice for how to preserve the typed information of each value? It's probably something simple I'm missing.

[davidhalter]

For py__simple_getitem__ you basically need to use name = ....get(index) and then just name.infer() :) That's actually the easier part of the two ;-)

[pappasam]

@davidhalter sorry, I tried to follow your above suggestion, but am completely lost about where ....get(index) comes from. Sorry, my progress on preserving type information is stalled 😞

[davidhalter]

@pappasam No problem :)

I was referring to what I wrote above: get_filters(is_instance=True).get('foo'). Instead of 'foo' you can use index. So you end up with something like

def py__simple_getitem__(self, index):
    if isinstance(index, str):
        return ValueSet.from_sets(name.infer() for name in self._definition_class.get_filters(is_instance=True).get('index'))
    return NO_VALUES

Maybe you find edge cases with a few tests. If you add tests for the getitem cases, please add them in test/completion/pep0484_typing.py. I think they make the most sense there. Tests for dict key completion can go into the normal test_* files.

[pappasam]

Unfortunately, despite using the above example and trying some other shots in the dark, I'm still unable to get Jedi to return the type of the dictionary values coming from TypedDict. The previous example now returns nothing on completion. Either this is a bit more complicated than we thought, or I'm missing something obvious.

Note: I changed 'index' to index, since I'm pretty sure that's what you meant.

[davidhalter]

I'll look into it. Thanks for trying! :) It's not easy to debug this kind of stuff. pytest -D sometimes help to get debug output from Jedi, but it's hard to read that, especially if you're not working with it regularly.

[davidhalter]

All right. I merged master into the typeddict branch and added a commit. I had to refactor the class filters for a bit, because they didn't properly work with what we wanted to do. It's pretty much the most complicated part of how finding names in Jedi works, so don't worry if you don't exactly understand what happened :)

[davidhalter]

Merged #1495

[milad2golnia]

Hi @davidhalter
I still don't get type suggestion for TypeDict when using Jedi.
Any progress here?

Here is my code example:

from typing import TypedDict, Union


class TLoginSMSPatternValues(TypedDict):
    code: int
    duration: int


def send_sms(
    apikey, patternCode, pattern_values: TLoginSMSPatternValues, originator, recipient
):
 pass


send_sms('', '', { # No Suggestion here.

[davidhalter]

This is not done yet and part of #1740