Create async iterator version of _lsdir()

[rlamy]

When dealing with large directories, using _ls()/_lsdir() forces us to wait until all the contents of the directory have been received, which can take a significant amount of time. This PR creates an _iterdir() async iterator which allows the results to be handled as soon as they're received from the network.

[martindurant]

Please add a docstring to _iterdict .

Do you think there should also be a sync version of this, iterdict() ?

[rlamy]

I don't think it makes a lot of sense to wrap _iterdir() with a sync iterator. We'd have to switch between sync and async code on each iteration, which seems inefficient.

[martindurant]

It may be inefficient, but perhaps still desirable for a case where the caller only wants files in the first few pages out of many.

[efiop]

Would it be possible to make _lsdir return an iterator so that the user could stop iterating when he had enough results? Introducing iterdir basically does that but with a brand new method, that seems excessive. My understanding was that _iterdir is more of an internal async method and not really a public API.

[martindurant]

_iterdir is more of an internal async method and not really a public API

As written, maybe (but all async methods are underscored); but actually DVC plans to call it, right? Which suggests that it is useful, and a sync version of it would be too. I don't actually know whether you can simply wrap an async iterator and get a sync iterator as you would expect for a normal method. If yes, do it; if no, then it's probably too much effort.

[efiop]

For now this will be used in another project related to dvc and for that purpose, we could use it as a hidden method and that was my understanding. Maybe @rlamy could clarify.

Maybe it makes sense to have an official iterdir/_iterdir method, I'm not sure, I'm just trying to understand how this is going to fit into the spec and whether it won't be something undesirable considering that ls/_ls already exists (btw, I got confused at called it _lsdir in my comment before, but I really meant _ls, because _lsdir is not official). Just to clarify, this PR is fine as a hidden method that is used by a hidden _lsdir, I only worry about the official spec, as @martindurant suggested adding a sync iterdir, which would make it official in my view.

[rlamy]

@efiop Indeed, for our (dvc-related) current purposes, it's fine to use a private hidden method.

Having iterdir in the spec could make sense, as there is precedent in the stdlib (pathlib.Path.iterdir). However, wrapping an async iterator with a sync iterator isn't trivial. I guess something like the following could work, but it probably ignores a number of edge cases:

def iterdir(self, ...):
    aiter = self.iterdir(...)
    while True:
        try:
            entry = sync(self.loop, aiter.__anext__)
            yield entry
        except StopAsyncIteration:
            break

[martindurant]

OK, going to merge this as an internal implementation detail for now, and we can consider more visible changes at another time.