asyncio.wait_for throwing a timeout error on aiohttp.request?

[digitaldavenyc]

I have a request that has a timeout set using the asyncio.wait_for but I'm still receiving a timeout error, it is my understanding that this shouldn't happen. The first address in the urls list is the unresponsive url and I need to account for this in production.

urls = ["52.1.14.108", "teakorlando.com", "mfaoil.com", "entrepreneurmag.co.za", "spc.int", "aaaaattttumblr.com", "canstruction-orlando.org", "blogspot.com", "spacepestremoval.com", "justinecirullo.com", "avangate.com", "nomi.com", "herokuapp.com", "mappinghacks.com", "pharmasave.com", "daq.net", ]

def generate(data):
    coroutines = []
    for link in data:
        coroutines.append(asyncio.Task(request(link)))

    results = yield from asyncio.gather(*coroutines)

    for result in results:
            body = yield from result[0].read()
            # Does something here....

def request(url):
    try:
        r = aiohttp.request(method='GET', url='http://' + url)

        result = yield from asyncio.wait_for(r, timeout=10)

        return result, url
    except (TimeoutError,
            aiohttp.errors.ClientConnectionError,
            ConnectionError,
            aiohttp.errors.ClientResponseError,
            aiohttp.errors.ServerDisconnectedError,
            ConnectionRefusedError,
            aiohttp.errors.ContentEncodingError,
            socket.gaierror) as e:
        return e.__class__.__name__, url

loop = asyncio.get_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(generate(urls))
loop.close()

Event though I have the asyncio.wait_for wrapped in an exception, a timeout error fires anyway.

---------------------------------------------------------------------------
TimeoutError                              Traceback (most recent call last)
<ipython-input-69-d926e3e202b5> in <module>()
      2 loop = asyncio.get_event_loop()
      3 asyncio.set_event_loop(loop)
----> 4 loop.run_until_complete(generate(urls))
      5 loop.close()
      6 print("Total time: {}".format(time.time() - start))

<ipython-input-55-92ed64271ef2> in generate(data)
      5         coroutines.append(asyncio.Task(request(link)))
      6 
----> 7     results = yield from asyncio.gather(*coroutines)
      8 
      9     for result in results:

<ipython-input-68-3e4c4e5c5946> in request(url)
      4 
      5         start = time.time()
----> 6         result = yield from asyncio.wait_for(r, timeout=10)
      7         duration = time.time() - start
      8 

[popravich]

Hi, @digitaldavenyc
I believe you're using builtin TimeoutError not asyncio.TimeoutError.
Also asyncio.gather drops out with first exception raised, so you may not get all tasks finished.
You should try using gather with return_exceptions=True

[asvetlov]

I believe it's not a bug

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.