Timeout Error and low speed

[liburningstar]

Long story short

Sorry for my poor english firstly. i am trying make http request with aiohttp/asyncio to get best speed.
i search and copy code from https://pawelmhm.github.io/asyncio/python/aiohttp/2016/04/22/asyncio-aiohttp.html,but something get wrong

Expected behaviour

it should process steable

Actual behaviour

But,in fact,sometime it works,sometime it seems "blocked"(when "blocked",cpu and network usage is 0%,only memory usage is between 100MB to 800MB),and finaly throw a timeout Error：raise asyncio.TimeoutError from None concurrent.futures._base.TimeoutError,here is code and output:

import time
import random
import asyncio
from aiohttp import ClientSession

async def fetch(url, session):
    async with session.get(url) as response:
        return await response.read()


async def bound_fetch(sem, url, session):
    async with sem:
        await fetch(url, session)
 
async def run(r):
    url = "http://www.zhihu.com/"
    tasks = []
    sem = asyncio.Semaphore(1000)
    async with ClientSession() as session:
        for i in range(r):
            task = asyncio.ensure_future(bound_fetch(sem, url, session))
            tasks.append(task)
        responses = asyncio.gather(*tasks)
        await responses

for count in range(99):
    number = 5000
    loop = asyncio.get_event_loop()
    start = time.time()
    future = asyncio.ensure_future(run(number))
    loop.run_until_complete(future)
    print(time.time()-start)

17.91987895965576
16.38098454475403
15.312330484390259
18.451438188552856
22.169893980026245
15.868573188781738
17.176181077957153
26.354613542556763
17.736180067062378
16.149995803833008
18.01749062538147
17.74019193649292
24.42871117591858
Traceback (most recent call last):
  File "E:\asy.py", line 45, in <module>
    loop.run_until_complete(future)
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\asyncio\base_events.py", line 584, in run_until_complete
    return future.result()
  File "E:\asy.py", line 38, in run
    await responses
  File "E:\asy.py", line 22, in bound_fetch
    await fetch(url, session)
  File "E:\asy.py", line 8, in fetch
    async with session.get(url) as response:
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\site-packages\aiohttp\client.py", line 1005, in __aenter__
    self._resp = await self._coro
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\site-packages\aiohttp\client.py", line 575, in _request
    break
  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\site-packages\aiohttp\helpers.py", line 585, in __exit__
    raise asyncio.TimeoutError from None
concurrent.futures._base.TimeoutError

i don not know how to fix it,and more,250 requests per seconds is normal or is too low，how to improve the to be faster.

Steps to reproduce

Run the code for times

Your environment

win10,python3.7.2,aiohttp(client)3.5.4

[aio-libs-bot]

GitMate.io thinks the contributor most likely able to help you is @asvetlov.

Possibly related issues are #3242 (Error), #432 (asyncio.wait_for throwing a timeout error on aiohttp.request?), #429 (Websocket error), #816 (Logging error), and #1203 (CookieJar error).

[liburningstar]

@pawelmhm

[leimao]

It seems that this has been a problem for many years. Rasing the TimeoutError will cause the entire program to crash and this exception cannot be caught.

[webknjaz]

It seems that this has been a problem for many years. Rasing the TimeoutError will cause the entire program to crash and this exception cannot be caught.

Why is that? It should be possible to catch a TimeoutError by placing try/except around an async with block.

[Dreamsorcerer]

I think this is just an issue with the server. If I log the requests that are still open (or trying to connect) I get a random selection of them. For example, from the 5000 requests, these requests were still open when the program idled:
{8, 14, 16, 17, 18, 22, 23, 27, 40, 89, 94, 99, 4327, 4421, 455, 806, 985, 1326, 3580}
If I wait long enough, these do still complete, by the time I got the TimeoutError/ClientConnectorError, it was down to:
{17, 40}

If I change the URL to point to a local aiohttp server, created with:

from aiohttp import web

async def hello(request):
    return web.Response(text="hello")

app = web.Application()
app.router.add_route("GET", "/", hello)
web.run_app(app)

Then there are no errors at all.

So, I don't see anything to fix in aiohttp.