FEAT: add stat support for chunked transfer-encoding

[quosa]

Hi,
I replaced the HttpBrowser in Locust.client with our client SDK that uses httplib directly. The switch itself (monkey-patching the log_request decorator) was fairly straightforward. However, I came across a problem in stats I could not easily solve: my client does most requests with accept-encoding: gzip and the HTTP 1.1 default keep-alive connections which results in Transfer-Encoding: chunked and Content-Encoding: gzip responses. Since chunked responses do not have the Content-Length header the stats module (stats.py:on_request_success) defaults to 0 content length.

I couldn't figure out how to elegantly patch the on_request_success function, but would you have a proposal?

[heyman]

Hi!

Sorry for not replying to this sooner.

In the current master branch of locust (and soon in the 0.6 release), the old HTTP client has been replaced with a small wrapper around python-requests (http://python-requests.org). The events.request_success event has also been modified to not take a response object as argument, but rather the content-length as an integer. So if you're rolling your own client it should now be easy to trigger the request_success event and sending in the length of the response body instead of the content-length header.

We could do this change in the locust code (report the length of response.content instead of the content-length header). However, this would trigger a full body download even if the user has made the request using prefetch=False (http://docs.python-requests.org/en/latest/user/advanced/#body-content-workflow), and I'm not sure if we really want to disable that feature.

One option could be to use the length of response.content unless prefetch is set to False, in which case we try to use the content-length header, and last we fallback to 0. However, this would result in us having two different ways of getting the content size which doesn't seem optimal. Despite this, it might be the solution that would work best in most situations. Any input on this @Jahaja @cgbystrom ?