[RESEND] Fix HttpPayloadParser dealing with chunked response (#4630) #4846

rhdxmr · 2020-07-01T05:44:17Z

I rebased my commits onto origin/3.7 and resend them.
I cherry-picked @JustAnotherArchivist 's commit because it belongs to the master branch but not to the origin/3.7 branch.

The previous PR which was based on origin/master: #4801

HttpPayloadParser waits for trailers indefinitely even if there are no trailers
at the response. This happens when only the last CRLF or the last LF are sent
via separate TCP segment.

When the connection is keep alive and if this bug occurs then users experience
response timeout. But this problem is not exposed when keep alive is disabled
because .feed_eof is called. (Instead of TimeoutError, ClientPayloadError is raised if keep alive is disabled)

What do these changes do?

Fix a bug that HttpPayloadParser waits for data indefintely that will never come.
The bug makes caller of 'await response.read()' awaits forever or for timeout.
There are a few conditions which need to be met in order to reproduce this bug.

keep alive enabled connection
response body size is large so that data is splitted into many TCP segments
Only the last CRLF or just LF is separated alone and is sent via the last TCP segment.
asyncio tcp protocol handler for the http response receives TCP payload of the last TCP segment. In order for this the last TCP payload should not be merged with the previous data at lower level such as TCP Segmentation Offload.
The response should not be the last response that the connection allows. The keep alive connection processes the limited number of requests, so if the response is the last thing, the connection is going to be closed by server so that .feed_eof method is called explicitly.

Are there changes in behavior for the user?

Improvement experience of users who are suffering from mysterous response timeout. There does not exist any log of which response time is slow in sever access log, but client writes log about response timeout.

Related issue number

#4630

I had a problem with intermittent response timeout so I conducted debug a lot.
And I managed to locate what caused the problem and fixed it on my own.
And then I was going to add test code that prove my modification really fix that thing. So I learned how to run test code in this project. And I ran test codes for the first time.
Surprisingly the issue 4630 just popped up with XPASS labeled. At this point I realized that the issue had been already reported 3 months ago. Sigh.. If the issue was fixed at that time, I would not spend my time for debugging this problem!

I did not mean to provide the fix on behalf of the person who reported the issue before me. I just did not know there was the issue before I started fixing the bug.

Checklist

I think the code is well written
Unit tests for the changes exist
[ ] Documentation reflects the changes => don't need that because this PR just fixes bug of HttpPayloadParser which has nothing to do with normal users.
If you provide code modification, please add yourself to CONTRIBUTORS.txt
- The format is <Name> <Surname>.
- Please keep alphabetical order, the file is sorted by names.
Add a new news fragment into the CHANGES folder
- name it <issue_id>.<type> for example (588.bugfix)
- if you don't have an issue_id change it to the pr id after creating the pr
- ensure type is one of the following:
  - .feature: Signifying a new feature.
  - .bugfix: Signifying a bug fix.
  - .doc: Signifying a documentation improvement.
  - .removal: Signifying a deprecation or removal of public API.
  - .misc: A ticket has been closed, but it is not of interest to users.
- Make sure to use full sentences with correct case and punctuation, for example: "Fix issue with non-ascii contents in doctest text files."

If the last CRLF or only the LF are received via separate TCP segment, HTTPPayloadParser misjudges that trailers should come after 0\r\n in the chunked response body. In this case, HttpPayloadParser starts waiting for trailers, but the only remaining data to be received is CRLF. Thus, HttpPayloadParser waits trailers indefinitely and this incurs TimeoutError in user code. However, if the connection is keep alive disabled, this problem is not reproduced because the server shutdown the connection explicitly after sending all data. If the connection is closed .feed_eof is called and it helps HttpPayloadParser finish its waiting.

@JustAnotherArchivist

PR aio-libs#4651 by @JustAnotherArchivist This change adds tests that demonstrate the failures described in aio-libs#4630. They are marked as xfail so that they don't affect the CI status. Once the issue is fixed, they'll reported as XPASS and pytest will fail, which would be a signal that it's time to remove the xfail markers keeping the contents of the tests to prevent regressions. (ref: https://pganssle-talks.github.io/xfail-lightning) Co-Authored-By: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>

codecov-commenter · 2020-07-01T05:49:13Z

Codecov Report

Merging #4846 into 3.7 will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##              3.7    #4846   +/-   ##
=======================================
  Coverage   97.89%   97.90%           
=======================================
  Files          44       44           
  Lines        8951     8957    +6     
  Branches     1407     1409    +2     
=======================================
+ Hits         8763     8769    +6     
  Misses         80       80           
  Partials      108      108

Impacted Files	Coverage Δ
aiohttp/http_parser.py	`97.28% <100.00%> (+0.03%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 550801f...3afa789. Read the comment docs.

rhdxmr · 2020-07-01T06:01:09Z

The errors of tests are related to aio-libs/yarl#410

rhdxmr · 2020-07-05T05:11:03Z

Hi @socketpair
Can you review my PR?

socketpair · 2020-07-05T07:37:36Z

Unfortunately tests fail on Python 3.8. Anyway I'm sure this was not triggerd by the PR. So, I would merge @asvetlov.

socketpair · 2020-07-15T08:46:21Z

@asvetlov ?

asvetlov

LGTM, thank you very much!

* Parse the last CRLF of chunked response correctly (#4630) If the last CRLF or only the LF are received via separate TCP segment, HTTPPayloadParser misjudges that trailers should come after 0\r\n in the chunked response body. In this case, HttpPayloadParser starts waiting for trailers, but the only remaining data to be received is CRLF. Thus, HttpPayloadParser waits trailers indefinitely and this incurs TimeoutError in user code. However, if the connection is keep alive disabled, this problem is not reproduced because the server shutdown the connection explicitly after sending all data. If the connection is closed .feed_eof is called and it helps HttpPayloadParser finish its waiting. Co-authored-by: JustAnotherArchivist <JustAnotherArchivist@users.noreply.github.com> Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua> Co-authored-by: Andrew Svetlov <andrew.svetlov@gmail.com>

rhdxmr and others added 4 commits July 1, 2020 13:28

Remove xfail decorators (aio-libs#4630)

212a64c

Add tests for covering all modified code (aio-libs#4630)

3afa789

rhdxmr requested review from asvetlov and webknjaz as code owners July 1, 2020 05:44

psf-chronographer bot added the bot:chronographer:provided There is a change note present in this PR label Jul 1, 2020

rhdxmr mentioned this pull request Jul 1, 2020

Fix HttpPayloadParser dealing with chunked response (#4630) #4801

Closed

4 tasks

Merge branch '3.7' into 3.7

cce5f2d

asvetlov approved these changes Oct 16, 2020

View reviewed changes

asvetlov merged commit 3454184 into aio-libs:3.7 Oct 16, 2020

asvetlov mentioned this pull request Oct 16, 2020

HttpPayloadParser fails with "Not enough data for satisfy transfer length header" on chunked transfer encoding if the data is split exactly where trailers could occur #4630

Closed

asvetlov mentioned this pull request Oct 16, 2020

Fix last chunk #5058

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RESEND] Fix HttpPayloadParser dealing with chunked response (#4630) #4846

[RESEND] Fix HttpPayloadParser dealing with chunked response (#4630) #4846

rhdxmr commented Jul 1, 2020 •

edited

Loading

codecov-commenter commented Jul 1, 2020 •

edited

Loading

rhdxmr commented Jul 1, 2020 •

edited

Loading

rhdxmr commented Jul 5, 2020

socketpair commented Jul 5, 2020

socketpair commented Jul 15, 2020

asvetlov left a comment

[RESEND] Fix HttpPayloadParser dealing with chunked response (#4630) #4846

[RESEND] Fix HttpPayloadParser dealing with chunked response (#4630) #4846

Conversation

rhdxmr commented Jul 1, 2020 • edited Loading

What do these changes do?

Are there changes in behavior for the user?

Related issue number

Checklist

codecov-commenter commented Jul 1, 2020 • edited Loading

Codecov Report

rhdxmr commented Jul 1, 2020 • edited Loading

rhdxmr commented Jul 5, 2020

socketpair commented Jul 5, 2020

socketpair commented Jul 15, 2020

asvetlov left a comment

Choose a reason for hiding this comment

rhdxmr commented Jul 1, 2020 •

edited

Loading

codecov-commenter commented Jul 1, 2020 •

edited

Loading

rhdxmr commented Jul 1, 2020 •

edited

Loading