Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SCL-178 Add concurency upload #33

Merged
merged 7 commits into from
Oct 9, 2020
Merged

SCL-178 Add concurency upload #33

merged 7 commits into from
Oct 9, 2020

Conversation

nicoferron
Copy link
Contributor

No description provided.

@coveralls
Copy link

coveralls commented Aug 24, 2020 •
edited
Loading

Coverage Status

Coverage increased (+74.007%) to 74.007% when pulling 4ee48f1 on SCL-178 into 56952d2 on master.

pyke369
pyke369 reviewed Sep 11, 2020
View reviewed changes
Copy link
Contributor

@pyke369 pyke369 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't test yet, but just looking at the code I don't see how upload resuming would work since there's no provisioned code to deal with 416 HTTP statuses and multi-valued Range headers in server responses.

Maximilien-R
Maximilien-R reviewed Sep 15, 2020
View reviewed changes
setup.py Show resolved Hide resolved
@nicoferron
Copy link
Contributor Author

Thanks for your feedbacks, I updated to deal with resuming upload and the package version has been updated to 0.2.4 In a different PR to remove the gap created in the previous update.
Can you take a look at this new version?

@nicoferron nicoferron requested review from pyke369 and Maximilien-R and removed request for mkniewallner September 23, 2020 09:03
pyke369
pyke369 reviewed Sep 24, 2020
View reviewed changes
Copy link
Contributor

@pyke369 pyke369 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here are my findings for this updated version. In order to test it, I spawned a local xupload server instance (on port 8000) and wrote the following test script:

$ cat client.py
#!/usr/bin/env python3

import xupload

upload = xupload.Xupload('http://localhost:8000/upload?uuid=00000000000000000000000000000000', 'video.mkv', 6)
print(upload.start())

Here is the run output:

$ time ./client.py
/usr/local/lib/python3.6/dist-packages/aiohttp-4.0.0a1-py3.6-linux-x86_64.egg/aiohttp/client.py:977: RuntimeWarning: coroutine 'noop' was never awaited
  self._resp.release()
{'hash':'ee8852245d06d40308a5f69175240b45c63e45fe', 'name':'video', 'size':'486090157'}
real 0m15,796s

Some issues I found:

  • first (small) problem is the warning you may see above
  • second (more serious) problem is the file gets somehow corrupted in transit:
$ ls -al video.mkv
486090157 video.mkv
$ sha1sum video.mkv
eb19fbb6a0ac30783b49a3ff9c5e23cc831601f7  video.mkv

$ ls -al 00000000000000000000000000000000.mkv
486090157 00000000000000000000000000000000.mkv
$ sha1sum 00000000000000000000000000000000.mkv
ee8852245d06d40308a5f69175240b45c63e45fe  00000000000000000000000000000000.mkv

(corresponding server logfile showing all the requests performed by the Python client)

13:25:57 {"completed":false,"concurrency":1,"duration":0.02,  "received":4263948,"remote":"127.0.0.1:26422","size":486090157,"start":0,        "throughput":1705.579}
13:25:57 {"completed":false,"concurrency":2,"duration":0.02,  "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":81015012, "throughput":1705.579}
13:25:57 {"completed":false,"concurrency":3,"duration":0.014, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":162030024,"throughput":2436.541}
13:25:57 {"completed":false,"concurrency":4,"duration":0.024, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":243045036,"throughput":1421.316}
13:25:57 {"completed":false,"concurrency":5,"duration":0.013, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":324060048,"throughput":2623.968}
13:25:57 {"completed":false,"concurrency":6,"duration":0.01,  "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":405075060,"throughput":3411.158}
13:25:57 {"completed":false,"concurrency":6,"duration":0.022, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":4263948,  "throughput":1550.526}
13:25:57 {"completed":false,"concurrency":6,"duration":0.015, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":166293972,"throughput":2274.105}
13:25:57 {"completed":false,"concurrency":6,"duration":0.009, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":328323996,"throughput":3790.176}
13:25:58 {"completed":false,"concurrency":6,"duration":0.012, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":85278960, "throughput":2842.632}
13:25:58 {"completed":false,"concurrency":6,"duration":0.013, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":409339008,"throughput":2623.968}
13:25:58 {"completed":false,"concurrency":6,"duration":0.008, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":170557920,"throughput":4263.948}
13:25:58 {"completed":false,"concurrency":6,"duration":0.013, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":247308984,"throughput":2623.968}
13:25:58 {"completed":false,"concurrency":6,"duration":0.012, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":332587944,"throughput":2842.632}
13:25:58 {"completed":false,"concurrency":6,"duration":0.008, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":413602956,"throughput":4263.948}
13:25:58 {"completed":false,"concurrency":6,"duration":0.019, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":8527896,  "throughput":1795.346}
13:25:58 {"completed":false,"concurrency":6,"duration":0.019, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":174821868,"throughput":1795.346}
13:25:58 {"completed":false,"concurrency":6,"duration":0.011, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":336851892,"throughput":3101.053}
13:25:59 {"completed":false,"concurrency":6,"duration":0.012, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":89542908, "throughput":2842.632}
13:25:59 {"completed":false,"concurrency":6,"duration":0.01,  "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":417866904,"throughput":3411.158}
13:25:59 {"completed":false,"concurrency":6,"duration":0.012, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":179085816,"throughput":2842.632}
13:25:59 {"completed":false,"concurrency":6,"duration":0.008, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":93806856  "throughput":4263.948}
13:25:59 {"completed":false,"concurrency":6,"duration":0.018, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":251572932,"throughput":1895.088}
13:25:59 {"completed":false,"concurrency":6,"duration":0.018, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":341115840,"throughput":1895.088}
13:25:59 {"completed":false,"concurrency":6,"duration":0.012, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":183349764,"throughput":2842.632}
13:25:59 {"completed":false,"concurrency":6,"duration":0.018, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":12791844, "throughput":1895.088}
13:25:59 {"completed":false,"concurrency":6,"duration":0.02,  "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":98070804, "throughput":1705.579}
13:25:59 {"completed":false,"concurrency":6,"duration":0.009, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":345379788,"throughput":3790.176}
13:26:00 {"completed":false,"concurrency":6,"duration":0.021, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":422130852,"throughput":1624.361}
13:26:00 {"completed":false,"concurrency":6,"duration":0.012, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":102334752,"throughput":2842.632}
13:26:00 {"completed":false,"concurrency":6,"duration":0.017, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":187613712,"throughput":2006.563}
13:26:00 {"completed":false,"concurrency":6,"duration":0.009, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":255836880,"throughput":3790.176}
13:26:00 {"completed":false,"concurrency":6,"duration":0.015, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":349643736,"throughput":2274.105}
13:26:00 {"completed":false,"concurrency":6,"duration":0.011, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":191877660,"throughput":3101.053}
13:26:00 {"completed":false,"concurrency":6,"duration":0.018, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":17055792, "throughput":1895.088}
13:26:00 {"completed":false,"concurrency":6,"duration":0.019, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":106598700,"throughput":1795.346}
13:26:00 {"completed":false,"concurrency":6,"duration":0.018, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":353907684,"throughput":1895.088}
13:26:01 {"completed":false,"concurrency":6,"duration":0.018, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":426394800,"throughput":1895.088}
13:26:01 {"completed":false,"concurrency":6,"duration":0.011, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":196141608,"throughput":3101.053}
13:26:01 {"completed":false,"concurrency":6,"duration":0.01,  "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":110862648,"throughput":3411.158}
13:26:01 {"completed":false,"concurrency":6,"duration":0.015, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":260100828,"throughput":2274.105}
13:26:01 {"completed":false,"concurrency":6,"duration":0.012, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":358171632,"throughput":2842.632}
13:26:01 {"completed":false,"concurrency":6,"duration":0.011, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":200405556,"throughput":3101.053}
13:26:01 {"completed":false,"concurrency":6,"duration":0.02,  "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":21319740, "throughput":1705.579}
13:26:01 {"completed":false,"concurrency":6,"duration":0.017, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":115126596,"throughput":2006.563}
13:26:01 {"completed":false,"concurrency":6,"duration":0.021, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":362435580,"throughput":1624.361}
13:26:01 {"completed":false,"concurrency":6,"duration":0.014, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":430658748,"throughput":2436.541}
13:26:01 {"completed":false,"concurrency":6,"duration":0.011, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":204669504,"throughput":3101.053}
13:26:01 {"completed":false,"concurrency":6,"duration":0.009, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":119390544,"throughput":3790.176}
13:26:02 {"completed":false,"concurrency":6,"duration":0.011, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":264364776,"throughput":3101.053}
13:26:02 {"completed":false,"concurrency":6,"duration":0.012, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":366699528,"throughput":2842.632}
13:26:02 {"completed":false,"concurrency":6,"duration":0.014, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":208933452,"throughput":2436.541}
13:26:02 {"completed":false,"concurrency":6,"duration":0.022, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":25583688, "throughput":1550.526}
13:26:02 {"completed":false,"concurrency":6,"duration":0.02,  "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":370963476,"throughput":1705.579}
13:26:02 {"completed":false,"concurrency":6,"duration":0.021, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":123654492,"throughput":1624.361}
13:26:02 {"completed":false,"concurrency":6,"duration":0.015, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":434922696,"throughput":2274.105}
13:26:02 {"completed":false,"concurrency":6,"duration":0.011, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":213197400,"throughput":3101.053}
13:26:02 {"completed":false,"concurrency":6,"duration":0.011, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":127918440,"throughput":3101.053}
13:26:03 {"completed":false,"concurrency":6,"duration":0.013, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":268628724,"throughput":2623.968}
13:26:03 {"completed":false,"concurrency":6,"duration":0.014, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":375227424,"throughput":2436.541}
13:26:03 {"completed":false,"concurrency":6,"duration":0.014, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":217461348,"throughput":2436.541}
13:26:03 {"completed":false,"concurrency":6,"duration":0.026, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":29847636, "throughput":1311.984}
13:26:03 {"completed":false,"concurrency":6,"duration":0.019, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":132182388,"throughput":1795.346}
13:26:03 {"completed":false,"concurrency":6,"duration":0.02,  "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":379491372,"throughput":1705.579}
13:26:03 {"completed":false,"concurrency":6,"duration":0.01,  "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":439186644,"throughput":3411.158}
13:26:03 {"completed":false,"concurrency":6,"duration":0.018, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":221725296,"throughput":1895.088}
13:26:03 {"completed":false,"concurrency":6,"duration":0.014, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":136446336,"throughput":2436.541}
13:26:03 {"completed":false,"concurrency":6,"duration":0.012, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":443450592,"throughput":2842.632}
13:26:04 {"completed":false,"concurrency":6,"duration":0.015, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":272892672,"throughput":2274.105}
13:26:04 {"completed":false,"concurrency":6,"duration":0.009, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":140710284,"throughput":3790.176}
13:26:04 {"completed":false,"concurrency":6,"duration":0.015, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":383755320,"throughput":2274.105}
13:26:04 {"completed":false,"concurrency":6,"duration":0.019, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":34111584, "throughput":1795.346}
13:26:04 {"completed":false,"concurrency":6,"duration":0.013, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":447714540,"throughput":2623.968}
13:26:04 {"completed":false,"concurrency":6,"duration":0.013, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":388019268,"throughput":2623.968}
13:26:04 {"completed":false,"concurrency":6,"duration":0.017, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":225989244,"throughput":2006.563}
13:26:04 {"completed":false,"concurrency":6,"duration":0.015, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":451978488,"throughput":2274.105}
13:26:04 {"completed":false,"concurrency":6,"duration":0.014, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":144974232,"throughput":2436.541}
13:26:05 {"completed":false,"concurrency":6,"duration":0.011, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":392283216,"throughput":3101.053}
13:26:05 {"completed":false,"concurrency":6,"duration":0.019, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":277156620,"throughput":1795.346}
13:26:05 {"completed":false,"concurrency":6,"duration":0.012, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":149238180,"throughput":2842.632}
13:26:05 {"completed":false,"concurrency":6,"duration":0.014, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":396547164,"throughput":2436.541}
13:26:05 {"completed":false,"concurrency":6,"duration":0.033, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":38375532, "throughput":1033.684}
13:26:05 {"completed":false,"concurrency":6,"duration":0.029, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":456242436,"throughput":1176.261}
13:26:05 {"completed":false,"concurrency":6,"duration":0.013, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":230253192,"throughput":2623.968}
13:26:05 {"completed":false,"concurrency":6,"duration":0.013, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":153502128,"throughput":2623.968}
13:26:05 {"completed":false,"concurrency":6,"duration":0.01,  "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":460506384,"throughput":3411.158}
13:26:06 {"completed":false,"concurrency":6,"duration":0.019, "received":4263948,"remote":"127.0.0.1:26424","size":486090157,"start":281420568,"throughput":1795.346}
13:26:06 {"completed":false,"concurrency":5,"duration":0.016, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":400811112,"throughput":2131.974}
13:26:06 {"completed":false,"concurrency":4,"duration":0.01,  "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":157766076,"throughput":3411.158}
13:26:06 {"completed":false,"concurrency":4,"duration":0.021, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":42639480, "throughput":1624.361}
13:26:06 {"completed":false,"concurrency":4,"duration":0.015, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":464770332,"throughput":2274.105}
13:26:06 {"completed":false,"concurrency":4,"duration":0.013, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":234517140,"throughput":2623.968}
13:26:06 {"completed":false,"concurrency":4,"duration":0.013, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":469034280,"throughput":2623.968}
13:26:07 {"completed":false,"concurrency":4,"duration":0.013, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":285684516,"throughput":2623.968}
13:26:07 {"completed":false,"concurrency":3,"duration":0.011, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":238781088,"throughput":3101.053}
13:26:07 {"completed":false,"concurrency":3,"duration":0.019, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":46903428, "throughput":1795.346}
13:26:07 {"completed":false,"concurrency":3,"duration":0.019, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":289948464,"throughput":1795.346}
13:26:07 {"completed":false,"concurrency":3,"duration":0.015, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":473298228,"throughput":2274.105}
13:26:07 {"completed":false,"concurrency":3,"duration":0.018, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":51167376, "throughput":1895.088}
13:26:08 {"completed":false,"concurrency":3,"duration":0.013, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":477562176,"throughput":2623.968}
13:26:08 {"completed":false,"concurrency":3,"duration":0.016, "received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":294212412,"throughput":2131.974}
13:26:08 {"completed":false,"concurrency":3,"duration":0.012, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":481826124,"throughput":2842.632}
13:26:08 {"completed":false,"concurrency":3,"duration":0.004, "received":85,     "remote":"127.0.0.1:26426","size":486090157,"start":486090072,"throughput":0.17}
13:26:08 {"completed":false,"concurrency":3,"duration":0.019, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":55431324, "throughput":1795.346}
13:26:08 {"completed":false,"concurrency":3,"duration":0.011, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":298476360,"throughput":3101.053}
13:26:09 {"completed":false,"concurrency":3,"duration":0.018, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":59695272, "throughput":1895.088}
13:26:09 {"completed":false,"concurrency":3,"duration":0.014, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":302740308,"throughput":2436.541}
13:26:09 {"completed":false,"concurrency":3,"duration":0.024, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":63959220, "throughput":1421.316}
13:26:10 {"completed":false,"concurrency":3,"duration":0.016, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":307004256,"throughput":2131.974}
13:26:10 {"completed":false,"concurrency":3,"duration":0.017, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":68223168, "throughput":2006.563}
13:26:10 {"completed":false,"concurrency":3,"duration":0.01,  "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":311268204,"throughput":3411.158}
13:26:11 {"completed":false,"concurrency":3,"duration":0.015, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":72487116, "throughput":2274.105}
13:26:11 {"completed":false,"concurrency":3,"duration":0.008, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":315532152,"throughput":4263.948}
13:26:12 {"completed":false,"concurrency":2,"duration":0.369, "received":4263948,"remote":"127.0.0.1:26428","size":486090157,"start":76751064, "throughput":92.443}
13:26:12 {"completed":true, "concurrency":1,"duration":14.667,"received":4263948,"remote":"127.0.0.1:26426","size":486090157,"start":319796100,"throughput":265.134}
  • third problem is upload performance (compared to the PHP version for instance):
$ time ./xupload.php http://localhost:8000/upload?uuid=00000000000000000000000000000000 video.mkv 6
Array
(
    [hash] => eb19fbb6a0ac30783b49a3ff9c5e23cc831601f7
    [name] => video
    [size] => 486090157
)
real 0m2,486s

$ ls -al 00000000000000000000000000000000.mkv
486090157 00000000000000000000000000000000.mkv
$ sha1sum 00000000000000000000000000000000.mkv
eb19fbb6a0ac30783b49a3ff9c5e23cc831601f7  00000000000000000000000000000000.mkv

(corresponding server logfile showing all the requests performed by the PHP client):

13:38:05 INFO {"completed":false,"concurrency":1,"duration":0.046,"received":4263948,"remote":"127.0.0.1:26466","size":486090157,"start":162030024,"throughput":741.556}
13:38:05 INFO {"completed":false,"concurrency":2,"duration":0.057,"received":4263948,"remote":"127.0.0.1:26462","size":486090157,"start":0,        "throughput":598.448}
13:38:05 INFO {"completed":false,"concurrency":3,"duration":0.054,"received":4263948,"remote":"127.0.0.1:26470","size":486090157,"start":324060048,"throughput":631.696}
13:38:05 INFO {"completed":false,"concurrency":4,"duration":0.054,"received":4263948,"remote":"127.0.0.1:26472","size":486090157,"start":405075060,"throughput":631.696}
13:38:05 INFO {"completed":false,"concurrency":5,"duration":0.056,"received":4263948,"remote":"127.0.0.1:26464","size":486090157,"start":81015012, "throughput":609.135}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.058,"received":4263948,"remote":"127.0.0.1:26468","size":486090157,"start":243045036,"throughput":588.13}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.051,"received":4263948,"remote":"127.0.0.1:26470","size":486090157,"start":328323996,"throughput":668.854}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.072,"received":4263948,"remote":"127.0.0.1:26462","size":486090157,"start":166293972,"throughput":473.772}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.05, "received":4263948,"remote":"127.0.0.1:26472","size":486090157,"start":409339008,"throughput":682.231}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.063,"received":4263948,"remote":"127.0.0.1:26464","size":486090157,"start":4263948,  "throughput":541.453}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.062,"received":4263948,"remote":"127.0.0.1:26468","size":486090157,"start":247308984,"throughput":550.186}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.063,"received":4263948,"remote":"127.0.0.1:26466","size":486090157,"start":85278960, "throughput":541.453}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.077,"received":4263948,"remote":"127.0.0.1:26464","size":486090157,"start":332587944,"throughput":443.007}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.061,"received":4263948,"remote":"127.0.0.1:26470","size":486090157,"start":89542908, "throughput":559.206}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.077,"received":4263948,"remote":"127.0.0.1:26462","size":486090157,"start":170557920,"throughput":443.007}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.068,"received":4263948,"remote":"127.0.0.1:26468","size":486090157,"start":8527896,  "throughput":501.64}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.067,"received":4263948,"remote":"127.0.0.1:26472","size":486090157,"start":251572932,"throughput":509.128}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.086,"received":4263948,"remote":"127.0.0.1:26466","size":486090157,"start":413602956,"throughput":396.646}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.067,"received":4263948,"remote":"127.0.0.1:26470","size":486090157,"start":12791844, "throughput":509.128}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.082,"received":4263948,"remote":"127.0.0.1:26464","size":486090157,"start":93806856, "throughput":415.994}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.067,"received":4263948,"remote":"127.0.0.1:26468","size":486090157,"start":417866904,"throughput":509.128}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.082,"received":4263948,"remote":"127.0.0.1:26462","size":486090157,"start":336851892,"throughput":415.994}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.066,"received":4263948,"remote":"127.0.0.1:26472","size":486090157,"start":255836880,"throughput":516.842}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.066,"received":4263948,"remote":"127.0.0.1:26466","size":486090157,"start":174821868,"throughput":516.842}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.066,"received":4263948,"remote":"127.0.0.1:26464","size":486090157,"start":341115840,"throughput":516.842}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.073,"received":4263948,"remote":"127.0.0.1:26472","size":486090157,"start":260100828,"throughput":467.281}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.103,"received":4263948,"remote":"127.0.0.1:26462","size":486090157,"start":17055792, "throughput":331.18}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.073,"received":4263948,"remote":"127.0.0.1:26470","size":486090157,"start":422130852,"throughput":467.281}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.073,"received":4263948,"remote":"127.0.0.1:26466","size":486090157,"start":98070804, "throughput":467.281}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.073,"received":4263948,"remote":"127.0.0.1:26468","size":486090157,"start":179085816,"throughput":467.281}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.068,"received":4263948,"remote":"127.0.0.1:26470","size":486090157,"start":426394800,"throughput":501.64}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.068,"received":4263948,"remote":"127.0.0.1:26466","size":486090157,"start":102334752,"throughput":501.64}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.068,"received":4263948,"remote":"127.0.0.1:26472","size":486090157,"start":264364776,"throughput":501.64}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.068,"received":4263948,"remote":"127.0.0.1:26468","size":486090157,"start":183349764,"throughput":501.64}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.076,"received":4263948,"remote":"127.0.0.1:26464","size":486090157,"start":21319740, "throughput":448.836}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.096,"received":4263948,"remote":"127.0.0.1:26462","size":486090157,"start":345379788,"throughput":355.329}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.073,"received":4263948,"remote":"127.0.0.1:26466","size":486090157,"start":349643736,"throughput":467.281}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.09, "received":4263948,"remote":"127.0.0.1:26464","size":486090157,"start":430658748,"throughput":379.017}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.081,"received":4263948,"remote":"127.0.0.1:26468","size":486090157,"start":25583688, "throughput":421.13}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.08, "received":4263948,"remote":"127.0.0.1:26470","size":486090157,"start":187613712,"throughput":426.394}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.099,"received":4263948,"remote":"127.0.0.1:26462","size":486090157,"start":106598700,"throughput":344.561}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.08, "received":4263948,"remote":"127.0.0.1:26472","size":486090157,"start":268628724,"throughput":426.394}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.057,"received":4263948,"remote":"127.0.0.1:26470","size":486090157,"start":191877660,"throughput":598.448}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.057,"received":4263948,"remote":"127.0.0.1:26472","size":486090157,"start":272892672,"throughput":598.448}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.058,"received":4263948,"remote":"127.0.0.1:26466","size":486090157,"start":110862648,"throughput":588.13}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.09, "received":4263948,"remote":"127.0.0.1:26464","size":486090157,"start":353907684,"throughput":379.017}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.09, "received":4263948,"remote":"127.0.0.1:26462","size":486090157,"start":434922696,"throughput":379.017}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.071,"received":4263948,"remote":"127.0.0.1:26468","size":486090157,"start":29847636, "throughput":480.444}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.059,"received":4263948,"remote":"127.0.0.1:26470","size":486090157,"start":115126596,"throughput":578.162}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.058,"received":4263948,"remote":"127.0.0.1:26466","size":486090157,"start":439186644,"throughput":588.13}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.06, "received":4263948,"remote":"127.0.0.1:26468","size":486090157,"start":358171632,"throughput":568.526}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.076,"received":4263948,"remote":"127.0.0.1:26462","size":486090157,"start":196141608,"throughput":448.836}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.067,"received":4263948,"remote":"127.0.0.1:26472","size":486090157,"start":34111584, "throughput":509.128}
13:38:05 INFO {"completed":false,"concurrency":6,"duration":0.083,"received":4263948,"remote":"127.0.0.1:26464","size":486090157,"start":277156620,"throughput":410.982}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.074,"received":4263948,"remote":"127.0.0.1:26472","size":486090157,"start":38375532, "throughput":460.967}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.095,"received":4263948,"remote":"127.0.0.1:26464","size":486090157,"start":362435580,"throughput":359.069}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.095,"received":4263948,"remote":"127.0.0.1:26462","size":486090157,"start":443450592,"throughput":359.069}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.095,"received":4263948,"remote":"127.0.0.1:26466","size":486090157,"start":119390544,"throughput":359.069}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.077,"received":4263948,"remote":"127.0.0.1:26468","size":486090157,"start":200405556,"throughput":443.007}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.076,"received":4263948,"remote":"127.0.0.1:26470","size":486090157,"start":281420568,"throughput":448.836}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.073,"received":4263948,"remote":"127.0.0.1:26472","size":486090157,"start":42639480, "throughput":467.281}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.075,"received":4263948,"remote":"127.0.0.1:26462","size":486090157,"start":447714540,"throughput":454.821}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.074,"received":4263948,"remote":"127.0.0.1:26464","size":486090157,"start":366699528,"throughput":460.967}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.074,"received":4263948,"remote":"127.0.0.1:26470","size":486090157,"start":285684516,"throughput":460.967}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.074,"received":4263948,"remote":"127.0.0.1:26466","size":486090157,"start":123654492,"throughput":460.967}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.074,"received":4263948,"remote":"127.0.0.1:26468","size":486090157,"start":204669504,"throughput":460.967}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.061,"received":4263948,"remote":"127.0.0.1:26466","size":486090157,"start":370963476,"throughput":559.206}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.059,"received":4263948,"remote":"127.0.0.1:26464","size":486090157,"start":451978488,"throughput":578.162}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.06, "received":4263948,"remote":"127.0.0.1:26468","size":486090157,"start":127918440,"throughput":568.526}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.061,"received":4263948,"remote":"127.0.0.1:26472","size":486090157,"start":289948464,"throughput":559.206}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.061,"received":4263948,"remote":"127.0.0.1:26470","size":486090157,"start":208933452,"throughput":559.206}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.089,"received":4263948,"remote":"127.0.0.1:26462","size":486090157,"start":46903428, "throughput":383.276}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.069,"received":4263948,"remote":"127.0.0.1:26472","size":486090157,"start":213197400,"throughput":494.37}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.082,"received":4263948,"remote":"127.0.0.1:26462","size":486090157,"start":375227424,"throughput":415.994}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.082,"received":4263948,"remote":"127.0.0.1:26466","size":486090157,"start":294212412,"throughput":415.994}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.082,"received":4263948,"remote":"127.0.0.1:26464","size":486090157,"start":132182388,"throughput":415.994}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.081,"received":4263948,"remote":"127.0.0.1:26468","size":486090157,"start":51167376, "throughput":421.13}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.081,"received":4263948,"remote":"127.0.0.1:26470","size":486090157,"start":456242436,"throughput":421.13}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.108,"received":4263948,"remote":"127.0.0.1:26466","size":486090157,"start":217461348,"throughput":315.848}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.092,"received":4263948,"remote":"127.0.0.1:26472","size":486090157,"start":460506384,"throughput":370.778}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.093,"received":4263948,"remote":"127.0.0.1:26468","size":486090157,"start":136446336,"throughput":366.791}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.105,"received":4263948,"remote":"127.0.0.1:26470","size":486090157,"start":55431324, "throughput":324.872}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.121,"received":4263948,"remote":"127.0.0.1:26464","size":486090157,"start":298476360,"throughput":281.913}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.121,"received":4263948,"remote":"127.0.0.1:26462","size":486090157,"start":379491372,"throughput":281.913}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.088,"received":4263948,"remote":"127.0.0.1:26466","size":486090157,"start":464770332,"throughput":387.631}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.075,"received":4263948,"remote":"127.0.0.1:26470","size":486090157,"start":302740308,"throughput":454.821}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.086,"received":4263948,"remote":"127.0.0.1:26472","size":486090157,"start":59695272, "throughput":396.646}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.099,"received":4263948,"remote":"127.0.0.1:26462","size":486090157,"start":221725296,"throughput":344.561}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.086,"received":4263948,"remote":"127.0.0.1:26468","size":486090157,"start":383755320,"throughput":396.646}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.099,"received":4263948,"remote":"127.0.0.1:26464","size":486090157,"start":140710284,"throughput":344.561}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.064,"received":4263948,"remote":"127.0.0.1:26466","size":486090157,"start":225989244,"throughput":532.993}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.085,"received":4263948,"remote":"127.0.0.1:26462","size":486090157,"start":469034280,"throughput":401.312}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.07, "received":4263948,"remote":"127.0.0.1:26472","size":486090157,"start":63959220, "throughput":487.308}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.092,"received":4263948,"remote":"127.0.0.1:26464","size":486090157,"start":307004256,"throughput":370.778}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.069,"received":4263948,"remote":"127.0.0.1:26470","size":486090157,"start":388019268,"throughput":494.37}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.069,"received":4263948,"remote":"127.0.0.1:26468","size":486090157,"start":144974232,"throughput":494.37}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.058,"received":4263948,"remote":"127.0.0.1:26470","size":486090157,"start":392283216,"throughput":588.13}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.083,"received":4263948,"remote":"127.0.0.1:26464","size":486090157,"start":230253192,"throughput":410.982}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.083,"received":4263948,"remote":"127.0.0.1:26462","size":486090157,"start":473298228,"throughput":410.982"}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.058,"received":4263948,"remote":"127.0.0.1:26468","size":486090157,"start":149238180,"throughput":588.13}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.059,"received":4263948,"remote":"127.0.0.1:26466","size":486090157,"start":311268204,"throughput":578.162"}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.068,"received":4263948,"remote":"127.0.0.1:26472","size":486090157,"start":68223168, "throughput":501.64}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.076,"received":4263948,"remote":"127.0.0.1:26466","size":486090157,"start":234517140,"throughput":448.836}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.088,"received":4263948,"remote":"127.0.0.1:26462","size":486090157,"start":396547164,"throughput":387.631}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.075,"received":4263948,"remote":"127.0.0.1:26464","size":486090157,"start":477562176,"throughput":454.821}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.076,"received":4263948,"remote":"127.0.0.1:26470","size":486090157,"start":153502128,"throughput":448.836}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.076,"received":4263948,"remote":"127.0.0.1:26468","size":486090157,"start":315532152,"throughput":448.836}
13:38:06 INFO {"completed":false,"concurrency":6,"duration":0.083,"received":4263948,"remote":"127.0.0.1:26472","size":486090157,"start":72487116, "throughput":410.982}
13:38:06 INFO {"completed":false,"concurrency":5,"duration":0.077,"received":4263948,"remote":"127.0.0.1:26468","size":486090157,"start":319796100,"throughput":443.007}
13:38:06 INFO {"completed":false,"concurrency":3,"duration":0.089,"received":4263948,"remote":"127.0.0.1:26462","size":486090157,"start":400811112,"throughput":383.276}
13:38:06 INFO {"completed":false,"concurrency":4,"duration":0.077,"received":4263948,"remote":"127.0.0.1:26470","size":486090157,"start":157766076,"throughput":443.007}
13:38:06 INFO {"completed":false,"concurrency":3,"duration":0.089,"received":4263948,"remote":"127.0.0.1:26464","size":486090157,"start":481826124,"throughput":383.276}
13:38:07 INFO {"completed":false,"concurrency":2,"duration":0.319,"received":4263948,"remote":"127.0.0.1:26472","size":486090157,"start":76751064, "throughput":106.932}
13:38:07 INFO {"completed":false,"concurrency":1,"duration":0.647,"received":4263948,"remote":"127.0.0.1:26466","size":486090157,"start":238781088,"throughput":52.722}
13:38:07 INFO {"completed":true,"concurrency":1,"duration":2.431, "received":85,     "remote":"127.0.0.1:26462","size":486090157,"start":486090072,"throughput":1599.638}

Please fix these issues before we can ship this new version.

@nicoferron
Copy link
Contributor Author

Thanks for spotting these issues and for all this helpful information, I've fixed the file issue. The warning seems to be related to the version of aiohttp you use which is the latest(4.0.0a1). PIP should pull the stable one (3.6.2) I excluded v4.0.0a1 to be safe.
About the performance issue, I'm not experiencing a significative difference between PHP and Python SDKs upload time (even in python 3.6 & aiohttp 4.0.0a1), let me know if this issue persists after this fix

@pyke369
Copy link
Contributor

pyke369 commented Oct 5, 2020 •
edited
Loading

Hello there, we meet again :-)

This new version seems better indeed, here are my revised comments:

  • I didn't use pip to install the SDK, put followed the documentation in the README file (i.e. invoked python3 setup.py install directly), so I would also suggest the following change in the setup.py file:
  'requests_toolbelt',
- 'aiohttp;python_version>"3.4"',
+ 'aiohttp!=4.0.0a1;python_version>"3.4"',
  'aiofiles;python_version>"3.4"',

It actually installed aiohttp v4.0.0a0 (instead of aiohttp v4.0.0a1), and this version doesn't seem to print the "noop coroutine" warning anymore:

Searching for aiohttp!=4.0.0a1
Reading https://pypi.python.org/simple/aiohttp/
Downloading https://files.pythonhosted.org/packages/02/d0/95aa6e81c978fe436d41a1a0511dfb66598888c7fc7d93ade5fee9483d7a/aiohttp-4.0.0a0-cp36-cp36m-manylinux1_x86_64.whl#sha256=abc43651f2c6d70b812cb874fc8caa00eb69fac11930cd2640d32a11ee7beef0
Best match: aiohttp 4.0.0a0
Processing aiohttp-4.0.0a0-cp36-cp36m-manylinux1_x86_64.whl
Installing aiohttp-4.0.0a0-cp36-cp36m-manylinux1_x86_64.whl to /usr/local/lib/python3.6/dist-packages
writing requirements to /usr/local/lib/python3.6/dist-packages/aiohttp-4.0.0a0-py3.6-linux-x86_64.egg/EGG-INFO/requires.txt
Adding aiohttp 4.0.0a0 to easy-install.pth file
Installed /usr/local/lib/python3.6/dist-packages/aiohttp-4.0.0a0-py3.6-linux-x86_64.egg
  • the uploaded document is not corrupted in transit anymore 👍:
$ sha1sum video.mkv
eb19fbb6a0ac30783b49a3ff9c5e23cc831601f7  video.mkv

$ ./client.py
{'hash': 'eb19fbb6a0ac30783b49a3ff9c5e23cc831601f7', 'name': 'video', 'size': '486090157', 'url': 'http://localhost:8000/files/00000000000000000000000000000000.mkv#aGFzaD1lYjE5ZmJiNmEwYWMzMDc4M2I0OWEzZmY5YzVlMjNjYzgzMTYwMWY3Jm5hbWU9dmlkZW8mc2l6ZT00ODYwOTAxNTc='}
  • the performance problem is still there though:
$ time ./client.py
{'hash': 'eb19fbb6a0ac30783b49a3ff9c5e23cc831601f7', 'name': 'video', 'size': '486090157', 'url': 'http://localhost:8000/files/00000000000000000000000000000000.mkv#aGFzaD1lYjE5ZmJiNmEwYWMzMDc4M2I0OWEzZmY5YzVlMjNjYzgzMTYwMWY3Jm5hbWU9dmlkZW8mc2l6ZT00ODYwOTAxNTc='}
real	0m14,174s

$ time ./xupload.php http://localhost:8000/upload?uuid=00000000000000000000000000000000 video.mkv 6
Array
(
    [hash] => eb19fbb6a0ac30783b49a3ff9c5e23cc831601f7
    [name] => video
    [size] => 486090157
    [url] => http://localhost:8000/files/00000000000000000000000000000000.mkv#aGFzaD1lYjE5ZmJiNmEwYWMzMDc4M2I0OWEzZmY5YzVlMjNjYzgzMTYwMWY3Jm5hbWU9dmlkZW8mc2l6ZT00ODYwOTAxNTc=
)
real	0m2,307s

Even though both implementations make the same number of requests to the server (115), the Python implementation (using aio) is clearly not on par with PHP (using native multi-curl C extension) or Go (native concurrency support in the language) when it comes to concurrency performance; even if each unitary request seems to run faster in the Python version, the Python runtime seems to lag between requests for the same "client" sometimes, i.e. the event loop handler does not seem tight enough or as responsive compared to the other implementations:

----                   python                   ----     ----                    php                     ----
{"concurrency":1,"duration":0.032,"received":4263948}    {"concurrency":1,"duration":0.054,"received":4263948}
{"concurrency":2,"duration":0.025,"received":4263948}    {"concurrency":2,"duration":0.055,"received":4263948}
{"concurrency":3,"duration":0.014,"received":4263948}    {"concurrency":3,"duration":0.059,"received":4263948}
{"concurrency":4,"duration":0.028,"received":4263948}    {"concurrency":4,"duration":0.055,"received":4263948}
{"concurrency":5,"duration":0.041,"received":4263948}    {"concurrency":5,"duration":0.071,"received":4263948}
{"concurrency":6,"duration":0.028,"received":4263948}    {"concurrency":6,"duration":0.07, "received":4263948}
{"concurrency":6,"duration":0.019,"received":4263948}    {"concurrency":6,"duration":0.068,"received":4263948}
{"concurrency":6,"duration":0.012,"received":4263948}    {"concurrency":6,"duration":0.079,"received":4263948}
{"concurrency":6,"duration":0.024,"received":4263948}    {"concurrency":6,"duration":0.069,"received":4263948}
{"concurrency":6,"duration":0.01, "received":4263948}    {"concurrency":6,"duration":0.069,"received":4263948}
{"concurrency":6,"duration":0.021,"received":4263948}    {"concurrency":6,"duration":0.079,"received":4263948}
{"concurrency":6,"duration":0.018,"received":4263948}    {"concurrency":6,"duration":0.069,"received":4263948}
{"concurrency":6,"duration":0.018,"received":4263948}    {"concurrency":6,"duration":0.074,"received":4263948}
{"concurrency":6,"duration":0.013,"received":4263948}    {"concurrency":6,"duration":0.055,"received":4263948}
{"concurrency":6,"duration":0.012,"received":4263948}    {"concurrency":6,"duration":0.074,"received":4263948}
{"concurrency":6,"duration":0.017,"received":4263948}    {"concurrency":6,"duration":0.054,"received":4263948}
{"concurrency":6,"duration":0.014,"received":4263948}    {"concurrency":6,"duration":0.06, "received":4263948}
{"concurrency":6,"duration":0.021,"received":4263948}    {"concurrency":6,"duration":0.059,"received":4263948}
{"concurrency":6,"duration":0.056,"received":4263948}    {"concurrency":6,"duration":0.059,"received":4263948}
{"concurrency":6,"duration":0.041,"received":4263948}    {"concurrency":6,"duration":0.055,"received":4263948}
{"concurrency":6,"duration":0.013,"received":4263948}    {"concurrency":6,"duration":0.067,"received":4263948}
{"concurrency":6,"duration":0.011,"received":4263948}    {"concurrency":6,"duration":0.055,"received":4263948}
{"concurrency":6,"duration":0.013,"received":4263948}    {"concurrency":6,"duration":0.067,"received":4263948}
{"concurrency":6,"duration":0.014,"received":4263948}    {"concurrency":6,"duration":0.068,"received":4263948}
{"concurrency":6,"duration":0.011,"received":4263948}    {"concurrency":6,"duration":0.051,"received":4263948}
{"concurrency":6,"duration":0.02, "received":4263948}    {"concurrency":6,"duration":0.051,"received":4263948}
{"concurrency":6,"duration":0.012,"received":4263948}    {"concurrency":6,"duration":0.051,"received":4263948}
{"concurrency":6,"duration":0.01, "received":4263948}    {"concurrency":6,"duration":0.079,"received":4263948}
{"concurrency":6,"duration":0.031,"received":4263948}    {"concurrency":6,"duration":0.056,"received":4263948}
{"concurrency":6,"duration":0.021,"received":4263948}    {"concurrency":6,"duration":0.058,"received":4263948}
{"concurrency":6,"duration":0.018,"received":4263948}    {"concurrency":6,"duration":0.065,"received":4263948}
{"concurrency":6,"duration":0.023,"received":4263948}    {"concurrency":6,"duration":0.055,"received":4263948}
{"concurrency":6,"duration":0.022,"received":4263948}    {"concurrency":6,"duration":0.065,"received":4263948}
{"concurrency":6,"duration":0.021,"received":4263948}    {"concurrency":6,"duration":0.059,"received":4263948}
{"concurrency":6,"duration":0.016,"received":4263948}    {"concurrency":6,"duration":0.073,"received":4263948}
{"concurrency":6,"duration":0.015,"received":4263948}    {"concurrency":6,"duration":0.086,"received":4263948}
{"concurrency":6,"duration":0.016,"received":4263948}    {"concurrency":6,"duration":0.063,"received":4263948}
{"concurrency":6,"duration":0.037,"received":4263948}    {"concurrency":6,"duration":0.063,"received":4263948}
{"concurrency":6,"duration":0.025,"received":4263948}    {"concurrency":6,"duration":0.054,"received":4263948}
{"concurrency":6,"duration":0.024,"received":4263948}    {"concurrency":6,"duration":0.079,"received":4263948}
{"concurrency":6,"duration":0.018,"received":4263948}    {"concurrency":6,"duration":0.055,"received":4263948}
{"concurrency":6,"duration":0.096,"received":4263948}    {"concurrency":6,"duration":0.055,"received":4263948}
{"concurrency":6,"duration":0.099,"received":4263948}    {"concurrency":6,"duration":0.065,"received":4263948}
{"concurrency":6,"duration":0.018,"received":4263948}    {"concurrency":6,"duration":0.064,"received":4263948}
{"concurrency":6,"duration":0.013,"received":4263948}    {"concurrency":6,"duration":0.086,"received":4263948}
{"concurrency":6,"duration":0.013,"received":4263948}    {"concurrency":6,"duration":0.067,"received":4263948}
{"concurrency":6,"duration":0.035,"received":4263948}    {"concurrency":6,"duration":0.086,"received":4263948}
{"concurrency":6,"duration":0.026,"received":4263948}    {"concurrency":6,"duration":0.065,"received":4263948}
{"concurrency":6,"duration":0.045,"received":4263948}    {"concurrency":6,"duration":0.05, "received":4263948}
{"concurrency":6,"duration":0.01, "received":4263948}    {"concurrency":6,"duration":0.065,"received":4263948}
{"concurrency":6,"duration":0.018,"received":4263948}    {"concurrency":6,"duration":0.05, "received":4263948}
{"concurrency":6,"duration":0.014,"received":4263948}    {"concurrency":6,"duration":0.072,"received":4263948}
{"concurrency":6,"duration":0.015,"received":4263948}    {"concurrency":6,"duration":0.073,"received":4263948}
{"concurrency":6,"duration":0.022,"received":4263948}    {"concurrency":6,"duration":0.072,"received":4263948}
{"concurrency":6,"duration":0.012,"received":4263948}    {"concurrency":6,"duration":0.064,"received":4263948}
{"concurrency":6,"duration":0.031,"received":4263948}    {"concurrency":6,"duration":0.057,"received":4263948}
{"concurrency":6,"duration":0.026,"received":4263948}    {"concurrency":6,"duration":0.056,"received":4263948}
{"concurrency":6,"duration":0.022,"received":4263948}    {"concurrency":6,"duration":0.074,"received":4263948}
{"concurrency":6,"duration":0.013,"received":4263948}    {"concurrency":6,"duration":0.078,"received":4263948}
{"concurrency":6,"duration":0.023,"received":4263948}    {"concurrency":6,"duration":0.056,"received":4263948}
{"concurrency":6,"duration":0.012,"received":4263948}    {"concurrency":6,"duration":0.083,"received":4263948}
{"concurrency":6,"duration":0.021,"received":4263948}    {"concurrency":6,"duration":0.06, "received":4263948}
{"concurrency":6,"duration":0.015,"received":4263948}    {"concurrency":6,"duration":0.063,"received":4263948}
{"concurrency":6,"duration":0.018,"received":4263948}    {"concurrency":6,"duration":0.059,"received":4263948}
{"concurrency":6,"duration":0.016,"received":4263948}    {"concurrency":6,"duration":0.061,"received":4263948}
{"concurrency":6,"duration":0.034,"received":4263948}    {"concurrency":6,"duration":0.065,"received":4263948}
{"concurrency":6,"duration":0.03, "received":4263948}    {"concurrency":6,"duration":0.072,"received":4263948}
{"concurrency":6,"duration":0.033,"received":4263948}    {"concurrency":6,"duration":0.049,"received":4263948}
{"concurrency":6,"duration":0.013,"received":4263948}    {"concurrency":6,"duration":0.049,"received":4263948}
{"concurrency":6,"duration":0.019,"received":4263948}    {"concurrency":6,"duration":0.05, "received":4263948}
{"concurrency":6,"duration":0.011,"received":4263948}    {"concurrency":6,"duration":0.072,"received":4263948}
{"concurrency":6,"duration":0.013,"received":4263948}    {"concurrency":6,"duration":0.056,"received":4263948}
{"concurrency":6,"duration":0.021,"received":4263948}    {"concurrency":6,"duration":0.054,"received":4263948}
{"concurrency":6,"duration":0.011,"received":4263948}    {"concurrency":6,"duration":0.052,"received":4263948}
{"concurrency":6,"duration":0.021,"received":4263948}    {"concurrency":6,"duration":0.054,"received":4263948}
{"concurrency":6,"duration":0.024,"received":4263948}    {"concurrency":6,"duration":0.054,"received":4263948}
{"concurrency":6,"duration":0.022,"received":4263948}    {"concurrency":6,"duration":0.062,"received":4263948}
{"concurrency":6,"duration":0.011,"received":4263948}    {"concurrency":6,"duration":0.056,"received":4263948}
{"concurrency":6,"duration":0.013,"received":4263948}    {"concurrency":6,"duration":0.047,"received":4263948}
{"concurrency":6,"duration":0.012,"received":4263948}    {"concurrency":6,"duration":0.056,"received":4263948}
{"concurrency":6,"duration":0.014,"received":4263948}    {"concurrency":6,"duration":0.056,"received":4263948}
{"concurrency":6,"duration":0.018,"received":4263948}    {"concurrency":6,"duration":0.054,"received":4263948}
{"concurrency":6,"duration":0.017,"received":4263948}    {"concurrency":6,"duration":0.062,"received":4263948}
{"concurrency":6,"duration":0.021,"received":4263948}    {"concurrency":6,"duration":0.064,"received":4263948}
{"concurrency":6,"duration":0.02, "received":4263948}    {"concurrency":6,"duration":0.049,"received":4263948}
{"concurrency":6,"duration":0.016,"received":4263948}    {"concurrency":6,"duration":0.067,"received":4263948}
{"concurrency":6,"duration":0.022,"received":4263948}    {"concurrency":6,"duration":0.066,"received":4263948}
{"concurrency":6,"duration":0.029,"received":4263948}    {"concurrency":6,"duration":0.066,"received":4263948}
{"concurrency":6,"duration":0.027,"received":4263948}    {"concurrency":6,"duration":0.049,"received":4263948}
{"concurrency":6,"duration":0.024,"received":4263948}    {"concurrency":6,"duration":0.057,"received":4263948}
{"concurrency":5,"duration":0.02, "received":4263948}    {"concurrency":6,"duration":0.054,"received":4263948}
{"concurrency":5,"duration":0.019,"received":4263948}    {"concurrency":6,"duration":0.054,"received":4263948}
{"concurrency":5,"duration":0.023,"received":4263948}    {"concurrency":6,"duration":0.055,"received":4263948}
{"concurrency":5,"duration":0.049,"received":4263948}    {"concurrency":6,"duration":0.055,"received":4263948}
{"concurrency":5,"duration":0.011,"received":4263948}    {"concurrency":6,"duration":0.062,"received":4263948}
{"concurrency":5,"duration":0.015,"received":4263948}    {"concurrency":6,"duration":0.063,"received":4263948}
{"concurrency":5,"duration":0.004,"received":85}         {"concurrency":6,"duration":0.073,"received":4263948}
{"concurrency":5,"duration":0.012,"received":4263948}    {"concurrency":6,"duration":0.073,"received":4263948}
{"concurrency":5,"duration":0.015,"received":4263948}    {"concurrency":6,"duration":0.072,"received":4263948}
{"concurrency":5,"duration":0.013,"received":4263948}    {"concurrency":6,"duration":0.066,"received":4263948}
{"concurrency":5,"duration":0.009,"received":4263948}    {"concurrency":6,"duration":0.067,"received":4263948}
{"concurrency":5,"duration":0.011,"received":4263948}    {"concurrency":6,"duration":0.067,"received":4263948}
{"concurrency":4,"duration":0.01, "received":4263948}    {"concurrency":6,"duration":0.049,"received":4263948}
{"concurrency":4,"duration":0.019,"received":4263948}    {"concurrency":6,"duration":0.049,"received":4263948}
{"concurrency":4,"duration":0.017,"received":4263948}    {"concurrency":6,"duration":0.065,"received":4263948}
{"concurrency":4,"duration":0.028,"received":4263948}    {"concurrency":6,"duration":0.057,"received":4263948}
{"concurrency":4,"duration":0.035,"received":4263948}    {"concurrency":6,"duration":0.074,"received":4263948}
{"concurrency":3,"duration":0.01, "received":4263948}    {"concurrency":6,"duration":0.074,"received":4263948}
{"concurrency":3,"duration":0.013,"received":4263948}    {"concurrency":5,"duration":0.079,"received":4263948}
{"concurrency":3,"duration":0.02, "received":4263948}    {"concurrency":4,"duration":0.177,"received":4263948}
{"concurrency":3,"duration":0.011,"received":4263948}    {"concurrency":4,"duration":0.179,"received":4263948}
{"concurrency":3,"duration":0.017,"received":4263948}    {"concurrency":3,"duration":0.195,"received":4263948}
{"concurrency":3,"duration":0.024,"received":4263948}    {"concurrency":2,"duration":0.51, "received":4263948}
{"concurrency":2,"duration":0.381,"received":4263948}    {"concurrency":1,"duration":0.609,"received":4263948}
{"concurrency":1,"duration":13.447"received":4263948}    {"concurrency":1,"duration":2.23, "received":85}

Anyways, this is probably showing b/c I'm testing towards a local xupload server, and will not be as blatant when the network latency is actually the limiting factor (i.e. in a real world environment where the client-server latency is more in the "ms range" than the "us range").

Since the only real issue (document corruption in transit) is now fixed, I'm going to approve this.

Maximilien-R
Maximilien-R reviewed Oct 8, 2020
View reviewed changes
xupload.py Outdated Show resolved Hide resolved
@nicoferron nicoferron merged commit 9862165 into master Oct 9, 2020
@dland dland deleted the SCL-178 branch October 1, 2024 13:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pyke369 pyke369 pyke369 approved these changes

@Maximilien-R Maximilien-R Maximilien-R approved these changes

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@nicoferron @coveralls @pyke369 @Maximilien-R