Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

wip: s3 parallelism #2554

Draft
wants to merge 1 commit into
base: develop
Choose a base branch
from
Draft

wip: s3 parallelism #2554

wants to merge 1 commit into from

Conversation

onursatici
Copy link
Contributor

No description provided.

@github-actions GitHub Actions
Copy link
Contributor

Benchmarks: TPC-H on NVME

Table of Results
name PR 05b27b1 base 81b1340 ratio (PR/base) unit
tpch_q01/arrow 78779271 7.35695e+07 1.07081 ns
tpch_q02/arrow 42661284 4.11476e+07 1.03679 ns
tpch_q03/arrow 34502625 3.34339e+07 1.03197 ns
tpch_q04/arrow 31751347 3.05241e+07 1.0402 ns
tpch_q05/arrow 55452651 5.46785e+07 1.01416 ns
tpch_q06/arrow 8073884 7.84182e+06 1.02959 ns
tpch_q07/arrow 100096221 9.65308e+07 1.03694 ns
tpch_q08/arrow 57727789 5.72194e+07 1.00889 ns
tpch_q09/arrow 85507650 8.3069e+07 1.02936 ns
tpch_q10/arrow 54538822 5.4485e+07 1.00099 ns
tpch_q11/arrow 24614816 2.43456e+07 1.01106 ns
tpch_q12/arrow 31497970 3.09688e+07 1.01709 ns
tpch_q13/arrow 26005826 2.51093e+07 1.03571 ns
tpch_q14/arrow 11480287 1.12399e+07 1.02139 ns
tpch_q15/arrow 24893949 2.39551e+07 1.03919 ns
tpch_q16/arrow 22359385 2.1103e+07 1.05954 ns
tpch_q17/arrow 82048440 7.83006e+07 1.04787 ns
tpch_q18/arrow 164355381 1.55433e+08 1.0574 ns
tpch_q19/arrow 26020614 2.59321e+07 1.00341 ns
tpch_q20/arrow 37996421 3.58362e+07 1.06028 ns
tpch_q21/arrow 157753931 1.55678e+08 1.01333 ns
tpch_q22/arrow 15946262 1.5885e+07 1.00386 ns
tpch_q01/parquet 155555142 1.41338e+08 1.10059 ns
tpch_q02/parquet 99830779 9.41264e+07 1.0606 ns
tpch_q03/parquet 109105228 1.03909e+08 1.05 ns
tpch_q04/parquet 60043105 6.01513e+07 0.998201 ns
tpch_q05/parquet 114589999 1.13803e+08 1.00691 ns
tpch_q06/parquet 28057637 2.83167e+07 0.990853 ns
tpch_q07/parquet 147766908 1.43371e+08 1.03066 ns
tpch_q08/parquet 157403355 1.49821e+08 1.05061 ns
tpch_q09/parquet 201454641 2.01912e+08 0.997733 ns
tpch_q10/parquet 152587466 1.52578e+08 1.00006 ns
tpch_q11/parquet 49196499 4.63423e+07 1.06159 ns
tpch_q12/parquet 100266386 9.75665e+07 1.02767 ns
tpch_q13/parquet 186892739 1.82009e+08 1.02683 ns
tpch_q14/parquet 49153831 4.84318e+07 1.01491 ns
tpch_q15/parquet 85974116 8.0619e+07 1.06643 ns
tpch_q16/parquet 48391600 4.67724e+07 1.03462 ns
tpch_q17/parquet 147781312 1.39517e+08 1.05924 ns
tpch_q18/parquet 247300654 2.44078e+08 1.0132 ns
tpch_q19/parquet 82925424 8.33196e+07 0.995269 ns
tpch_q20/parquet 95569356 1.00092e+08 0.954813 ns
tpch_q21/parquet 215192296 2.04332e+08 1.05315 ns
tpch_q22/parquet 50336724 4.80282e+07 1.04807 ns
tpch_q01/vortex-file-compressed 87606068 6.28855e+07 1.3931 ns
tpch_q02/vortex-file-compressed 71583824 4.45177e+07 1.60799 ns
tpch_q03/vortex-file-compressed 67940758 3.27231e+07 2.07623 ns
tpch_q04/vortex-file-compressed 31507158 2.05025e+07 1.53675 ns
tpch_q05/vortex-file-compressed 89332828 5.28908e+07 1.68901 ns
tpch_q06/vortex-file-compressed 13682508 1.10427e+07 1.23905 ns
tpch_q07/vortex-file-compressed 116690322 8.51906e+07 1.36976 ns
tpch_q08/vortex-file-compressed 112186977 6.06681e+07 1.84919 ns
tpch_q09/vortex-file-compressed 154343462 8.87914e+07 1.73827 ns
tpch_q10/vortex-file-compressed 66546129 5.62691e+07 1.18264 ns
tpch_q11/vortex-file-compressed 43724444 2.42585e+07 1.80244 ns
tpch_q12/vortex-file-compressed 30720588 2.92214e+07 1.0513 ns
tpch_q13/vortex-file-compressed 33880175 3.06992e+07 1.10362 ns
tpch_q14/vortex-file-compressed 17750720 1.43439e+07 1.23751 ns
tpch_q15/vortex-file-compressed 32732604 2.79722e+07 1.17018 ns
tpch_q16/vortex-file-compressed 37143681 2.5066e+07 1.48183 ns
tpch_q17/vortex-file-compressed 96707624 7.69412e+07 1.2569 ns
tpch_q18/vortex-file-compressed 196262576 1.41843e+08 1.38367 ns
tpch_q19/vortex-file-compressed 54363398 3.08625e+07 1.76147 ns
tpch_q20/vortex-file-compressed 61458987 3.86416e+07 1.59049 ns
tpch_q21/vortex-file-compressed 172597452 1.2461e+08 1.3851 ns
tpch_q22/vortex-file-compressed 59202431 2.82324e+07 2.09697 ns

@github-actions GitHub Actions
Copy link
Contributor

Benchmarks: TPC-H on S3

Table of Results
name PR 05b27b1 base 81b1340 ratio (PR/base) unit
tpch_q01/parquet 323945676 3.30902e+08 0.978977 ns
tpch_q02/parquet 985626011 7.97528e+08 1.23585 ns
tpch_q03/parquet 543421836 4.79405e+08 1.13353 ns
tpch_q04/parquet 261107021 2.61969e+08 0.996711 ns
tpch_q05/parquet 622588721 6.31569e+08 0.985781 ns
tpch_q06/parquet 193634862 1.99841e+08 0.968945 ns
tpch_q07/parquet 709233617 7.32948e+08 0.967645 ns
tpch_q08/parquet 867976412 8.69676e+08 0.998046 ns
tpch_q09/parquet 739790792 7.36455e+08 1.00453 ns
tpch_q10/parquet 624151949 6.09324e+08 1.02433 ns
tpch_q11/parquet 295340456 3.13442e+08 0.942249 ns
tpch_q12/parquet 298923884 2.99698e+08 0.997416 ns
tpch_q13/parquet 431446275 4.35667e+08 0.990312 ns
tpch_q14/parquet 282757704 2.78371e+08 1.01576 ns
tpch_q15/parquet 504768596 5.06847e+08 0.995899 ns
tpch_q16/parquet 308475552 2.85373e+08 1.08096 ns
tpch_q17/parquet 440806679 4.42373e+08 0.996459 ns
tpch_q18/parquet 627690923 6.10071e+08 1.02888 ns
tpch_q19/parquet 324762578 3.30946e+08 0.981315 ns
tpch_q20/parquet 564373078 5.87303e+08 0.960957 ns
tpch_q21/parquet 716494356 7.11911e+08 1.00644 ns
tpch_q22/parquet 290122350 3.21239e+08 0.903135 ns
tpch_q01/vortex-file-compressed 238417054 3.31174e+08 0.719915 ns
tpch_q02/vortex-file-compressed 475276950 4.63201e+08 1.02607 ns
tpch_q03/vortex-file-compressed 348139000 4.41245e+08 0.788993 ns
tpch_q04/vortex-file-compressed 221730440 3.79696e+08 0.583968 ns
tpch_q05/vortex-file-compressed 406292177 5.06518e+08 0.802127 ns
tpch_q06/vortex-file-compressed 174802827 4.30623e+08 0.40593 ns
tpch_q07/vortex-file-compressed 449697050 6.35537e+08 0.707585 ns
tpch_q08/vortex-file-compressed 596118735 7.9204e+08 0.752637 ns
tpch_q09/vortex-file-compressed 554805335 6.54891e+08 0.847172 ns
tpch_q10/vortex-file-compressed 441127550 5.40188e+08 0.816619 ns
tpch_q11/vortex-file-compressed 196526239 1.66179e+08 1.18262 ns
tpch_q12/vortex-file-compressed 243124866 5.31127e+08 0.457752 ns
tpch_q13/vortex-file-compressed 174283837 1.52432e+08 1.14335 ns
tpch_q14/vortex-file-compressed 152316557 2.93125e+08 0.519631 ns
tpch_q15/vortex-file-compressed 365132078 7.30281e+08 0.499988 ns
tpch_q16/vortex-file-compressed 218624396 2.09813e+08 1.042 ns
tpch_q17/vortex-file-compressed 293040733 4.43654e+08 0.660516 ns
tpch_q18/vortex-file-compressed 387690116 4.78978e+08 0.809411 ns
tpch_q19/vortex-file-compressed 276037166 4.68262e+08 0.589493 ns
tpch_q20/vortex-file-compressed 382931410 5.26041e+08 0.72795 ns
tpch_q21/vortex-file-compressed 587891298 9.83105e+08 0.597994 ns
tpch_q22/vortex-file-compressed 184536661 1.52226e+08 1.21225 ns

@github-actions GitHub Actions
Copy link
Contributor

Benchmarks: Clickbench on NVME

Table of Results
name PR 05b27b1 base 81b1340 ratio (PR/base) unit
clickbench_q00/parquet 2663708 2.12247e+06 1.255 ns
clickbench_q01/parquet 59848836 5.9912e+07 0.998946 ns
clickbench_q02/parquet 118845673 1.14296e+08 1.03981 ns
clickbench_q03/parquet 85391421 8.39559e+07 1.0171 ns
clickbench_q04/parquet 653571516 6.21643e+08 1.05136 ns
clickbench_q05/parquet 713905811 7.08042e+08 1.00828 ns
clickbench_q06/parquet 2211137 2.23942e+06 0.987372 ns
clickbench_q07/parquet 62582903 6.21373e+07 1.00717 ns
clickbench_q08/parquet 722228067 7.22739e+08 0.999293 ns
clickbench_q09/parquet 1022650725 1.005e+09 1.01757 ns
clickbench_q10/parquet 265576727 2.61605e+08 1.01518 ns
clickbench_q11/parquet 318652771 3.0583e+08 1.04193 ns
clickbench_q12/parquet 752855312 7.28395e+08 1.03358 ns
clickbench_q13/parquet 1004042030 9.90078e+08 1.0141 ns
clickbench_q14/parquet 732510053 7.23626e+08 1.01228 ns
clickbench_q15/parquet 735625425 7.2638e+08 1.01273 ns
clickbench_q16/parquet 1592979778 1.57131e+09 1.01379 ns
clickbench_q17/parquet 1414656605 1.40747e+09 1.00511 ns
clickbench_q18/parquet 2974349442 2.96442e+09 1.00335 ns
clickbench_q19/parquet 72529152 7.0236e+07 1.03265 ns
clickbench_q20/parquet 1113015512 1.12675e+09 0.987814 ns
clickbench_q21/parquet 1283264968 1.31686e+09 0.974491 ns
clickbench_q22/parquet 1871692734 1.88292e+09 0.994037 ns
clickbench_q23/parquet 7693929210 7.69572e+09 0.999767 ns
clickbench_q24/parquet 448212595 4.4803e+08 1.00041 ns
clickbench_q25/parquet 392441722 3.90153e+08 1.00587 ns
clickbench_q26/parquet 499353423 4.89369e+08 1.0204 ns
clickbench_q27/parquet 1592103049 1.59273e+09 0.999609 ns
clickbench_q28/parquet 11470232183 1.13498e+10 1.01061 ns
clickbench_q29/parquet 442173226 4.46644e+08 0.989991 ns
clickbench_q30/parquet 699152187 6.83145e+08 1.02343 ns
clickbench_q31/parquet 745551281 7.34458e+08 1.0151 ns
clickbench_q32/parquet 2791017892 2.75696e+09 1.01235 ns
clickbench_q33/parquet 2873408561 2.79836e+09 1.02682 ns
clickbench_q34/parquet 2785936411 2.75493e+09 1.01126 ns
clickbench_q35/parquet 876641036 8.56062e+08 1.02404 ns
clickbench_q36/parquet 182207683 1.74792e+08 1.04243 ns
clickbench_q37/parquet 87760954 8.77635e+07 0.999971 ns
clickbench_q38/parquet 113278238 1.08412e+08 1.04488 ns
clickbench_q39/parquet 324065989 3.09427e+08 1.04731 ns
clickbench_q40/parquet 55167136 5.46896e+07 1.00873 ns
clickbench_q41/parquet 53694558 5.25096e+07 1.02257 ns
clickbench_q42/parquet 71215772 7.09566e+07 1.00365 ns
clickbench_q00/vortex-file-compressed 5081972 4.59678e+06 1.10555 ns
clickbench_q01/vortex-file-compressed 19462473 2.15577e+07 0.902809 ns
clickbench_q02/vortex-file-compressed 106634250 4.06959e+07 2.62027 ns
clickbench_q03/vortex-file-compressed 103846643 6.3706e+07 1.63009 ns
clickbench_q04/vortex-file-compressed 607808237 5.71446e+08 1.06363 ns
clickbench_q05/vortex-file-compressed 585993072 5.96906e+08 0.981717 ns
clickbench_q06/vortex-file-compressed 4864038 4.67266e+06 1.04096 ns
clickbench_q07/vortex-file-compressed 23552545 2.63055e+07 0.895347 ns
clickbench_q08/vortex-file-compressed 714673644 6.51648e+08 1.09672 ns
clickbench_q09/vortex-file-compressed 752712976 7.94621e+08 0.947261 ns
clickbench_q10/vortex-file-compressed 111312843 1.14251e+08 0.974282 ns
clickbench_q11/vortex-file-compressed 130346454 1.31004e+08 0.994984 ns
clickbench_q12/vortex-file-compressed 463459904 4.75769e+08 0.974128 ns
clickbench_q13/vortex-file-compressed 679239909 6.9145e+08 0.982341 ns
clickbench_q14/vortex-file-compressed 428442038 4.39698e+08 0.974401 ns
clickbench_q15/vortex-file-compressed 685222429 6.72848e+08 1.01839 ns
clickbench_q16/vortex-file-compressed 1349164369 1.33483e+09 1.01074 ns
clickbench_q17/vortex-file-compressed 1320260801 1.32706e+09 0.994879 ns
clickbench_q18/vortex-file-compressed 2793830282 2.80118e+09 0.997377 ns
clickbench_q19/vortex-file-compressed 29814433 3.51448e+07 0.848332 ns
clickbench_q20/vortex-file-compressed 602157365 7.43137e+08 0.810291 ns
clickbench_q21/vortex-file-compressed 702636483 8.12035e+08 0.865279 ns
clickbench_q22/vortex-file-compressed 1185667637 1.14772e+09 1.03307 ns
clickbench_q23/vortex-file-compressed 2211386406 2.018e+09 1.09583 ns
clickbench_q24/vortex-file-compressed 154674323 1.81405e+08 0.852647 ns
clickbench_q25/vortex-file-compressed 164169661 1.95966e+08 0.837744 ns
clickbench_q26/vortex-file-compressed 209367454 2.34808e+08 0.891656 ns
clickbench_q27/vortex-file-compressed 1067034996 1.1985e+09 0.890307 ns
clickbench_q28/vortex-file-compressed 10758769321 1.03326e+10 1.04124 ns
clickbench_q29/vortex-file-compressed 2201157938 7.53255e+08 2.92219 ns
clickbench_q30/vortex-file-compressed 385963543 3.76248e+08 1.02582 ns
clickbench_q31/vortex-file-compressed 406140206 3.8974e+08 1.04208 ns
clickbench_q32/vortex-file-compressed 2645919154 2.68761e+09 0.984488 ns
clickbench_q33/vortex-file-compressed 2269431248 2.36507e+09 0.959564 ns
clickbench_q34/vortex-file-compressed 2270845518 2.3729e+09 0.956991 ns
clickbench_q35/vortex-file-compressed 967967797 9.09924e+08 1.06379 ns
clickbench_q36/vortex-file-compressed 115041066 1.0252e+08 1.12214 ns
clickbench_q37/vortex-file-compressed 61757435 5.83782e+07 1.05788 ns
clickbench_q38/vortex-file-compressed 54420997 4.29087e+07 1.2683 ns
clickbench_q39/vortex-file-compressed 185080445 1.6197e+08 1.14269 ns
clickbench_q40/vortex-file-compressed 37616534 3.2909e+07 1.14305 ns
clickbench_q41/vortex-file-compressed 34623972 3.10663e+07 1.11452 ns
clickbench_q42/vortex-file-compressed 83205851 5.25286e+07 1.58401 ns

@AdamGS
Copy link
Contributor

AdamGS commented Feb 28, 2025

used to be clearly worse right? @gatesn

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@onursatici @AdamGS