Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a StatsLayout to store logical statistics #2340

Merged
merged 28 commits into from
Feb 19, 2025
Merged

Add a StatsLayout to store logical statistics #2340

merged 28 commits into from
Feb 19, 2025

Conversation

gatesn
Copy link
Contributor

@gatesn gatesn commented Feb 13, 2025 •
edited
Loading

We could call this a ZoneMapLayout?

Anyway, at write-time, we chunk to a logical block size, compute stats, and then re-chunk for actual compression. We may wish to compress the stats tables.

@gatesn gatesn added the benchmark Run benchmarks on this branch label Feb 13, 2025
@github-actions github-actions bot removed the benchmark Run benchmarks on this branch label Feb 13, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 13, 2025 •
edited
Loading

Benchmarks: random_access

Table of Results
name PR b5d1c6d base eadc1fe ratio (PR/base) unit
random-access/vortex-tokio-local-disk 2144745 2183422 0.982286 ns
random-access/parquet-tokio-local-disk 258834298 257845479 1.00383 ns

@gatesn gatesn marked this pull request as draft February 13, 2025 00:07
@gatesn gatesn added the benchmark Run benchmarks on this branch label Feb 13, 2025
@github-actions github-actions bot removed the benchmark Run benchmarks on this branch label Feb 13, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 13, 2025 •
edited
Loading

Benchmarks: TPC-H

Table of Results
name PR 23670ae base ea4a432 ratio (PR/base) unit
tpch_q01/arrow 593051254 6.21445e+08 0.95431 ns
tpch_q01/parquet 805235685 8.45321e+08 0.95258 ns
tpch_q01/vortex-file-compressed 539790040 5.72115e+08 0.943499 ns
tpch_q02/arrow 160693482 1.68019e+08 0.956398 ns
tpch_q02/parquet 206102862 2.07096e+08 0.995206 ns
tpch_q02/vortex-file-compressed 179909631 1.72782e+08 1.04125 ns
tpch_q03/arrow 211523219 2.11644e+08 0.999429 ns
tpch_q03/parquet 421027763 4.31799e+08 0.975054 ns
tpch_q03/vortex-file-compressed 260064109 2.62834e+08 0.98946 ns
tpch_q04/arrow 198724024 2.04501e+08 0.971752 ns
tpch_q04/parquet 244171083 2.45951e+08 0.992764 ns
tpch_q04/vortex-file-compressed 198805869 1.79805e+08 1.10568 ns
tpch_q05/arrow 384797818 3.92326e+08 0.980811 ns
tpch_q05/parquet 581764235 5.94368e+08 0.978794 ns
tpch_q05/vortex-file-compressed 416382107 4.21801e+08 0.987154 ns
tpch_q06/arrow 30288222 2.90074e+07 1.04415 ns
tpch_q06/parquet 157559991 1.58121e+08 0.996454 ns
tpch_q06/vortex-file-compressed 75013668 6.93563e+07 1.08157 ns
tpch_q07/arrow 716147927 7.26018e+08 0.986406 ns
tpch_q07/parquet 877156591 9.08194e+08 0.965825 ns
tpch_q07/vortex-file-compressed 730981112 7.48633e+08 0.976421 ns
tpch_q08/arrow 279975687 2.99146e+08 0.935915 ns
tpch_q08/parquet 587558099 6.2243e+08 0.943975 ns
tpch_q08/vortex-file-compressed 410525358 4.18825e+08 0.980183 ns
tpch_q09/arrow 526030017 5.68291e+08 0.925636 ns
tpch_q09/parquet 836274907 9.00289e+08 0.928896 ns
tpch_q09/vortex-file-compressed 664895507 7.21944e+08 0.920979 ns
tpch_q10/arrow 283843825 3.17258e+08 0.89468 ns
tpch_q10/parquet 530402468 5.69176e+08 0.931878 ns
tpch_q10/vortex-file-compressed 302244126 3.22275e+08 0.937846 ns
tpch_q11/arrow 150306782 1.70246e+08 0.882882 ns
tpch_q11/parquet 168302882 1.83714e+08 0.916116 ns
tpch_q11/vortex-file-compressed 145510791 1.56189e+08 0.93163 ns
tpch_q12/arrow 190528946 2.00357e+08 0.950947 ns
tpch_q12/parquet 338093200 3.51437e+08 0.96203 ns
tpch_q12/vortex-file-compressed 305694246 2.41814e+08 1.26417 ns
tpch_q13/arrow 189316536 2.05722e+08 0.920252 ns
tpch_q13/parquet 330845997 3.66299e+08 0.903212 ns
tpch_q13/vortex-file-compressed 190635860 2.10723e+08 0.904676 ns
tpch_q14/arrow 41209199 4.32234e+07 0.953401 ns
tpch_q14/parquet 242571328 2.49951e+08 0.970477 ns
tpch_q14/vortex-file-compressed 81386909 7.34433e+07 1.10816 ns
tpch_q15/arrow 75458760 8.20303e+07 0.919889 ns
tpch_q15/parquet 340499634 3.50714e+08 0.970875 ns
tpch_q15/vortex-file-compressed 153318523 1.27814e+08 1.19955 ns
tpch_q16/arrow 110744366 1.15716e+08 0.957036 ns
tpch_q16/parquet 128355878 1.31742e+08 0.974296 ns
tpch_q16/vortex-file-compressed 113362050 1.21264e+08 0.934836 ns
tpch_q17/arrow 725387066 7.64839e+08 0.948419 ns
tpch_q17/parquet 748907533 8.02359e+08 0.933382 ns
tpch_q17/vortex-file-compressed 622623944 6.50386e+08 0.957314 ns
tpch_q18/arrow 1441085311 1.42815e+09 1.00906 ns
tpch_q18/parquet 1616749548 1.53281e+09 1.05476 ns
tpch_q18/vortex-file-compressed 1382216420 1.30407e+09 1.05992 ns
tpch_q19/arrow 156210829 1.57162e+08 0.993945 ns
tpch_q19/parquet 434065031 4.33485e+08 1.00134 ns
tpch_q19/vortex-file-compressed 164388263 1.72042e+08 0.955513 ns
tpch_q20/arrow 208376213 1.95093e+08 1.06809 ns
tpch_q20/parquet 355191812 3.37121e+08 1.0536 ns
tpch_q20/vortex-file-compressed 248350762 2.29507e+08 1.08211 ns
tpch_q21/arrow 1146685904 1.09553e+09 1.0467 ns
tpch_q21/parquet 1339877412 1.25642e+09 1.06643 ns
tpch_q21/vortex-file-compressed 1186284446 1.03359e+09 1.14773 ns
tpch_q22/arrow 90766353 8.512e+07 1.06633 ns
tpch_q22/parquet 126142755 1.1931e+08 1.05727 ns
tpch_q22/vortex-file-compressed 93809518 8.95536e+07 1.04752 ns

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 13, 2025 •
edited
Loading

Benchmarks: Clickbench

Table of Results
name PR 23670ae base ea4a432 ratio (PR/base) unit
clickbench_q00/parquet 3316068 2.92511e+06 1.13366 ns
clickbench_q01/parquet 62553517 6.39816e+07 0.97768 ns
clickbench_q02/parquet 124299203 1.30118e+08 0.955284 ns
clickbench_q03/parquet 93467217 9.46784e+07 0.987208 ns
clickbench_q04/parquet 743041987 7.59776e+08 0.977975 ns
clickbench_q05/parquet 783753497 7.6306e+08 1.02712 ns
clickbench_q06/parquet 2688685 2.6703e+06 1.00689 ns
clickbench_q07/parquet 71689559 6.49335e+07 1.10405 ns
clickbench_q08/parquet 818202851 8.07459e+08 1.01331 ns
clickbench_q09/parquet 1143019897 1.12162e+09 1.01908 ns
clickbench_q10/parquet 269926815 2.73696e+08 0.98623 ns
clickbench_q11/parquet 325751467 3.18661e+08 1.02225 ns
clickbench_q12/parquet 784317434 7.76134e+08 1.01054 ns
clickbench_q13/parquet 1021104853 1.04103e+09 0.980861 ns
clickbench_q14/parquet 757740715 7.6432e+08 0.991392 ns
clickbench_q15/parquet 825266328 8.1169e+08 1.01673 ns
clickbench_q16/parquet 1623802995 1.62329e+09 1.00032 ns
clickbench_q17/parquet 1437317274 1.43955e+09 0.998449 ns
clickbench_q18/parquet 3054839705 3.08771e+09 0.989355 ns
clickbench_q19/parquet 70528254 7.46824e+07 0.944376 ns
clickbench_q20/parquet 1136348047 1.14276e+09 0.994393 ns
clickbench_q21/parquet 1323300602 1.38594e+09 0.954801 ns
clickbench_q22/parquet 1966617605 1.96768e+09 0.999459 ns
clickbench_q23/parquet 8205686246 8.29433e+09 0.989312 ns
clickbench_q24/parquet 453750958 4.67335e+08 0.970934 ns
clickbench_q25/parquet 410115651 4.08242e+08 1.00459 ns
clickbench_q26/parquet 501672942 5.09763e+08 0.984129 ns
clickbench_q27/parquet 1542313917 1.66508e+09 0.926268 ns
clickbench_q28/parquet 11685286961 1.15036e+10 1.0158 ns
clickbench_q29/parquet 438295028 4.52566e+08 0.968467 ns
clickbench_q30/parquet 676267742 6.95853e+08 0.971855 ns
clickbench_q31/parquet 715061922 7.32978e+08 0.975557 ns
clickbench_q32/parquet 2864343409 2.91923e+09 0.981197 ns
clickbench_q33/parquet 2973454940 2.98553e+09 0.995954 ns
clickbench_q34/parquet 2958851998 2.96936e+09 0.996462 ns
clickbench_q35/parquet 880935950 9.17983e+08 0.959643 ns
clickbench_q36/parquet 184188288 1.88263e+08 0.978359 ns
clickbench_q37/parquet 84554242 8.57514e+07 0.986039 ns
clickbench_q38/parquet 111384019 1.13197e+08 0.983981 ns
clickbench_q39/parquet 323456096 3.3445e+08 0.967128 ns
clickbench_q40/parquet 53528905 5.3761e+07 0.995684 ns
clickbench_q41/parquet 52521666 5.09098e+07 1.03166 ns
clickbench_q42/parquet 70174890 7.17079e+07 0.978622 ns
clickbench_q00/vortex-file-compressed 2233942 2.39105e+06 0.934293 ns
clickbench_q01/vortex-file-compressed 26235098 2.51962e+07 1.04123 ns
clickbench_q02/vortex-file-compressed 62402405 6.39632e+07 0.975598 ns
clickbench_q03/vortex-file-compressed 64804066 6.89439e+07 0.939953 ns
clickbench_q04/vortex-file-compressed 634320160 6.62816e+08 0.957008 ns
clickbench_q05/vortex-file-compressed 690950282 6.86514e+08 1.00646 ns
clickbench_q06/vortex-file-compressed 2368939 2.31731e+06 1.02228 ns
clickbench_q07/vortex-file-compressed 36896725 3.28706e+07 1.12249 ns
clickbench_q08/vortex-file-compressed 743924078 7.60505e+08 0.978197 ns
clickbench_q09/vortex-file-compressed 889157975 9.27242e+08 0.958928 ns
clickbench_q10/vortex-file-compressed 211759162 2.19459e+08 0.964912 ns
clickbench_q11/vortex-file-compressed 232341933 2.38293e+08 0.975026 ns
clickbench_q12/vortex-file-compressed 591181483 6.18063e+08 0.956507 ns
clickbench_q13/vortex-file-compressed 806784257 8.47742e+08 0.951686 ns
clickbench_q14/vortex-file-compressed 542392668 5.62474e+08 0.964297 ns
clickbench_q15/vortex-file-compressed 745030082 7.77448e+08 0.958302 ns
clickbench_q16/vortex-file-compressed 1489334863 1.56002e+09 0.954687 ns
clickbench_q17/vortex-file-compressed 1455938289 1.46213e+09 0.995768 ns
clickbench_q18/vortex-file-compressed 3017636093 2.98523e+09 1.01086 ns
clickbench_q19/vortex-file-compressed 53160257 4.7201e+07 1.12625 ns
clickbench_q20/vortex-file-compressed 1009564170 9.41814e+08 1.07194 ns
clickbench_q21/vortex-file-compressed 1141797022 1.04108e+09 1.09674 ns
clickbench_q22/vortex-file-compressed 1460292381 1.50299e+09 0.97159 ns
clickbench_q23/vortex-file-compressed 2195767616 2.19576e+09 1.00001 ns
clickbench_q24/vortex-file-compressed 280487658 3.04656e+08 0.92067 ns
clickbench_q25/vortex-file-compressed 281309944 3.29988e+08 0.852487 ns
clickbench_q26/vortex-file-compressed 352457834 3.82302e+08 0.921937 ns
clickbench_q27/vortex-file-compressed 1493142253 1.87181e+09 0.797698 ns
clickbench_q28/vortex-file-compressed 10900374532 1.09995e+10 0.990989 ns
clickbench_q29/vortex-file-compressed 691408699 6.8972e+08 1.00245 ns
clickbench_q30/vortex-file-compressed 447927223 4.86278e+08 0.921134 ns
clickbench_q31/vortex-file-compressed 470440420 5.10089e+08 0.922271 ns
clickbench_q32/vortex-file-compressed 2845958998 3.05259e+09 0.93231 ns
clickbench_q33/vortex-file-compressed 2664495558 2.60214e+09 1.02396 ns
clickbench_q34/vortex-file-compressed 2653825018 2.61944e+09 1.01313 ns
clickbench_q35/vortex-file-compressed 1050846072 9.95339e+08 1.05577 ns
clickbench_q36/vortex-file-compressed 200842545 2.46507e+08 0.814755 ns
clickbench_q37/vortex-file-compressed 83898837 1.32657e+08 0.632451 ns
clickbench_q38/vortex-file-compressed 120621985 1.13219e+08 1.06539 ns
clickbench_q39/vortex-file-compressed 293947391 2.82289e+08 1.0413 ns
clickbench_q40/vortex-file-compressed 40251863 4.81755e+07 0.835526 ns
clickbench_q41/vortex-file-compressed 56367341 5.95699e+07 0.946238 ns
clickbench_q42/vortex-file-compressed 62369889 6.78801e+07 0.918825 ns

@gatesn gatesn added the benchmark Run benchmarks on this branch label Feb 13, 2025
@github-actions github-actions bot removed the benchmark Run benchmarks on this branch label Feb 13, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 13, 2025 •
edited
Loading

Benchmarks: compress

Table of Results
name PR b5d1c6d base eadc1fe ratio (PR/base) unit
compress time/wide table cols=10 chunks=1 rows=1000 3.73195e+06 3.52041e+06 1.06009 ns
compress time/wide table cols=10 chunks=1 rows=1000 throughput 0.0321928 0.0341273 0.943315 bytes/ns
parquet_rs-zstd compress time/wide table cols=10 chunks=1 rows=1000 727259 728258 0.998628 ns
parquet_rs-zstd compress time/wide table cols=10 chunks=1 rows=1000 throughput 0.165198 0.164972 1.00137 bytes/ns
decompress time/wide table cols=10 chunks=1 rows=1000 222895 211082 1.05597 ns
decompress time/wide table cols=10 chunks=1 rows=1000 throughput 0.539007 0.569173 0.946999 bytes/ns
parquet_rs-zstd decompress time/wide table cols=10 chunks=1 rows=1000 244468 244139 1.00135 ns
parquet_rs-zstd decompress time/wide table cols=10 chunks=1 rows=1000 throughput 0.491442 0.492104 0.998653 bytes/ns
compress time/wide table cols=100 chunks=1 rows=1000 3.79653e+07 3.57596e+07 1.06168 ns
compress time/wide table cols=100 chunks=1 rows=1000 throughput 0.03164 0.0335916 0.941902 bytes/ns
parquet_rs-zstd compress time/wide table cols=100 chunks=1 rows=1000 8.4459e+06 7.83312e+06 1.07823 ns
parquet_rs-zstd compress time/wide table cols=100 chunks=1 rows=1000 throughput 0.142225 0.153352 0.927446 bytes/ns
decompress time/wide table cols=100 chunks=1 rows=1000 1.64912e+06 1.50498e+06 1.09578 ns
decompress time/wide table cols=100 chunks=1 rows=1000 throughput 0.7284 0.798163 0.912596 bytes/ns
parquet_rs-zstd decompress time/wide table cols=100 chunks=1 rows=1000 2.51735e+06 2.55114e+06 0.986758 ns
parquet_rs-zstd decompress time/wide table cols=100 chunks=1 rows=1000 throughput 0.477177 0.470858 1.01342 bytes/ns
compress time/wide table cols=1000 chunks=1 rows=1000 4.0439e+08 3.65085e+08 1.10766 ns
compress time/wide table cols=1000 chunks=1 rows=1000 throughput 0.0297041 0.032902 0.902803 bytes/ns
parquet_rs-zstd compress time/wide table cols=1000 chunks=1 rows=1000 1.02935e+08 9.53353e+07 1.07971 ns
parquet_rs-zstd compress time/wide table cols=1000 chunks=1 rows=1000 throughput 0.116695 0.125998 0.926172 bytes/ns
decompress time/wide table cols=1000 chunks=1 rows=1000 2.12217e+07 1.89559e+07 1.11953 ns
decompress time/wide table cols=1000 chunks=1 rows=1000 throughput 0.566026 0.633684 0.893231 bytes/ns
parquet_rs-zstd decompress time/wide table cols=1000 chunks=1 rows=1000 3.29817e+07 2.90148e+07 1.13672 ns
parquet_rs-zstd decompress time/wide table cols=1000 chunks=1 rows=1000 throughput 0.364202 0.413996 0.879724 bytes/ns
compress time/wide table cols=10 chunks=50 rows=1000 7.59214e+07 7.23592e+07 1.04923 ns
compress time/wide table cols=10 chunks=50 rows=1000 throughput 0.00166571 0.00174771 0.95308 bytes/ns
parquet_rs-zstd compress time/wide table cols=10 chunks=50 rows=1000 1.11016e+06 1.00415e+06 1.10558 ns
parquet_rs-zstd compress time/wide table cols=10 chunks=50 rows=1000 throughput 0.113914 0.125941 0.904507 bytes/ns
decompress time/wide table cols=10 chunks=50 rows=1000 961623 926964 1.03739 ns
decompress time/wide table cols=10 chunks=50 rows=1000 throughput 0.13151 0.136427 0.963957 bytes/ns
parquet_rs-zstd decompress time/wide table cols=10 chunks=50 rows=1000 252468 251647 1.00326 ns
parquet_rs-zstd decompress time/wide table cols=10 chunks=50 rows=1000 throughput 0.500908 0.502542 0.996748 bytes/ns
compress time/wide table cols=100 chunks=50 rows=1000 7.46324e+08 7.17341e+08 1.0404 ns
compress time/wide table cols=100 chunks=50 rows=1000 throughput 0.00168889 0.00175713 0.961166 bytes/ns
parquet_rs-zstd compress time/wide table cols=100 chunks=50 rows=1000 1.12945e+07 1.37699e+07 0.820229 ns
parquet_rs-zstd compress time/wide table cols=100 chunks=50 rows=1000 throughput 0.1116 0.0915374 1.21917 bytes/ns
decompress time/wide table cols=100 chunks=50 rows=1000 8.10826e+06 8.3485e+06 0.971223 ns
decompress time/wide table cols=100 chunks=50 rows=1000 throughput 0.155454 0.150981 1.02963 bytes/ns
parquet_rs-zstd decompress time/wide table cols=100 chunks=50 rows=1000 2.52804e+06 2.60716e+06 0.969653 ns
parquet_rs-zstd decompress time/wide table cols=100 chunks=50 rows=1000 throughput 0.498593 0.483463 1.0313 bytes/ns
compress time/wide table cols=1000 chunks=50 rows=1000 7.56922e+09 7.4803e+09 1.01189 ns
compress time/wide table cols=1000 chunks=50 rows=1000 throughput 0.0016647 0.00168449 0.988253 bytes/ns
parquet_rs-zstd compress time/wide table cols=1000 chunks=50 rows=1000 1.56323e+08 1.76149e+08 0.887447 ns
parquet_rs-zstd compress time/wide table cols=1000 chunks=50 rows=1000 throughput 0.0806055 0.0715331 1.12683 bytes/ns
decompress time/wide table cols=1000 chunks=50 rows=1000 8.61609e+07 8.79942e+07 0.979166 ns
decompress time/wide table cols=1000 chunks=50 rows=1000 throughput 0.146243 0.143197 1.02128 bytes/ns
parquet_rs-zstd decompress time/wide table cols=1000 chunks=50 rows=1000 2.74613e+07 2.99594e+07 0.916615 ns
parquet_rs-zstd decompress time/wide table cols=1000 chunks=50 rows=1000 throughput 0.458845 0.420585 1.09097 bytes/ns
compress time/taxi 1.94716e+09 1.56359e+09 1.24532 ns
compress time/taxi throughput 0.241793 0.301109 0.803008 bytes/ns
parquet_rs-zstd compress time/taxi 1.87182e+09 1.86707e+09 1.00254 ns
parquet_rs-zstd compress time/taxi throughput 0.251525 0.252165 0.997462 bytes/ns
decompress time/taxi 1.59734e+08 1.6355e+08 0.976669 ns
decompress time/taxi throughput 2.94746 2.8787 1.02389 bytes/ns
parquet_rs-zstd decompress time/taxi 2.92988e+08 3.07035e+08 0.954249 ns
parquet_rs-zstd decompress time/taxi throughput 1.60693 1.53341 1.04794 bytes/ns
compress time/AirlineSentiment 187176 157498 1.18844 ns
compress time/AirlineSentiment throughput 0.0108988 0.0129526 0.84144 bytes/ns
parquet_rs-zstd compress time/AirlineSentiment 47981.9 48140.2 0.996711 ns
parquet_rs-zstd compress time/AirlineSentiment throughput 0.042516 0.0423762 1.0033 bytes/ns
decompress time/AirlineSentiment 89678.2 87122 1.02934 ns
decompress time/AirlineSentiment throughput 0.022748 0.0234154 0.971496 bytes/ns
parquet_rs-zstd decompress time/AirlineSentiment 28552.8 28639.8 0.996961 ns
parquet_rs-zstd decompress time/AirlineSentiment throughput 0.0714466 0.0712295 1.00305 bytes/ns
compress time/Arade 3.72689e+09 3.0993e+09 1.20249 ns
compress time/Arade throughput 0.211176 0.253938 0.831605 bytes/ns
parquet_rs-zstd compress time/Arade 3.46707e+09 3.46093e+09 1.00177 ns
parquet_rs-zstd compress time/Arade throughput 0.227001 0.227404 0.998231 bytes/ns
decompress time/Arade 4.62987e+08 4.58891e+08 1.00893 ns
decompress time/Arade throughput 1.69989 1.71507 0.991153 bytes/ns
parquet_rs-zstd decompress time/Arade 6.2643e+08 6.31023e+08 0.992722 ns
parquet_rs-zstd decompress time/Arade throughput 1.25637 1.24723 1.00733 bytes/ns
compress time/Bimbo 1.35977e+10 8.77594e+09 1.54943 ns
compress time/Bimbo throughput 0.523715 0.811462 0.645397 bytes/ns
parquet_rs-zstd compress time/Bimbo 2.24026e+10 2.29276e+10 0.977102 ns
parquet_rs-zstd compress time/Bimbo throughput 0.317881 0.310602 1.02343 bytes/ns
decompress time/Bimbo 3.52221e+09 3.63191e+09 0.969796 ns
decompress time/Bimbo throughput 2.02184 1.96077 1.03114 bytes/ns
parquet_rs-zstd decompress time/Bimbo 2.54364e+09 2.50665e+09 1.01475 ns
parquet_rs-zstd decompress time/Bimbo throughput 2.79967 2.84098 0.98546 bytes/ns
compress time/CMSprovider 2.80131e+10 2.27466e+10 1.23153 ns
compress time/CMSprovider throughput 0.183812 0.22637 0.811999 bytes/ns
parquet_rs-zstd compress time/CMSprovider 1.89049e+10 1.87374e+10 1.00894 ns
parquet_rs-zstd compress time/CMSprovider throughput 0.272371 0.274807 0.991136 bytes/ns
decompress time/CMSprovider 5.59574e+09 5.56868e+09 1.00486 ns
decompress time/CMSprovider throughput 0.920192 0.924664 0.995163 bytes/ns
parquet_rs-zstd decompress time/CMSprovider 3.34843e+09 3.41387e+09 0.980832 ns
parquet_rs-zstd decompress time/CMSprovider throughput 1.53778 1.50831 1.01954 bytes/ns
compress time/Euro2016 2.34211e+09 2.19046e+09 1.06923 ns
compress time/Euro2016 throughput 0.167906 0.17953 0.93525 bytes/ns
parquet_rs-zstd compress time/Euro2016 1.74322e+09 1.77578e+09 0.981662 ns
parquet_rs-zstd compress time/Euro2016 throughput 0.225591 0.221454 1.01868 bytes/ns
decompress time/Euro2016 3.57239e+08 4.05426e+08 0.881143 ns
decompress time/Euro2016 throughput 1.10082 0.969979 1.13489 bytes/ns
parquet_rs-zstd decompress time/Euro2016 4.502e+08 4.7025e+08 0.957363 ns
parquet_rs-zstd decompress time/Euro2016 throughput 0.873512 0.836268 1.04454 bytes/ns
compress time/Food 1.1115e+09 8.89961e+08 1.24893 ns
compress time/Food throughput 0.299344 0.373859 0.800688 bytes/ns
parquet_rs-zstd compress time/Food 1.1432e+09 1.16e+09 0.985518 ns
parquet_rs-zstd compress time/Food throughput 0.291044 0.286829 1.01469 bytes/ns
decompress time/Food 1.44591e+08 1.44102e+08 1.00339 ns
decompress time/Food throughput 2.30112 2.30891 0.996624 bytes/ns
parquet_rs-zstd decompress time/Food 2.16326e+08 2.15325e+08 1.00465 ns
parquet_rs-zstd decompress time/Food throughput 1.53805 1.5452 0.995371 bytes/ns
compress time/HashTags 4.16043e+09 3.81344e+09 1.09099 ns
compress time/HashTags throughput 0.193369 0.210964 0.916597 bytes/ns
parquet_rs-zstd compress time/HashTags 2.77629e+09 2.80302e+09 0.990465 ns
parquet_rs-zstd compress time/HashTags throughput 0.289775 0.287012 1.00963 bytes/ns
decompress time/HashTags 1.09286e+09 1.0089e+09 1.08322 ns
decompress time/HashTags throughput 0.736141 0.797404 0.923172 bytes/ns
parquet_rs-zstd decompress time/HashTags 7.18268e+08 7.46944e+08 0.961608 ns
parquet_rs-zstd decompress time/HashTags throughput 1.12005 1.07705 1.03992 bytes/ns
compress time/TPC-H l_comment chunked 1.18416e+09 1.02073e+09 1.16012 ns
compress time/TPC-H l_comment chunked throughput 0.210443 0.244138 0.861983 bytes/ns
parquet_rs-zstd compress time/TPC-H l_comment chunked 9.50166e+08 9.58954e+08 0.990836 ns
parquet_rs-zstd compress time/TPC-H l_comment chunked throughput 0.262268 0.259865 1.00925 bytes/ns
decompress time/TPC-H l_comment chunked 9.41055e+07 1.03933e+08 0.905447 ns
decompress time/TPC-H l_comment chunked throughput 2.64808 2.39769 1.10443 bytes/ns
parquet_rs-zstd decompress time/TPC-H l_comment chunked 2.42782e+08 2.47788e+08 0.979795 ns
parquet_rs-zstd decompress time/TPC-H l_comment chunked throughput 1.02643 1.00569 1.02062 bytes/ns
compress time/TPC-H l_comment canonical 1.14588e+09 1.06452e+09 1.07643 ns
compress time/TPC-H l_comment canonical throughput 0.217472 0.234094 0.928998 bytes/ns
parquet_rs-zstd compress time/TPC-H l_comment canonical 9.62282e+08 9.86183e+08 0.975764 ns
parquet_rs-zstd compress time/TPC-H l_comment canonical throughput 0.258966 0.252689 1.02484 bytes/ns
decompress time/TPC-H l_comment canonical 9.4342e+07 1.07418e+08 0.878267 ns
decompress time/TPC-H l_comment canonical throughput 2.64143 2.31988 1.13861 bytes/ns
parquet_rs-zstd decompress time/TPC-H l_comment canonical 2.41487e+08 2.49336e+08 0.968519 ns
parquet_rs-zstd decompress time/TPC-H l_comment canonical throughput 1.03193 0.999445 1.0325 bytes/ns

@gatesn gatesn marked this pull request as ready for review February 14, 2025 20:07
@gatesn gatesn enabled auto-merge (squash) February 14, 2025 20:07
@gatesn gatesn requested a review from lwwmanning February 14, 2025 20:26
@cloudflare-workers-and-pages Cloudflare Workers and Pages
Copy link

cloudflare-workers-and-pages bot commented Feb 19, 2025 •
edited
Loading

Deploying vortex-bench with  Cloudflare Pages  Cloudflare Pages

Latest commit: 4bb7595
Status: ✅  Deploy successful!
Preview URL: https://d6fcdc52.vortex-bench.pages.dev
Branch Preview URL: https://ngates-stats-layout.vortex-bench.pages.dev

View logs

@gatesn gatesn added the benchmark Run benchmarks on this branch label Feb 19, 2025
@github-actions github-actions bot removed the benchmark Run benchmarks on this branch label Feb 19, 2025
@github-actions GitHub Actions
Copy link
Contributor

Benchmarks: TPC-H on S3

Table of Results
name PR b5d1c6d base eadc1fe ratio (PR/base) unit
tpch_q01/parquet 307581189 306762527 1.00267 ns
tpch_q02/parquet 760973742 767966029 0.990895 ns
tpch_q03/parquet 454894139 461688757 0.985283 ns
tpch_q04/parquet 261069247 232368956 1.12351 ns
tpch_q05/parquet 623824421 632304314 0.986589 ns
tpch_q06/parquet 196305317 219563511 0.894071 ns
tpch_q07/parquet 703665136 727373088 0.967406 ns
tpch_q08/parquet 861543381 838966847 1.02691 ns
tpch_q09/parquet 757266539 746303606 1.01469 ns
tpch_q10/parquet 584503089 584611636 0.999814 ns
tpch_q11/parquet 293666681 293045748 1.00212 ns
tpch_q12/parquet 299921372 289912966 1.03452 ns
tpch_q13/parquet 419493282 414731144 1.01148 ns
tpch_q14/parquet 286734936 276711524 1.03622 ns
tpch_q15/parquet 492172751 513801527 0.957904 ns
tpch_q16/parquet 268746774 297945928 0.901998 ns
tpch_q17/parquet 440631394 431460181 1.02126 ns
tpch_q18/parquet 623003293 609522035 1.02212 ns
tpch_q19/parquet 320742859 311647320 1.02919 ns
tpch_q20/parquet 544717793 536051879 1.01617 ns
tpch_q21/parquet 701349785 708410428 0.990033 ns
tpch_q22/parquet 299005744 330593375 0.904452 ns
tpch_q01/vortex-file-compressed 460057511 502087257 0.91629 ns
tpch_q02/vortex-file-compressed 428293227 407987523 1.04977 ns
tpch_q03/vortex-file-compressed 571974054 630083406 0.907775 ns
tpch_q04/vortex-file-compressed 612422267 545822007 1.12202 ns
tpch_q05/vortex-file-compressed 496195155 504333630 0.983863 ns
tpch_q06/vortex-file-compressed 514914156 561421119 0.917162 ns
tpch_q07/vortex-file-compressed 702375811 734952248 0.955675 ns
tpch_q08/vortex-file-compressed 667777410 641481766 1.04099 ns
tpch_q09/vortex-file-compressed 600717105 556722066 1.07903 ns
tpch_q10/vortex-file-compressed 478435217 560369310 0.853786 ns
tpch_q11/vortex-file-compressed 150144922 153641149 0.977244 ns
tpch_q12/vortex-file-compressed 931613507 954719055 0.975799 ns
tpch_q13/vortex-file-compressed 129870552 178836058 0.726199 ns
tpch_q14/vortex-file-compressed 415448354 463707815 0.895927 ns
tpch_q15/vortex-file-compressed 947689815 1019408425 0.929647 ns
tpch_q16/vortex-file-compressed 209319834 195976533 1.06809 ns
tpch_q17/vortex-file-compressed 401909627 433508991 0.927108 ns
tpch_q18/vortex-file-compressed 414078252 570702137 0.725559 ns
tpch_q19/vortex-file-compressed 434109345 473764115 0.916298 ns
tpch_q20/vortex-file-compressed 616537043 691380335 0.891748 ns
tpch_q21/vortex-file-compressed 1440878457 1315527166 1.09529 ns
tpch_q22/vortex-file-compressed 160597623 173360384 0.92638 ns

@gatesn gatesn requested a review from robert3005 February 19, 2025 10:25
@github-actions GitHub Actions
Copy link
Contributor

Benchmarks: Clickbench on NVME

Table of Results
name PR b5d1c6d base eadc1fe ratio (PR/base) unit
clickbench_q00/parquet 2162026 3356243 0.64418 ns
clickbench_q01/parquet 63659443 65629982 0.969975 ns
clickbench_q02/parquet 121519339 125341261 0.969508 ns
clickbench_q03/parquet 86292988 87306958 0.988386 ns
clickbench_q04/parquet 629242872 644653864 0.976094 ns
clickbench_q05/parquet 716510988 703857873 1.01798 ns
clickbench_q06/parquet 2194707 2180810 1.00637 ns
clickbench_q07/parquet 63934799 62880829 1.01676 ns
clickbench_q08/parquet 711335148 734810445 0.968053 ns
clickbench_q09/parquet 999899594 1008482008 0.99149 ns
clickbench_q10/parquet 253197378 254044807 0.996664 ns
clickbench_q11/parquet 305905428 301600271 1.01427 ns
clickbench_q12/parquet 732100046 717871095 1.01982 ns
clickbench_q13/parquet 953625930 988134415 0.965077 ns
clickbench_q14/parquet 697435878 711014555 0.980902 ns
clickbench_q15/parquet 722436332 719532806 1.00404 ns
clickbench_q16/parquet 1560499394 1468502471 1.06265 ns
clickbench_q17/parquet 1392763793 1373562148 1.01398 ns
clickbench_q18/parquet 2934072262 2863530350 1.02463 ns
clickbench_q19/parquet 67384488 66085783 1.01965 ns
clickbench_q20/parquet 1163602448 1140057168 1.02065 ns
clickbench_q21/parquet 1265418513 1226486387 1.03174 ns
clickbench_q22/parquet 1885018782 1867132940 1.00958 ns
clickbench_q23/parquet 7748501627 7713324227 1.00456 ns
clickbench_q24/parquet 428132229 432900653 0.988985 ns
clickbench_q25/parquet 390608234 387244870 1.00869 ns
clickbench_q26/parquet 485416449 479859334 1.01158 ns
clickbench_q27/parquet 1554764037 1551766119 1.00193 ns
clickbench_q28/parquet 11324176569 11245315580 1.00701 ns
clickbench_q29/parquet 431342538 436814811 0.987472 ns
clickbench_q30/parquet 658136156 668337012 0.984737 ns
clickbench_q31/parquet 695620869 695224338 1.00057 ns
clickbench_q32/parquet 2712807918 2752520494 0.985572 ns
clickbench_q33/parquet 2692487252 2783411099 0.967334 ns
clickbench_q34/parquet 2644280158 2749903405 0.96159 ns
clickbench_q35/parquet 824197582 860262924 0.958076 ns
clickbench_q36/parquet 174200472 171911928 1.01331 ns
clickbench_q37/parquet 82417626 80377642 1.02538 ns
clickbench_q38/parquet 107228629 105412746 1.01723 ns
clickbench_q39/parquet 309405960 314260512 0.984552 ns
clickbench_q40/parquet 62226127 51417288 1.21022 ns
clickbench_q41/parquet 54256839 48452517 1.11979 ns
clickbench_q42/parquet 76021487 68375269 1.11183 ns
clickbench_q00/vortex-file-compressed 2152765 2280943 0.943805 ns
clickbench_q01/vortex-file-compressed 24840078 22364345 1.1107 ns
clickbench_q02/vortex-file-compressed 53694658 54854314 0.978859 ns
clickbench_q03/vortex-file-compressed 61489271 60920718 1.00933 ns
clickbench_q04/vortex-file-compressed 596448643 592881456 1.00602 ns
clickbench_q05/vortex-file-compressed 613733019 608793274 1.00811 ns
clickbench_q06/vortex-file-compressed 2252372 2258552 0.997264 ns
clickbench_q07/vortex-file-compressed 32738069 29730345 1.10117 ns
clickbench_q08/vortex-file-compressed 678784217 691158343 0.982097 ns
clickbench_q09/vortex-file-compressed 843038509 832836063 1.01225 ns
clickbench_q10/vortex-file-compressed 153544062 154454704 0.994104 ns
clickbench_q11/vortex-file-compressed 178544202 179114735 0.996815 ns
clickbench_q12/vortex-file-compressed 502058454 500818845 1.00248 ns
clickbench_q13/vortex-file-compressed 747573675 751696613 0.994515 ns
clickbench_q14/vortex-file-compressed 482712567 483360950 0.998659 ns
clickbench_q15/vortex-file-compressed 678453701 687543310 0.98678 ns
clickbench_q16/vortex-file-compressed 1403005114 1338205075 1.04842 ns
clickbench_q17/vortex-file-compressed 1311177819 1317019923 0.995564 ns
clickbench_q18/vortex-file-compressed 2764193913 2748763245 1.00561 ns
clickbench_q19/vortex-file-compressed 43907242 46041096 0.953653 ns
clickbench_q20/vortex-file-compressed 811050240 804090332 1.00866 ns
clickbench_q21/vortex-file-compressed 915374998 896275940 1.02131 ns
clickbench_q22/vortex-file-compressed 1228971549 1300546764 0.944965 ns
clickbench_q23/vortex-file-compressed 1907355840 1970782988 0.967816 ns
clickbench_q24/vortex-file-compressed 238474480 235969507 1.01062 ns
clickbench_q25/vortex-file-compressed 231773991 228816776 1.01292 ns
clickbench_q26/vortex-file-compressed 288995083 287276015 1.00598 ns
clickbench_q27/vortex-file-compressed 1319192755 1289463449 1.02306 ns
clickbench_q28/vortex-file-compressed 10416839252 10381969948 1.00336 ns
clickbench_q29/vortex-file-compressed 682308738 689012401 0.990271 ns
clickbench_q30/vortex-file-compressed 447370003 432219093 1.03505 ns
clickbench_q31/vortex-file-compressed 490047972 454133109 1.07908 ns
clickbench_q32/vortex-file-compressed 2943035342 2641281951 1.11425 ns
clickbench_q33/vortex-file-compressed 2416225069 2363451753 1.02233 ns
clickbench_q34/vortex-file-compressed 2439627907 2394575768 1.01881 ns
clickbench_q35/vortex-file-compressed 965044099 905696861 1.06553 ns
clickbench_q36/vortex-file-compressed 100702317 97735252 1.03036 ns
clickbench_q37/vortex-file-compressed 56486190 56615932 0.997708 ns
clickbench_q38/vortex-file-compressed 78019904 35805544 2.17899 ns
clickbench_q39/vortex-file-compressed 162384973 165081269 0.983667 ns
clickbench_q40/vortex-file-compressed 27837939 32762212 0.849697 ns
clickbench_q41/vortex-file-compressed 29386127 29402161 0.999455 ns
clickbench_q42/vortex-file-compressed 47804445 49277767 0.970102 ns

@github-actions GitHub Actions
Copy link
Contributor

Benchmarks: TPC-H on NVME

Table of Results
name PR b5d1c6d base eadc1fe ratio (PR/base) unit
tpch_q01/arrow 79440410 86098070 0.922674 ns
tpch_q02/arrow 41314100 42196315 0.979093 ns
tpch_q03/arrow 36081400 34532796 1.04484 ns
tpch_q04/arrow 32127924 31783510 1.01084 ns
tpch_q05/arrow 59571316 58509131 1.01815 ns
tpch_q06/arrow 8682871 8292982 1.04701 ns
tpch_q07/arrow 101337599 100567462 1.00766 ns
tpch_q08/arrow 58822988 58574511 1.00424 ns
tpch_q09/arrow 86198524 85698147 1.00584 ns
tpch_q10/arrow 55506799 53122666 1.04488 ns
tpch_q11/arrow 26678824 25422826 1.0494 ns
tpch_q12/arrow 32293981 32325898 0.999013 ns
tpch_q13/arrow 25215514 25046890 1.00673 ns
tpch_q14/arrow 12518470 12133295 1.03175 ns
tpch_q15/arrow 26161433 25774212 1.01502 ns
tpch_q16/arrow 22020201 21520697 1.02321 ns
tpch_q17/arrow 80943754 82092017 0.986012 ns
tpch_q18/arrow 166886044 157818633 1.05745 ns
tpch_q19/arrow 26904434 25650796 1.04887 ns
tpch_q20/arrow 38107228 36323847 1.0491 ns
tpch_q21/arrow 160595514 164589644 0.975733 ns
tpch_q22/arrow 17313072 18528379 0.934408 ns
tpch_q01/parquet 144577522 154762109 0.934192 ns
tpch_q02/parquet 105795515 95681507 1.1057 ns
tpch_q03/parquet 117094478 108572811 1.07849 ns
tpch_q04/parquet 65736775 59992976 1.09574 ns
tpch_q05/parquet 118550905 113467177 1.0448 ns
tpch_q06/parquet 29232553 28751101 1.01675 ns
tpch_q07/parquet 150947460 146489063 1.03044 ns
tpch_q08/parquet 151620500 154237321 0.983034 ns
tpch_q09/parquet 203047670 206855146 0.981594 ns
tpch_q10/parquet 153629259 156587657 0.981107 ns
tpch_q11/parquet 47535951 47209387 1.00692 ns
tpch_q12/parquet 98144754 99926830 0.982166 ns
tpch_q13/parquet 166679081 180255504 0.924682 ns
tpch_q14/parquet 48621336 50262564 0.967347 ns
tpch_q15/parquet 85921891 88155075 0.974668 ns
tpch_q16/parquet 44608601 45455002 0.981379 ns
tpch_q17/parquet 162652519 149718037 1.08639 ns
tpch_q18/parquet 247143367 251092706 0.984271 ns
tpch_q19/parquet 90941586 82952621 1.09631 ns
tpch_q20/parquet 113583655 99757278 1.1386 ns
tpch_q21/parquet 211787999 202567166 1.04552 ns
tpch_q22/parquet 50731794 51267719 0.989547 ns
tpch_q01/vortex-file-compressed 66370597 75361764 0.880693 ns
tpch_q02/vortex-file-compressed 45727287 54214421 0.843452 ns
tpch_q03/vortex-file-compressed 40532814 43786691 0.925688 ns
tpch_q04/vortex-file-compressed 30708839 30685479 1.00076 ns
tpch_q05/vortex-file-compressed 58154175 60963987 0.95391 ns
tpch_q06/vortex-file-compressed 18343986 17056533 1.07548 ns
tpch_q07/vortex-file-compressed 96543684 92556065 1.04308 ns
tpch_q08/vortex-file-compressed 66748989 64430597 1.03598 ns
tpch_q09/vortex-file-compressed 92212402 90059666 1.0239 ns
tpch_q10/vortex-file-compressed 58167251 56582202 1.02801 ns
tpch_q11/vortex-file-compressed 24046000 23558896 1.02068 ns
tpch_q12/vortex-file-compressed 48316725 41850069 1.15452 ns
tpch_q13/vortex-file-compressed 28743586 29420977 0.976976 ns
tpch_q14/vortex-file-compressed 21762514 20303605 1.07185 ns
tpch_q15/vortex-file-compressed 40212741 37060565 1.08505 ns
tpch_q16/vortex-file-compressed 29270585 26051782 1.12355 ns
tpch_q17/vortex-file-compressed 80174298 77763481 1.031 ns
tpch_q18/vortex-file-compressed 146585550 144247362 1.01621 ns
tpch_q19/vortex-file-compressed 32494108 34806401 0.933567 ns
tpch_q20/vortex-file-compressed 43769203 41661558 1.05059 ns
tpch_q21/vortex-file-compressed 137122558 131538471 1.04245 ns
tpch_q22/vortex-file-compressed 28626719 28015212 1.02183 ns

@gatesn gatesn merged commit 233e6c4 into develop Feb 19, 2025
34 of 35 checks passed
@gatesn gatesn deleted the ngates/stats-layout branch February 19, 2025 12:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@a10y a10y a10y left review comments

@robert3005 robert3005 robert3005 approved these changes

@lwwmanning lwwmanning Awaiting requested review from lwwmanning

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@gatesn @a10y @robert3005 @lwwmanning