Dandandan commented on issue #18489:
URL: https://github.com/apache/datafusion/issues/18489#issuecomment-4026643871
```
Comparing HEAD and parquet-morsel-driven-execution-237164415184908839
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃
parquet-morsel-driven-execution-237164415184908839 ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.57 / 4.76 ±6.28 / 17.32 ms │
1.60 / 4.85 ±6.37 / 17.59 ms │ no change │
│ QQuery 1 │ 15.13 / 15.69 ±0.31 / 16.03 ms │
14.35 / 15.37 ±0.64 / 15.99 ms │ no change │
│ QQuery 2 │ 56.19 / 56.70 ±0.29 / 56.99 ms │
49.74 / 50.96 ±0.79 / 51.90 ms │ +1.11x faster │
│ QQuery 3 │ 48.62 / 50.37 ±1.39 / 51.98 ms │
41.96 / 43.58 ±1.14 / 45.19 ms │ +1.16x faster │
│ QQuery 4 │ 307.41 / 319.09 ±10.72 / 335.84 ms │
268.96 / 276.50 ±4.04 / 280.61 ms │ +1.15x faster │
│ QQuery 5 │ 352.09 / 358.93 ±4.57 / 364.57 ms │
313.74 / 314.84 ±0.79 / 315.93 ms │ +1.14x faster │
│ QQuery 6 │ 10.79 / 11.13 ±0.25 / 11.53 ms │
9.23 / 11.03 ±1.27 / 12.92 ms │ no change │
│ QQuery 7 │ 22.50 / 24.38 ±2.31 / 28.27 ms │
18.92 / 22.86 ±6.99 / 36.83 ms │ +1.07x faster │
│ QQuery 8 │ 438.63 / 452.22 ±11.18 / 463.00 ms │
396.94 / 405.01 ±7.83 / 419.17 ms │ +1.12x faster │
│ QQuery 9 │ 675.49 / 686.53 ±7.98 / 699.07 ms │ 545.69
/ 556.37 ±14.47 / 585.01 ms │ +1.23x faster │
│ QQuery 10 │ 118.77 / 122.00 ±2.85 / 126.75 ms │
104.30 / 107.78 ±4.49 / 116.64 ms │ +1.13x faster │
│ QQuery 11 │ 132.65 / 134.43 ±1.93 / 137.58 ms │
116.12 / 117.47 ±0.84 / 118.61 ms │ +1.14x faster │
│ QQuery 12 │ 377.82 / 385.46 ±4.51 / 389.53 ms │ 346.79
/ 364.18 ±10.20 / 376.44 ms │ +1.06x faster │
│ QQuery 13 │ 489.63 / 496.69 ±4.07 / 501.16 ms │ 484.18
/ 493.86 ±10.31 / 509.42 ms │ no change │
│ QQuery 14 │ 397.72 / 406.41 ±7.98 / 416.57 ms │
361.38 / 367.02 ±5.10 / 374.35 ms │ +1.11x faster │
│ QQuery 15 │ 373.45 / 391.06 ±11.03 / 406.11 ms │ 348.17
/ 363.42 ±15.05 / 389.64 ms │ +1.08x faster │
│ QQuery 16 │ 762.42 / 776.35 ±20.65 / 817.20 ms │
709.93 / 715.67 ±4.23 / 720.47 ms │ +1.08x faster │
│ QQuery 17 │ 739.48 / 750.35 ±8.08 / 760.76 ms │ 700.82
/ 716.52 ±20.33 / 752.94 ms │ no change │
│ QQuery 18 │ 1449.37 / 1529.30 ±45.53 / 1577.29 ms │ 1317.78
/ 1327.87 ±6.04 / 1335.93 ms │ +1.15x faster │
│ QQuery 19 │ 36.23 / 41.52 ±5.37 / 51.72 ms │
32.32 / 34.11 ±2.08 / 37.50 ms │ +1.22x faster │
│ QQuery 20 │ 714.74 / 729.33 ±13.39 / 750.51 ms │ 492.24
/ 564.86 ±56.45 / 622.06 ms │ +1.29x faster │
│ QQuery 21 │ 748.25 / 761.88 ±16.66 / 792.56 ms │ 573.81
/ 595.05 ±11.01 / 602.54 ms │ +1.28x faster │
│ QQuery 22 │ 1170.49 / 1189.87 ±13.85 / 1213.80 ms │ 949.78 /
1000.36 ±29.40 / 1041.18 ms │ +1.19x faster │
│ QQuery 23 │ 386.00 / 393.81 ±5.35 / 401.65 ms │ 254.24
/ 287.79 ±57.36 / 402.36 ms │ +1.37x faster │
│ QQuery 24 │ 114.83 / 118.60 ±2.15 / 121.16 ms │
76.23 / 79.86 ±4.03 / 87.70 ms │ +1.49x faster │
│ QQuery 25 │ 192.39 / 195.89 ±3.48 / 202.16 ms │
159.12 / 160.90 ±1.73 / 164.19 ms │ +1.22x faster │
│ QQuery 26 │ 159.50 / 163.57 ±4.88 / 172.91 ms │
109.77 / 113.62 ±2.26 / 116.40 ms │ +1.44x faster │
│ QQuery 27 │ 982.07 / 999.55 ±18.07 / 1034.41 ms │ 702.45
/ 715.99 ±18.40 / 752.44 ms │ +1.40x faster │
│ QQuery 28 │ 7862.22 / 7932.97 ±38.29 / 7965.93 ms │ 6793.56 /
6912.84 ±97.61 / 7027.74 ms │ +1.15x faster │
│ QQuery 29 │ 328.95 / 370.54 ±56.67 / 476.11 ms │
285.82 / 293.40 ±9.57 / 311.61 ms │ +1.26x faster │
│ QQuery 30 │ 370.40 / 390.37 ±15.00 / 407.11 ms │
342.93 / 354.62 ±9.15 / 367.91 ms │ +1.10x faster │
│ QQuery 31 │ 352.39 / 393.62 ±29.13 / 434.32 ms │
350.73 / 354.69 ±3.35 / 359.53 ms │ +1.11x faster │
│ QQuery 32 │ 1094.79 / 1261.09 ±86.88 / 1325.95 ms │ 1069.30 /
1088.76 ±32.21 / 1152.97 ms │ +1.16x faster │
│ QQuery 33 │ 1516.87 / 1531.79 ±9.94 / 1546.68 ms │ 1531.00 /
1560.80 ±21.19 / 1595.69 ms │ no change │
│ QQuery 34 │ 1511.00 / 1547.46 ±26.61 / 1585.46 ms │ 1519.85
/ 1534.32 ±9.37 / 1547.66 ms │ no change │
│ QQuery 35 │ 403.43 / 420.82 ±13.96 / 443.04 ms │
343.40 / 348.38 ±5.86 / 359.67 ms │ +1.21x faster │
│ QQuery 36 │ 112.43 / 116.93 ±5.52 / 127.75 ms │
74.81 / 80.05 ±3.35 / 84.88 ms │ +1.46x faster │
│ QQuery 37 │ 57.32 / 61.01 ±2.87 / 64.99 ms │
48.72 / 49.67 ±0.78 / 50.63 ms │ +1.23x faster │
│ QQuery 38 │ 53.38 / 57.73 ±3.65 / 62.21 ms │
39.07 / 41.84 ±1.54 / 43.68 ms │ +1.38x faster │
│ QQuery 39 │ 175.06 / 190.16 ±8.43 / 198.75 ms │
121.37 / 133.61 ±6.76 / 141.23 ms │ +1.42x faster │
│ QQuery 40 │ 35.02 / 37.52 ±1.94 / 40.11 ms │
26.64 / 29.15 ±1.35 / 30.39 ms │ +1.29x faster │
│ QQuery 41 │ 30.56 / 32.30 ±1.38 / 34.51 ms │
22.78 / 24.57 ±1.36 / 26.76 ms │ +1.31x faster │
│ QQuery 42 │ 24.00 / 24.72 ±0.37 / 25.07 ms │
18.10 / 19.08 ±1.28 / 21.56 ms │ +1.30x faster │
└───────────┴───────────────────────────────────────┴────────────────────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃
┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │
25934.91ms │
│ Total Time (parquet-morsel-driven-execution-237164415184908839) │
22653.47ms │
│ Average Time (HEAD) │
603.14ms │
│ Average Time (parquet-morsel-driven-execution-237164415184908839) │
526.82ms │
│ Queries Faster │
36 │
│ Queries Slower │
0 │
│ Queries with No Change │
7 │
│ Queries with Failure │
0 │
└───────────────────────────────────────────────────────────────────┴────────────┘
```
Current morsel PR comparing `main + filter pushdown` vs main + `filter
pushdown + 1MB morsels target + split (only) at limited depth`! See:
https://github.com/adriangb/datafusion/pull/6#issuecomment-4026537017
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
