zhangxffff commented on PR #20528:
URL: https://github.com/apache/datafusion/pull/20528#issuecomment-3953066233
### Benchmark result
run with `cargo bench --bench in_list -- "in_list_cols"`
**Int32**: 10-20x speedup across all scenarios. Improvement is greater with
nulls (up to 20x) since the original row-by-row path has higher per-row null
checking overhead.
**Utf8**: 2.3-11x speedup, with larger gains at lower match rates where
vectorized comparison dominates over `or_kleene` merging cost.
```
(zhangxffff) zhangxffff@95d3d60664da ~/W/datafusion (main)> critcmp after
before
group after
before
----- -----
------
in_list_cols/Int32/list=28/match=0%/nulls=0% 1.00 92.6±1.06µs
? ?/sec 9.97 923.4±13.94µs ? ?/sec
in_list_cols/Int32/list=28/match=0%/nulls=20% 1.00 103.7±0.79µs
? ?/sec 17.29 1792.5±15.14µs ? ?/sec
in_list_cols/Int32/list=28/match=100%/nulls=0% 1.00 92.7±1.41µs
? ?/sec 10.26 950.9±10.76µs ? ?/sec
in_list_cols/Int32/list=28/match=100%/nulls=20% 1.00 104.4±1.72µs
? ?/sec 17.28 1804.9±16.47µs ? ?/sec
in_list_cols/Int32/list=28/match=50%/nulls=0% 1.00 92.5±0.67µs
? ?/sec 15.65 1448.1±13.22µs ? ?/sec
in_list_cols/Int32/list=28/match=50%/nulls=20% 1.00 106.2±2.56µs
? ?/sec 19.70 2.1±0.02ms ? ?/sec
in_list_cols/Int32/list=3/match=0%/nulls=0% 1.00 10.1±0.20µs
? ?/sec 9.74 98.6±0.91µs ? ?/sec
in_list_cols/Int32/list=3/match=0%/nulls=20% 1.00 11.2±0.16µs
? ?/sec 16.17 181.9±1.58µs ? ?/sec
in_list_cols/Int32/list=3/match=100%/nulls=0% 1.00 10.2±0.09µs
? ?/sec 9.98 101.7±0.91µs ? ?/sec
in_list_cols/Int32/list=3/match=100%/nulls=20% 1.00 11.3±0.12µs
? ?/sec 16.28 184.0±3.56µs ? ?/sec
in_list_cols/Int32/list=3/match=50%/nulls=0% 1.00 10.1±0.08µs
? ?/sec 14.79 149.4±1.51µs ? ?/sec
in_list_cols/Int32/list=3/match=50%/nulls=20% 1.00 11.2±0.13µs
? ?/sec 18.23 204.1±1.75µs ? ?/sec
in_list_cols/Int32/list=8/match=0%/nulls=0% 1.00 26.7±0.41µs
? ?/sec 9.85 263.5±2.04µs ? ?/sec
in_list_cols/Int32/list=8/match=0%/nulls=20% 1.00 30.2±0.23µs
? ?/sec 16.49 498.3±3.58µs ? ?/sec
in_list_cols/Int32/list=8/match=100%/nulls=0% 1.00 26.6±0.44µs
? ?/sec 10.21 271.9±3.34µs ? ?/sec
in_list_cols/Int32/list=8/match=100%/nulls=20% 1.00 29.7±0.31µs
? ?/sec 17.07 507.8±6.23µs ? ?/sec
in_list_cols/Int32/list=8/match=50%/nulls=0% 1.00 26.8±0.27µs
? ?/sec 15.17 406.1±2.29µs ? ?/sec
in_list_cols/Int32/list=8/match=50%/nulls=20% 1.00 29.9±0.63µs
? ?/sec 19.82 592.9±10.95µs ? ?/sec
in_list_cols/Utf8/list=28/match=0% 1.00 158.5±4.42µs
? ?/sec 10.19 1615.5±10.33µs ? ?/sec
in_list_cols/Utf8/list=28/match=100% 1.00 722.8±11.34µs
? ?/sec 2.29 1655.1±11.99µs ? ?/sec
in_list_cols/Utf8/list=28/match=50% 1.00 1070.4±11.22µs
? ?/sec 2.97 3.2±0.02ms ? ?/sec
in_list_cols/Utf8/list=3/match=0% 1.00 15.5±0.43µs
? ?/sec 11.09 171.6±2.69µs ? ?/sec
in_list_cols/Utf8/list=3/match=100% 1.00 70.0±1.08µs
? ?/sec 2.34 163.4±2.15µs ? ?/sec
in_list_cols/Utf8/list=3/match=50% 1.00 107.5±2.05µs
? ?/sec 2.97 318.9±4.12µs ? ?/sec
in_list_cols/Utf8/list=8/match=0% 1.00 42.4±1.63µs
? ?/sec 10.88 461.3±4.24µs ? ?/sec
in_list_cols/Utf8/list=8/match=100% 1.00 194.9±1.06µs
? ?/sec 2.40 467.1±4.39µs ? ?/sec
in_list_cols/Utf8/list=8/match=50% 1.00 296.0±3.38µs
? ?/sec 3.03 897.1±8.81µs ? ?/sec
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
