lyne7-sc commented on PR #25363:
URL: https://github.com/apache/datafusion/pull/25363#issuecomment-6206826829
@2010YOUY01 I agree with
> The tradeoff is:
ArrayMap: Faster, more memory
This PR: Slower, less memory
My initial thinking was that ArrayMap mainly targets dense build keys,
while the bitmap could complement it for sparse, bounded key ranges with low
probe hit rates. ArrayMap can still handle those cases with adjusted
thresholds, but HashMap with a bitmap could offer much of the benefit over
plain HashMap at a lower memory cost.
Typical examples are TPCH Q8 and Q9: at SF10, the filtered build keys occupy
about 0.67% and 5.44% of their key ranges, and about 99.3% and 94.6% of probe
keys don’t match, respectively. In isolated join tests with both range
thresholds raised to 2M, the bitmap prefilter reduced Q8 execution time from
171 ms to 42 ms with about 0.67 MiB of peak pool reservation, while ArrayMap
took 22 ms with about 8.03 MiB. For Q9, it reduced execution time from 197 ms
to 56 ms with about 4.31 MiB, while ArrayMap took 32 ms with about 9.81 MiB.
ArrayMap was still affordable in these tests, but they illustrate the tradeoff
between performance and memory usage.
I’m also unsure whether this path is worth maintaining, so I don’t have a
strong opinion on reverting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
