2010YOUY01 merged PR #16512:
URL: https://github.com/apache/datafusion/pull/16512
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@dat
2010YOUY01 commented on PR #16512:
URL: https://github.com/apache/datafusion/pull/16512#issuecomment-3025966862
> On my machine, avg bandwidth (throughput) for Q2 is
>
> (uncompressed) plain: (w) 938.4 MB/s (r) 1042.7 MB/s zstd: (w) 234.6 MB/s
(r) 276.0 MB/s lz4_frame: (w) 521.3 MB/s
ding-young commented on PR #16512:
URL: https://github.com/apache/datafusion/pull/16512#issuecomment-3018735400
On my machine, avg bandwidth (throughput) is
TBD...
And, when I ran `strace -c -e trace=write,read cargo bench --bench spill_io
compression` only for plain encoding
ding-young commented on code in PR #16512:
URL: https://github.com/apache/datafusion/pull/16512#discussion_r2174808708
##
datafusion/physical-plan/benches/spill_io.rs:
##
@@ -119,5 +127,450 @@ fn bench_spill_io(c: &mut Criterion) {
group.finish();
}
-criterion_group!(ben
2010YOUY01 commented on PR #16512:
URL: https://github.com/apache/datafusion/pull/16512#issuecomment-3018229716
> @2010YOUY01 Thank you for detailed review! The bandwidth result is quite
interesting.. Btw, I'd like to measure the bandwidth, but it looks like
Criterion doesn’t expose the mea
ding-young commented on PR #16512:
URL: https://github.com/apache/datafusion/pull/16512#issuecomment-3018158695
@2010YOUY01 Thank you for detailed review! The bandwidth result is quite
interesting..
Btw, I'd like to measure the bandwidth, but it looks like Criterion doesn’t
expose the m
2010YOUY01 commented on code in PR #16512:
URL: https://github.com/apache/datafusion/pull/16512#discussion_r216805
##
datafusion/physical-plan/benches/spill_io.rs:
##
@@ -119,5 +126,414 @@ fn bench_spill_io(c: &mut Criterion) {
group.finish();
}
-criterion_group!(ben
ding-young commented on PR #16512:
URL: https://github.com/apache/datafusion/pull/16512#issuecomment-378588
As expected, although `lz4_frame` has a lower compression ratio than `zstd`,
it runs faster, making it a reasonable tradeoff. However, since it's roughly 2x
slower than the uncomp
ding-young commented on PR #16512:
URL: https://github.com/apache/datafusion/pull/16512#issuecomment-2996151512
To run bench, `cargo bench --bench spill_io`
### Q2
- spill_compression/q2/uncompressed
time: [51.207 ms 51.521 ms 51.841 ms]
[q2 | Uncompressed]
ding-young opened a new pull request, #16512:
URL: https://github.com/apache/datafusion/pull/16512
## Which issue does this PR close?
- Related to #16367
## Rationale for this change
## What changes are included in this PR?
This pr adds some microbench
10 matches
Mail list logo
