Re: [DISCUSS] KIP-390: Support Compression Level (rebooted)

Thanks! We could move that to a "Future Work" section instead of "Rejected Alternatives" if the results are inconclusive. Ismael On Wed, Jun 9, 2021 at 8:09 AM Dongjin Lee wrote: > > I am OK with doing compression level first, but I don't want to rule out > the buffer size change without unders

Re: [DISCUSS] KIP-390: Support Compression Level (rebooted)

> I am OK with doing compression level first, but I don't want to rule out the buffer size change without understanding better. I see. I am now retrying buffer size configuration & benchmark. As soon as I get a promising result, I will update the KIP. Thanks, Dongjin On Wed, Jun 9, 2021 at 12:36

Re: [DISCUSS] KIP-390: Support Compression Level (rebooted)

Btw, I am OK with doing compression level first, but I don't want to rule out the buffer size change without understanding better. Ismael On Tue, Jun 8, 2021 at 8:33 AM Ismael Juma wrote: > Hi Dongjin, > > I was thinking of a simple test: Snappy with 1 KB block size vs 32 KB > block size. If th

Re: [DISCUSS] KIP-390: Support Compression Level (rebooted)

Hi Dongjin, I was thinking of a simple test: Snappy with 1 KB block size vs 32 KB block size. If the compression rate is similar for both, then it seems very wasteful to use 32 KB. I suspect you will see a significant difference though. Ismael On Tue, Jun 8, 2021 at 8:27 AM Dongjin Lee wrote:

Re: [DISCUSS] KIP-390: Support Compression Level (rebooted)

Hi Ismael, I added the linear write benchmark result to the proposal. Like the producer benchmark, the least compression level showed the best MB/sec for any case. I tested several configurations, but the result was almost the same. If you have any proposals for the benchmark, don't hesitate to g

Re: [DISCUSS] KIP-390: Support Compression Level (rebooted)

Hi Ismael, Thanks for the reply. > So you're saying that reducing the buffer size didn't reduce the compression rate for codecs like lz4? Of course, there were some improvements in compressed size when I tried the 'buffer.size' option, but the gain was not significant. I tried several datasets,

Re: [DISCUSS] KIP-390: Support Compression Level (rebooted)

Thanks Dongjin. So you're saying that reducing the buffer size didn't reduce the compression rate for codecs like lz4? If so, that would suggest reducing the default value, but that seems odd. Ismael On Sat, Jun 5, 2021, 9:25 AM Dongjin Lee wrote: > Hello Kafka dev, > > I hope to reboot the dis

[DISCUSS] KIP-390: Support Compression Level (rebooted)

Hello Kafka dev, I hope to reboot the discussion of KIP-390: Support Compression Level . It proposes to add a new option, 'compression.level', that controls the compression level. This KIP has been submitted m