It is not double compression. When I say re-compression, brokers decompress the messages and compress again with new codec.
On Wed, Feb 14, 2018 at 5:18 PM, Uddhav Arote <aroteudd...@gmail.com> wrote: > Thanks. > > I am using console-producer with following settings with lz4 broker > compression codec > 1. None producer compression codec > 2. Snappy producer compression codec > 3. lz4 producer compression codec > > I send a 354 Byte message with each of the above settings. However, I do > not see any kind of double compression happening, when the producer and > broker compression codecs are different > > *Output 1:* > offset: 0 position: 0 CreateTime: 1518607686194 isvalid: true payloadsize: > 61 magic: 1 compresscodec: LZ4CompressionCodec crc: 3693540371 payload: > xxxx compressed 354Byte message > > using --deep-iteration > offset: 0 position: 0 CreateTime: 1518607686194 isvalid: true payloadsize: > 354 magic: 1 compresscodec: NoCompressionCodec crc: 4190573446 payload: 354 > byte message > > > *Output 2:*offset: 7 position: 517 CreateTime: 1518608039723 isvalid: true > payloadsize: 61 magic: 1 compresscodec: LZ4CompressionCodec crc: 4075439033 > payload: compressed 354B message > > using --deep-iteration > offset: 7 position: 517 CreateTime: 1518608039723 isvalid: true > payloadsize: 354 magic: 1 compresscodec: NoCompressionCodec crc: 4061704088 > payload: Same 354B message > > *Output 3:* > offset: 11 position: 883 CreateTime: 1518608269618 isvalid: true > payloadsize: 61 magic: 1 compresscodec: LZ4CompressionCodec crc: 981370250 > payload: compressed 354B message > > using --deep-iteration > offset: 11 position: 883 CreateTime: 1518608269618 isvalid: true > payloadsize: 354 magic: 1 compresscodec: NoCompressionCodec crc: 468622988 > payload: same 354B message > > Please note the compression codecs in the --deep-iteration case, > Case 1 is OK, but in case 2 shouldn't it be SnappyCompression and 3 may be > LZ4Compression > > Or is it visible when the message batch into large batches? > > Thanks > Uddhav > > On Wed, Feb 14, 2018 at 6:05 PM, Manikumar <manikumar.re...@gmail.com> > wrote: > > > If the broker "compression.type" is "producer", then the broker retains > > the original compression codec set by the producer. > > If the producer and broker codecs are different, then broker recompress > > the data using broker "compression.type". > > > > On Wed, Feb 14, 2018 at 10:58 AM, Uddhav Arote <aroteudd...@gmail.com> > > wrote: > > > > > Hi Kafka users, > > > > > > I am trying to understand the behavior of compression in Kafka. > Consider > > a > > > scenario, where producer sets compression.codec "snappy" and broker's > > > compression.code "lz4"? > > > In this scenario, what is the behavior of the compression? > > > > > > As far as I have understood is the following, > > > The messages compressed by the producer are wrapped in the wrapper > > message > > > and send to the broker. If the broker compression.codec is "producer", > > the > > > message is written as is to the log. > > > In the code, > > > > > > https://github.com/apache/kafka/blob/962bc638f9c2ab249e5008a587ee78 > > > e3ba35fcb9/core/src/main/scala/kafka/log/LogValidator.scala#L218 > > > > > > > > > what I understand is that if the producer and broker codecs are not > same, > > > then the compression should happen again. > > > > > > But I am not sure about this. Can somebody tell me how this works? > > > > > > Thanks, > > > Uddhav > > > > > >
