I'm not using Java client, I'm developing my own: https://github.com/oleksiyk/kafka And I'm talking about MessageSet on the wire, not the one passed to library user.
Example: Consumer is not started. Producer sends a batch #1 of 5 messages, compressed in a single message. delay 50 ms Producer sends a batch #2 of 5 messages, compressed in a single message. Start consumer and it will receive a single MessageSet with two messages in it, both messages are corresponding compressed batches. Both have codec=2 (using Snappy). In order to receive all 10 original messages I need to decompress both messages in this single MessageSet. Yes, I will then concatenate these messages and send a single MessageSet with 10 messages to library user, but I just want to clarify, the Protocol Guide says there should be single compressed message but I'm able to receive 2, 3 and more, all in single MessageSet. Kafka 0.9 I can provide the actual buffers received. Thanks! On Tue, 16 Feb 2016 at 20:01 Alexis Midon <alexis.mi...@airbnb.com.invalid> wrote: > What makes you think there are 2? would you have data or code to share? > > When compression is enabled, multiple messages will be packed and > compressed in a MessageSet. That MessageSet will then have a single > message. > The interface however will let you iterate over the unpacked messages. See > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/message/ByteBufferMessageSet.scala#L166 > > > > https://github.com/apache/kafka/blob/trunk/core/src/test/scala/unit/kafka/message/ByteBufferMessageSetTest.scala > > > > On Tue, Feb 16, 2016 at 3:33 AM Oleksiy Krivoshey <oleks...@gmail.com> > wrote: > > > Hi! > > > > The Kafka 0.9 protocol guide > > > > > https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-Compression > > explicitly > > states that there should be only single compressed message in a > MessageSet, > > but I'm definitely receiving two compressed messages in a single > > MessageSet. > > > > Can someone please confirm if this should be expected behaviour? > > > > Thanks! > > >
