- it would be interesting to see the actual ProduceRequests/Responses and FetchReq/Resp. - at this point I would dive into the broker source code and follow the fetch request handling. https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaApis.scala#L418
On Tue, Feb 16, 2016 at 10:24 AM Oleksiy Krivoshey <oleks...@gmail.com> wrote: > I'm not using Java client, I'm developing my own: > https://github.com/oleksiyk/kafka > And I'm talking about MessageSet on the wire, not the one passed to library > user. > > Example: > Consumer is not started. > Producer sends a batch #1 of 5 messages, compressed in a single message. > delay 50 ms > Producer sends a batch #2 of 5 messages, compressed in a single message. > > Start consumer and it will receive a single MessageSet with two messages in > it, both messages are corresponding compressed batches. Both have codec=2 > (using Snappy). In order to receive all 10 original messages I need to > decompress both messages in this single MessageSet. Yes, I will then > concatenate these messages and send a single MessageSet with 10 messages to > library user, but I just want to clarify, the Protocol Guide says there > should be single compressed message but I'm able to receive 2, 3 and more, > all in single MessageSet. > > Kafka 0.9 > > I can provide the actual buffers received. > > Thanks! > > > On Tue, 16 Feb 2016 at 20:01 Alexis Midon <alexis.mi...@airbnb.com > .invalid> > wrote: > > > What makes you think there are 2? would you have data or code to share? > > > > When compression is enabled, multiple messages will be packed and > > compressed in a MessageSet. That MessageSet will then have a single > > message. > > The interface however will let you iterate over the unpacked messages. > See > > > > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/message/ByteBufferMessageSet.scala#L166 > > > > > > > > > https://github.com/apache/kafka/blob/trunk/core/src/test/scala/unit/kafka/message/ByteBufferMessageSetTest.scala > > > > > > > > On Tue, Feb 16, 2016 at 3:33 AM Oleksiy Krivoshey <oleks...@gmail.com> > > wrote: > > > > > Hi! > > > > > > The Kafka 0.9 protocol guide > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-Compression > > > explicitly > > > states that there should be only single compressed message in a > > MessageSet, > > > but I'm definitely receiving two compressed messages in a single > > > MessageSet. > > > > > > Can someone please confirm if this should be expected behaviour? > > > > > > Thanks! > > > > > >
