Hi Theo, seems like you found the answer yourself, the server may return partial messages. While parsing a MessageSet you simply ignore any message whose length > remainingBufferLength.
Regards, Magnus 2015-03-01 8:55 GMT+01:00 Theo Hultberg <t...@iconara.net>: > That the message set size was always the same when this happened made me > start looking for that number, and it turned out that was what I had set as > the MaxBytes field of the requests. This make me think that what happens is > that the next message to fetch is larger than this, but it's sent anyway. > Is this correct? Is this how it is supposed to work? I'm looking through > other clients to see how they deal with this, and I think I might have > found the same thing in the Go client ( > https://github.com/Shopify/sarama/blob/master/message_set.go#L72-L76), and > the corresponding comment in the protocol guide. Is what I have found the > thing described as "As an optimization the server is allowed to return a > partial message at the end of the message set. Clients should handle this > case."? > > yours, > Theo > > On Sat, Feb 28, 2015 at 10:19 PM, Theo Hultberg <t...@iconara.net> wrote: > > > Here's an example of a frame that looks malformed, I've split the bytes > > apart and annotated the pieces with the names from the Kafka protocol > guide > > ( > > > https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchResponse > > ). > > > > Notice that the MessageSetSize is 0x00000800 (2048), but the MessageSize > > is 0x000010B3 (4275). The size prefix of the Value field agrees with > > MessageSize, but the size of the frame agrees with MessageSetSize, > > because there are only 2022 bytes left in the frame. The bytes after the > > crc field to the end of the frame don't checksum to the value of the crc > > field either. > > > > (topics) 00 00 00 01 > > TopicName 00 1F (+ 31 character topic name) > > (partitions) 00 00 00 01 > > Partition 00 00 00 02 > > ErrorCode 00 00 > > HighwaterMarkOffset 00 00 00 00 00 3F 5B 17 > > MessageSetSize 00 00 08 00 > > Offset 00 00 00 00 00 3F 5B 16 > > MessageSize 00 00 10 B3 > > Crc B0 13 F4 E3 > > MagicByte 00 > > Attributes 02 > > Key FF FF FF FF > > Value 00 00 10 A5 (+ 2022 bytes left in frame) > > > > What's going on here, what have I misunderstood? > > > > yours, > > Theo > > > > On Sat, Feb 28, 2015 at 8:35 PM, Theo Hultberg <t...@iconara.net> wrote: > > > >> Hi, > >> > >> Mostly out of curiosity I'm writing a Kafka client, and I'm getting > stuck > >> at decoding fetch responses. Most of the time everything goes fine, but > >> quite often I get frames back that seem to be wrong. I'm sure I've > >> misunderstood something about the spec, so maybe someone can get me on > the > >> right track. > >> > >> I get frames where the MessageSetSize is 2048, but it contains messages > >> whose MessageSize is larger than this. The MessageSetSize is always > 2048, > >> which makes me suspicious. > >> > >> I've tried feeding these message sets to the Java client (into > >> MemoryRecords.readableRecords), but when I iterate over it, it is empty. > >> Further inspecting what it does with the bytes I see that it gets an > >> EOFException and stops iterating, and I don't really understand what's > >> going on. > >> > >> I'm doing these fetch requests against a cluster running 0.8.2.1. > >> > >> yours, > >> Theo > >> > > > > >
