Yes that is a bit of a caveat in using zero-copy when sending FetchResponses. i.e., the broker cannot introspect the message-set and lop off any trailing piece. I think this is just something that needs to be documented clearly on that wiki. So there is some overhead for the client implementation in that the implementation has to interpret that in conjunction with MaxBytes.
Thanks, Joel On Thu, Jun 25, 2015 at 02:24:32PM -0500, Grant Henke wrote: > The fetch/message protocol described here > <https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchResponse> > shows > that the MessageSet returned from a FetchRequest should be structured as: > > MessageSet => [Offset MessageSize Message] > Offset => int64 > MessageSize => int32 > > > However, it looks like the Kafka broker can actually return an incomplete > message for the last message in the set due to the size set in MaxBytes. In > the returned MessageSet the last MessageSize returned is larger than the > total remaining bytes. It looks like an incomplete value can be returned > for any part of the message set (Offset and MessageSize) too, though I have > not confirmed. > > This means the MessageSize can't be trusted and the protocol is not > actually correct. Is this expected behavior? If expected, is it good > behavior? > > -- > Grant Henke > Solutions Consultant | Cloudera > ghe...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
