This is correct. It would be possible to change the server behavior to avoid the smaller fetch responses but it would be more complex to implement and there wouldn't be much benefit to this.
You can tell you are at the end of the log because either (1) you get an empty message set back, or (2) the offset of the last message you fetched is at the log end offset. -Jay On Fri, Jun 28, 2013 at 9:53 PM, Ran RanUser <ranuse...@gmail.com> wrote: > Using Kafka 0.8, when specifying a starting offset and appropriate > fetchSize, SimpleConsumer will only return up to the highest offset of the > log segment containing the starting offset. > > For example, > > log segment #1 contains offsets 1 - 10 > log segment #2 contiains offsets 11 - 100 > > A fetch request specifying offset 9, will only return offsets 9 and 10. > Specifying starting offset 11 will return 11 - 100 > > Must one check fetchResponse.highwatermark each time to determine if there > are higher numbered offsets, and continue submitting fetchRequests ? > > I assumed SimpleConsumer would return offsets/messages spanning log > segments, up to the specified fetchSize. > > > Thanks!
