Hi guys, I realized I never thanked yall for your input - thanks! Jason: I apologize for assuming your stance on the issue! Feels like we all agreed on the solution. +1
Follow-up: Jason made a point about defining prefetch and fairness behaviour in the KIP. I am now working on putting that down in writing. To do be able to do this I think I need to understand the current prefetch behaviour in the new consumer API (0.9) a bit better. Some specific questions: - How does a specific consumer balance incoming messages from multiple partitions? Is the consumer simply issuing Multi-Fetch requests[1] for the consumed assigned partitions of the relevant topics? Or is the consumer fetching from one partition at a time and balancing between them internally? That is, is the responsibility of partition balancing (and fairness) on the broker side or consumer side? - Is the above documented somewhere? [1] https://cwiki.apache.org/confluence/display/KAFKA/Writing+a+Driver+for+Kafka, see "Multi-Fetch". Thanks, Jens On Wed, Dec 23, 2015 at 2:44 AM, Ismael Juma <ism...@juma.me.uk> wrote: > On Wed, Dec 23, 2015 at 1:24 AM, Gwen Shapira <g...@confluent.io> wrote: > > > Given the background, it sounds like you'll generally want each call to > > poll() to return the same number of events (which is the number you > planned > > on having enough memory / time for). It also sounds like tuning the > number > > of events will be closely tied to tuning the session timeout. That is - > if > > I choose to lower the session timeout for some reason, I will have to > > modify the number of records returning too. > > > > If those assumptions are correct, I think a configuration makes more > sense. > > 1. We are unlikely to want this parameter to be change at the lifetime of > > the consumer > > 2. The correct value is tied to another configuration parameter, so they > > will be controlled together. > > > > I was thinking the same thing. > > Ismael > -- Jens Rantil Backend engineer Tink AB Email: jens.ran...@tink.se Phone: +46 708 84 18 32 Web: www.tink.se Facebook <https://www.facebook.com/#!/tink.se> Linkedin <http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary> Twitter <https://twitter.com/tink>
