That's a good point Jun and I agree that it makes sense to keep both limits in that case.
Ismael On Fri, Aug 12, 2016 at 2:29 AM, Jun Rao <j...@confluent.io> wrote: > Hi, Andrey, > > One potential benefit of keeping the per partition limit is for Kafka > stream. When reading messages from different partitions, KStream prefers to > read from partitions with smaller timestamps first and only advances the > KStream timestamp when it sees at least one message from every partition. > Being able to fill up multiple partitions in a single fetch response can > help KStream advance the timestamp quicker when there is backlog from the > input. So, it's probably better if we just add a new response limit while > keeping the per partition limit. > > Also, for fairness across partitions, you mentioned "The solution is to > start fetching from first empty partition in round-robin fashion or to > perform random shuffle of partitions.". It seems the former is more > deterministic. Did you use that in your implementation and should we > recommend that for non-java clients as well? > > Thanks, > > Jun > > On Wed, Aug 10, 2016 at 10:55 AM, Andrey L. Neporada < > anepor...@yandex-team.ru> wrote: > > > Hi, Jun! > > > > Thanks for feedback! > > > > > On 10 Aug 2016, at 17:42, Jun Rao <j...@confluent.io> wrote: > > > > > > Hi, Andrey, > > > > > > Thanks for the reply. A couple of more comments inline below. > > > > > > On Wed, Aug 10, 2016 at 3:56 AM, Andrey L. Neporada < > > > anepor...@yandex-team.ru <mailto:anepor...@yandex-team.ru>> wrote: > > > > > >> > > >> Yes, such cooperative configuration for fetch request may look a bit > > weird. > > >> But I don’t see other options if we want to remove partition limits > from > > >> fetch request. > > >> In this case we need some server-side configuration for partition > > limits. > > >> > > >> > > > What if we keep the current partition level limit in the fetch request > > and > > > just add an additional response level limit? The default partition > limit > > > can be much smaller than the max message size and will only be used for > > > fairness across partitions. > > > > > > > Yes, we can just add global response limit and leave partition limits as > > is. > > In fact, my initial implementation (https://github.com/apache/kaf > > ka/pull/1683) of this KIP preserves per-partition limits. > > However, as it seems from KAFKA-2063 discussion, some people prefer to > > deprecate partition level limit. > > I have no real opinion on this topic - hope we can choose best option > here. > > > > ... > > >> > > >> No, I mean that actual response side can be bigger than limit_bytes, > but > > >> less than limit_bytes + message.max.bytes. > > >> This behaviour is a result of algorithm proposed in KIP (and in PR). > > >> > > >> > > > Got it. An alternative is to only add a partition's data to the > response > > up > > > to the remaining response limit. The only exception is that this is the > > > first partition and the first message in that partition is larger than > > the > > > response limit. Then the bound will be max(limit_bytes, > > message.max.bytes), > > > which is tighter. > > > > > > > Yes, this one looks better. > > > > > > Thanks, > > Andrey. >
