Thanks for the comment, Dong. I think the batch-split-ratio makes sense but is kind of redundant to batch-split-rate.
Also the batch-split-ratio may be a little more involved to make right: 1. A all-time batch split ratio is easy to get but not that useful. 2. A time-windowed batch-split-ratio is more complicated to make accurate. This is because it is kind of a "stateful" metric relies on the number of batches sent in a time window and number of batches got split in the same time window. But the sending and the splitting time are not necessarily falling in the same window. Besides, a rough estimation of the batch split ratio can be derived from the existing metrics. And I think batch-split-rate is already a good indication on whether the batch split has caused performance problem or not. So I am not sure if it is worth having an explicit batch-split-ratio metric in this case. Thanks, Jiangjie (Becket) Qin On Wed, Mar 22, 2017 at 10:54 AM, Dong Lin <lindon...@gmail.com> wrote: > Never mind about my second comment. I misunderstood the semantics of > producer's batch.size. > > On Wed, Mar 22, 2017 at 10:20 AM, Dong Lin <lindon...@gmail.com> wrote: > > > Hey Becket, > > > > In addition to the batch-split-rate, should we also add batch-split-ratio > > sensor to gauge the probability that we have to split batch? > > > > Also, in the case that the batch size configured for the producer is > > smaller than the max message size configured for the broker, why can't we > > just split the batch if its size exceeds the configured batch size? The > > benefit of this approach is that the semantics of producer is > > straightforward because we enforce the batch size that user has > configured. > > The implementation would also be simpler because we don't have to reply > on > > KIP-4 to fetch the max message size from broker. I guess you are worrying > > about the overhead of "unnecessary" split if a batch size is between > > user-configured batch size and broker's max message size. But is overhead > > really a concern? If overhead is too large because user has configured a > > very low batch size for producer, shouldn't user adjust produce config? > > > > Thanks, > > Dong > > > > On Wed, Mar 15, 2017 at 2:50 PM, Becket Qin <becket....@gmail.com> > wrote: > > > >> I see, then we are thinking about the same thing :) > >> > >> On Wed, Mar 15, 2017 at 2:26 PM, Ismael Juma <ism...@juma.me.uk> wrote: > >> > >> > I meant finishing what's described in the following section and then > >> > starting a discussion followed by a vote: > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >> > 4+-+Command+line+and+centralized+administrative+operations#KIP-4- > >> > Commandlineandcentralizedadministrativeoperations-DescribeCo > >> nfigsRequest > >> > > >> > We have only voted on KIP-4 Metadata, KIP-4 Create Topics, KIP-4 > Delete > >> > Topics so far. > >> > > >> > Ismael > >> > > >> > On Wed, Mar 15, 2017 at 8:58 PM, Becket Qin <becket....@gmail.com> > >> wrote: > >> > > >> > > Hi Ismael, > >> > > > >> > > KIP-4 is also the one that I was thinking about. We have introduced > a > >> > > DescribeConfigRequest there so the producer can easily get the > >> > > configurations. By "another KIP" do you mean a new (or maybe > extended) > >> > > protocol or using that protocol in clients? > >> > > > >> > > Thanks, > >> > > > >> > > Jiangjie (Becket) Qin > >> > > > >> > > On Wed, Mar 15, 2017 at 1:21 PM, Ismael Juma <ism...@juma.me.uk> > >> wrote: > >> > > > >> > > > Hi Becket, > >> > > > > >> > > > How were you thinking of retrieving the configuration items you > >> > > mentioned? > >> > > > I am asking because I was planning to post a KIP for Describe > >> Configs > >> > > (one > >> > > > of the protocols in KIP-4), which would expose such information. > But > >> > > maybe > >> > > > you are thinking of extending Metadata request? > >> > > > > >> > > > Ismael > >> > > > > >> > > > On Wed, Mar 15, 2017 at 7:33 PM, Becket Qin <becket....@gmail.com > > > >> > > wrote: > >> > > > > >> > > > > Hi Jason, > >> > > > > > >> > > > > Good point. I was thinking about that, too. I was not sure if > >> that is > >> > > the > >> > > > > right thing to do by default. > >> > > > > > >> > > > > If we assume people always set the batch size to max message > size, > >> > > > > splitting the oversized batch makes a lot of sense. But it seems > >> > > possible > >> > > > > that users want to control the memory footprint so they would > set > >> the > >> > > > batch > >> > > > > size to smaller than the max message size so the producer can > have > >> > hold > >> > > > > batches for more partitions. In this case, splitting the batch > >> might > >> > > not > >> > > > be > >> > > > > the desired behavior. > >> > > > > > >> > > > > I think the most intuitive approach to this is allow the > producer > >> to > >> > > get > >> > > > > the max message size configuration (as well as some other > >> > > configurations > >> > > > > such as timestamp type) from the broker side and use that to > >> decide > >> > > > > whether a batch should be split or not. I probably should add > >> this to > >> > > the > >> > > > > KIP wiki. > >> > > > > > >> > > > > Thanks, > >> > > > > > >> > > > > Jiangjie (Becket) Qin > >> > > > > > >> > > > > On Wed, Mar 15, 2017 at 9:47 AM, Jason Gustafson < > >> ja...@confluent.io > >> > > > >> > > > > wrote: > >> > > > > > >> > > > > > Hey Becket, > >> > > > > > > >> > > > > > Thanks for the KIP! The approach seems reasonable. One > >> > clarification: > >> > > > is > >> > > > > > the intent to do the splitting after the broker rejects the > >> request > >> > > > with > >> > > > > > MESSAGE_TOO_LARGE, or prior to sending if the configured batch > >> size > >> > > is > >> > > > > > exceeded? > >> > > > > > > >> > > > > > -Jason > >> > > > > > > >> > > > > > On Mon, Mar 13, 2017 at 8:10 PM, Becket Qin < > >> becket....@gmail.com> > >> > > > > wrote: > >> > > > > > > >> > > > > > > Bump up the thread for further comments. If there is no more > >> > > comments > >> > > > > on > >> > > > > > > the KIP I will start the voting thread on Wed. > >> > > > > > > > >> > > > > > > Thanks, > >> > > > > > > > >> > > > > > > Jiangjie (Becket) Qin > >> > > > > > > > >> > > > > > > On Tue, Mar 7, 2017 at 9:48 AM, Becket Qin < > >> becket....@gmail.com > >> > > > >> > > > > wrote: > >> > > > > > > > >> > > > > > > > Hi Dong, > >> > > > > > > > > >> > > > > > > > Thanks for the comments. > >> > > > > > > > > >> > > > > > > > The patch is mostly for proof of concept in case there is > >> any > >> > > > concern > >> > > > > > > > about the implementation which is indeed a little tricky. > >> > > > > > > > > >> > > > > > > > The new metric has already been mentioned in the Public > >> > Interface > >> > > > > > Change > >> > > > > > > > section. > >> > > > > > > > > >> > > > > > > > I added the reasoning about how the compression ratio > >> > > > > > > > improving/deteriorate steps are determined in the wiki. > >> > > > > > > > > >> > > > > > > > Thanks, > >> > > > > > > > > >> > > > > > > > Jiangjie (Becket) Qin > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > On Mon, Mar 6, 2017 at 4:42 PM, Dong Lin < > >> lindon...@gmail.com> > >> > > > > wrote: > >> > > > > > > > > >> > > > > > > >> Hey Becket, > >> > > > > > > >> > >> > > > > > > >> I am wondering if we should first vote for the KIP before > >> > > > reviewing > >> > > > > > the > >> > > > > > > >> patch. I have two comments below: > >> > > > > > > >> > >> > > > > > > >> - Should we specify the new sensors as part of interface > >> > change > >> > > in > >> > > > > the > >> > > > > > > >> KIP? > >> > > > > > > >> - The KIP proposes to increase estimated compression > ratio > >> by > >> > > 0.05 > >> > > > > for > >> > > > > > > >> each > >> > > > > > > >> underestimation and decrement the estimation by 0.005 for > >> each > >> > > > > > > >> overestimation. Why are these two values chosen? I think > >> there > >> > > is > >> > > > > some > >> > > > > > > >> tradeoff in selecting the value. Can the KIP be more > >> explicit > >> > > > about > >> > > > > > the > >> > > > > > > >> tradeoff and explain how these two values would impact > >> > > producer's > >> > > > > > > >> performance? > >> > > > > > > >> > >> > > > > > > >> Thanks, > >> > > > > > > >> Dong > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> On Sat, Mar 4, 2017 at 11:42 AM, Becket Qin < > >> > > becket....@gmail.com > >> > > > > > >> > > > > > > wrote: > >> > > > > > > >> > >> > > > > > > >> > I have updated the KIP based on the latest discussion. > >> > Please > >> > > > > check > >> > > > > > > and > >> > > > > > > >> let > >> > > > > > > >> > me know if there is any further concern. > >> > > > > > > >> > > >> > > > > > > >> > Thanks, > >> > > > > > > >> > > >> > > > > > > >> > Jiangjie (Becket) Qin > >> > > > > > > >> > > >> > > > > > > >> > On Sat, Mar 4, 2017 at 10:56 AM, Becket Qin < > >> > > > becket....@gmail.com > >> > > > > > > >> > > > > > > >> wrote: > >> > > > > > > >> > > >> > > > > > > >> > > Actually second thought on this, rate might be better > >> for > >> > > two > >> > > > > > > reasons: > >> > > > > > > >> > > 1. Most of the metrics in the producer we already > have > >> are > >> > > > using > >> > > > > > > rate > >> > > > > > > >> > > instead of count. > >> > > > > > > >> > > 2. If a service is bounced, the count will be reset > to > >> 0, > >> > > but > >> > > > it > >> > > > > > > does > >> > > > > > > >> not > >> > > > > > > >> > > affect rate. > >> > > > > > > >> > > > >> > > > > > > >> > > I'll make the change. > >> > > > > > > >> > > > >> > > > > > > >> > > Thanks, > >> > > > > > > >> > > > >> > > > > > > >> > > Jiangjie (Becket) Qin > >> > > > > > > >> > > > >> > > > > > > >> > > On Sat, Mar 4, 2017 at 10:27 AM, Becket Qin < > >> > > > > becket....@gmail.com > >> > > > > > > > >> > > > > > > >> > wrote: > >> > > > > > > >> > > > >> > > > > > > >> > >> Hi Dong, > >> > > > > > > >> > >> > >> > > > > > > >> > >> Yes, there is a sensor in the patch about the split > >> > > > occurrence. > >> > > > > > > >> > >> > >> > > > > > > >> > >> Currently it is a count instead of rate. In > practice, > >> it > >> > > > seems > >> > > > > > > count > >> > > > > > > >> is > >> > > > > > > >> > >> easier to use in this case. But I am open to change. > >> > > > > > > >> > >> > >> > > > > > > >> > >> Thanks, > >> > > > > > > >> > >> > >> > > > > > > >> > >> Jiangjie (Becket) Qin > >> > > > > > > >> > >> > >> > > > > > > >> > >> On Fri, Mar 3, 2017 at 7:43 PM, Dong Lin < > >> > > > lindon...@gmail.com> > >> > > > > > > >> wrote: > >> > > > > > > >> > >> > >> > > > > > > >> > >>> Hey Becket, > >> > > > > > > >> > >>> > >> > > > > > > >> > >>> I haven't looked at the patch yet. But since we are > >> > going > >> > > to > >> > > > > try > >> > > > > > > the > >> > > > > > > >> > >>> split-on-oversize solution, should the KIP also > add a > >> > > sensor > >> > > > > > that > >> > > > > > > >> shows > >> > > > > > > >> > >>> the > >> > > > > > > >> > >>> rate of split per second and the probability of > >> split? > >> > > > > > > >> > >>> > >> > > > > > > >> > >>> Thanks, > >> > > > > > > >> > >>> Dong > >> > > > > > > >> > >>> > >> > > > > > > >> > >>> > >> > > > > > > >> > >>> On Fri, Mar 3, 2017 at 6:39 PM, Becket Qin < > >> > > > > > becket....@gmail.com> > >> > > > > > > >> > wrote: > >> > > > > > > >> > >>> > >> > > > > > > >> > >>> > Just to clarify, the implementation is basically > >> what > >> > I > >> > > > > > > mentioned > >> > > > > > > >> > above > >> > > > > > > >> > >>> > (split/resend + adjusted estimation evolving > >> > algorithm) > >> > > > and > >> > > > > > > >> changing > >> > > > > > > >> > >>> the > >> > > > > > > >> > >>> > compression ratio estimation to be per topic. > >> > > > > > > >> > >>> > > >> > > > > > > >> > >>> > Thanks, > >> > > > > > > >> > >>> > > >> > > > > > > >> > >>> > Jiangjie (Becket) Qin > >> > > > > > > >> > >>> > > >> > > > > > > >> > >>> > On Fri, Mar 3, 2017 at 6:36 PM, Becket Qin < > >> > > > > > > becket....@gmail.com> > >> > > > > > > >> > >>> wrote: > >> > > > > > > >> > >>> > > >> > > > > > > >> > >>> > > I went ahead and have a patch submitted here: > >> > > > > > > >> > >>> > > https://github.com/apache/kafka/pull/2638 > >> > > > > > > >> > >>> > > > >> > > > > > > >> > >>> > > Per Joel's suggestion, I changed the > compression > >> > ratio > >> > > > to > >> > > > > be > >> > > > > > > per > >> > > > > > > >> > >>> topic as > >> > > > > > > >> > >>> > > well. It seems working well. Since there is an > >> > > important > >> > > > > > > >> behavior > >> > > > > > > >> > >>> change > >> > > > > > > >> > >>> > > and a new sensor is added, I'll keep the KIP > and > >> > > update > >> > > > it > >> > > > > > > >> > according. > >> > > > > > > >> > >>> > > > >> > > > > > > >> > >>> > > Thanks, > >> > > > > > > >> > >>> > > > >> > > > > > > >> > >>> > > Jiangjie (Becket) Qin > >> > > > > > > >> > >>> > > > >> > > > > > > >> > >>> > > On Mon, Feb 27, 2017 at 3:50 PM, Joel Koshy < > >> > > > > > > >> jjkosh...@gmail.com> > >> > > > > > > >> > >>> wrote: > >> > > > > > > >> > >>> > > > >> > > > > > > >> > >>> > >> > > >> > > > > > > >> > >>> > >> > Lets say we sent the batch over the wire and > >> > > > received a > >> > > > > > > >> > >>> > >> > RecordTooLargeException, how do we split it > as > >> > once > >> > > > we > >> > > > > > add > >> > > > > > > >> the > >> > > > > > > >> > >>> message > >> > > > > > > >> > >>> > >> to > >> > > > > > > >> > >>> > >> > the batch we loose the message level > >> granularity. > >> > > We > >> > > > > will > >> > > > > > > >> have > >> > > > > > > >> > to > >> > > > > > > >> > >>> > >> > decompress, do deep iteration and split and > >> again > >> > > > > > compress. > >> > > > > > > >> > right? > >> > > > > > > >> > >>> > This > >> > > > > > > >> > >>> > >> > looks like a performance bottle neck in case > >> of > >> > > multi > >> > > > > > topic > >> > > > > > > >> > >>> producers > >> > > > > > > >> > >>> > >> like > >> > > > > > > >> > >>> > >> > mirror maker. > >> > > > > > > >> > >>> > >> > > >> > > > > > > >> > >>> > >> > >> > > > > > > >> > >>> > >> Yes, but these should be outliers if we do > >> > estimation > >> > > > on > >> > > > > a > >> > > > > > > >> > per-topic > >> > > > > > > >> > >>> > basis > >> > > > > > > >> > >>> > >> and if we target a conservative-enough > >> compression > >> > > > ratio. > >> > > > > > The > >> > > > > > > >> > >>> producer > >> > > > > > > >> > >>> > >> should also avoid sending over the wire if it > >> can > >> > be > >> > > > made > >> > > > > > > >> aware of > >> > > > > > > >> > >>> the > >> > > > > > > >> > >>> > >> max-message size limit on the broker, and > split > >> if > >> > it > >> > > > > > > >> determines > >> > > > > > > >> > >>> that a > >> > > > > > > >> > >>> > >> record exceeds the broker's config. Ideally > this > >> > > should > >> > > > > be > >> > > > > > > >> part of > >> > > > > > > >> > >>> topic > >> > > > > > > >> > >>> > >> metadata but is not - so it could be off a > >> periodic > >> > > > > > > >> > describe-configs > >> > > > > > > >> > >>> > >> <https://cwiki.apache.org/ > >> > > > confluence/display/KAFKA/KIP- > >> > > > > > 4+-+ > >> > > > > > > >> > >>> > >> Command+line+and+centralized+ > >> > > > > > administrative+operations#KIP- > >> > > > > > > >> > >>> > >> 4-Commandlineandcentralizedadmin > >> > > > > > istrativeoperations-Describe > >> > > > > > > >> > >>> > >> ConfigsRequest> > >> > > > > > > >> > >>> > >> (which isn't available yet). This doesn't > remove > >> > the > >> > > > need > >> > > > > > to > >> > > > > > > >> split > >> > > > > > > >> > >>> and > >> > > > > > > >> > >>> > >> recompress though. > >> > > > > > > >> > >>> > >> > >> > > > > > > >> > >>> > >> > >> > > > > > > >> > >>> > >> > On Mon, Feb 27, 2017 at 10:51 AM, Becket > Qin < > >> > > > > > > >> > >>> becket....@gmail.com> > >> > > > > > > >> > >>> > >> wrote: > >> > > > > > > >> > >>> > >> > > >> > > > > > > >> > >>> > >> > > Hey Mayuresh, > >> > > > > > > >> > >>> > >> > > > >> > > > > > > >> > >>> > >> > > 1) The batch would be split when an > >> > > > > > > >> RecordTooLargeException is > >> > > > > > > >> > >>> > >> received. > >> > > > > > > >> > >>> > >> > > 2) Not lower the actual compression ratio, > >> but > >> > > > lower > >> > > > > > the > >> > > > > > > >> > >>> estimated > >> > > > > > > >> > >>> > >> > > compression ratio "according to" the > Actual > >> > > > > Compression > >> > > > > > > >> > >>> Ratio(ACR). > >> > > > > > > >> > >>> > >> > > > >> > > > > > > >> > >>> > >> > > An example, let's start with Estimated > >> > > Compression > >> > > > > > Ratio > >> > > > > > > >> > (ECR) = > >> > > > > > > >> > >>> > 1.0. > >> > > > > > > >> > >>> > >> Say > >> > > > > > > >> > >>> > >> > > the compression ratio of ACR is ~0.8, > >> instead > >> > of > >> > > > > > letting > >> > > > > > > >> the > >> > > > > > > >> > ECR > >> > > > > > > >> > >>> > >> dropped > >> > > > > > > >> > >>> > >> > to > >> > > > > > > >> > >>> > >> > > 0.8 very quickly, we only drop 0.001 every > >> time > >> > > > when > >> > > > > > ACR > >> > > > > > > < > >> > > > > > > >> > ECR. > >> > > > > > > >> > >>> > >> However, > >> > > > > > > >> > >>> > >> > > once we see an ACR > ECR, we increment ECR > >> by > >> > > 0.05. > >> > > > > If > >> > > > > > a > >> > > > > > > >> > >>> > >> > > RecordTooLargeException is received, we > >> reset > >> > the > >> > > > ECR > >> > > > > > > back > >> > > > > > > >> to > >> > > > > > > >> > >>> 1.0 > >> > > > > > > >> > >>> > and > >> > > > > > > >> > >>> > >> > split > >> > > > > > > >> > >>> > >> > > the batch. > >> > > > > > > >> > >>> > >> > > > >> > > > > > > >> > >>> > >> > > Thanks, > >> > > > > > > >> > >>> > >> > > > >> > > > > > > >> > >>> > >> > > Jiangjie (Becket) Qin > >> > > > > > > >> > >>> > >> > > > >> > > > > > > >> > >>> > >> > > > >> > > > > > > >> > >>> > >> > > > >> > > > > > > >> > >>> > >> > > On Mon, Feb 27, 2017 at 10:30 AM, Mayuresh > >> > > Gharat < > >> > > > > > > >> > >>> > >> > > gharatmayures...@gmail.com> wrote: > >> > > > > > > >> > >>> > >> > > > >> > > > > > > >> > >>> > >> > > > Hi Becket, > >> > > > > > > >> > >>> > >> > > > > >> > > > > > > >> > >>> > >> > > > Seems like an interesting idea. > >> > > > > > > >> > >>> > >> > > > I had couple of questions : > >> > > > > > > >> > >>> > >> > > > 1) How do we decide when the batch > should > >> be > >> > > > split? > >> > > > > > > >> > >>> > >> > > > 2) What do you mean by slowly lowering > the > >> > > > "actual" > >> > > > > > > >> > >>> compression > >> > > > > > > >> > >>> > >> ratio? > >> > > > > > > >> > >>> > >> > > > An example would really help here. > >> > > > > > > >> > >>> > >> > > > > >> > > > > > > >> > >>> > >> > > > Thanks, > >> > > > > > > >> > >>> > >> > > > > >> > > > > > > >> > >>> > >> > > > Mayuresh > >> > > > > > > >> > >>> > >> > > > > >> > > > > > > >> > >>> > >> > > > On Fri, Feb 24, 2017 at 3:17 PM, Becket > >> Qin < > >> > > > > > > >> > >>> becket....@gmail.com > >> > > > > > > >> > >>> > > > >> > > > > > > >> > >>> > >> > > wrote: > >> > > > > > > >> > >>> > >> > > > > >> > > > > > > >> > >>> > >> > > > > Hi Jay, > >> > > > > > > >> > >>> > >> > > > > > >> > > > > > > >> > >>> > >> > > > > Yeah, I got your point. > >> > > > > > > >> > >>> > >> > > > > > >> > > > > > > >> > >>> > >> > > > > I think there might be a solution > which > >> do > >> > > not > >> > > > > > > require > >> > > > > > > >> > >>> adding a > >> > > > > > > >> > >>> > >> new > >> > > > > > > >> > >>> > >> > > > > configuration. We can start from a > very > >> > > > > > conservative > >> > > > > > > >> > >>> compression > >> > > > > > > >> > >>> > >> > ratio > >> > > > > > > >> > >>> > >> > > > say > >> > > > > > > >> > >>> > >> > > > > 1.0 and lower it very slowly according > >> to > >> > the > >> > > > > > actual > >> > > > > > > >> > >>> compression > >> > > > > > > >> > >>> > >> > ratio > >> > > > > > > >> > >>> > >> > > > > until we hit a point that we have to > >> split > >> > a > >> > > > > batch. > >> > > > > > > At > >> > > > > > > >> > that > >> > > > > > > >> > >>> > >> point, we > >> > > > > > > >> > >>> > >> > > > > exponentially back off on the > >> compression > >> > > > ratio. > >> > > > > > The > >> > > > > > > >> idea > >> > > > > > > >> > is > >> > > > > > > >> > >>> > >> somewhat > >> > > > > > > >> > >>> > >> > > > like > >> > > > > > > >> > >>> > >> > > > > TCP. This should help avoid frequent > >> split. > >> > > > > > > >> > >>> > >> > > > > > >> > > > > > > >> > >>> > >> > > > > The upper bound of the batch size is > >> also a > >> > > > > little > >> > > > > > > >> awkward > >> > > > > > > >> > >>> today > >> > > > > > > >> > >>> > >> > > because > >> > > > > > > >> > >>> > >> > > > we > >> > > > > > > >> > >>> > >> > > > > say the batch size is based on > >> compressed > >> > > size, > >> > > > > but > >> > > > > > > >> users > >> > > > > > > >> > >>> cannot > >> > > > > > > >> > >>> > >> set > >> > > > > > > >> > >>> > >> > it > >> > > > > > > >> > >>> > >> > > > to > >> > > > > > > >> > >>> > >> > > > > the max message size because that will > >> > result > >> > > > in > >> > > > > > > >> oversized > >> > > > > > > >> > >>> > >> messages. > >> > > > > > > >> > >>> > >> > > With > >> > > > > > > >> > >>> > >> > > > > this change we will be able to allow > the > >> > > users > >> > > > to > >> > > > > > set > >> > > > > > > >> the > >> > > > > > > >> > >>> > message > >> > > > > > > >> > >>> > >> > size > >> > > > > > > >> > >>> > >> > > to > >> > > > > > > >> > >>> > >> > > > > close to max message size. > >> > > > > > > >> > >>> > >> > > > > > >> > > > > > > >> > >>> > >> > > > > However the downside is that there > >> could be > >> > > > > latency > >> > > > > > > >> spikes > >> > > > > > > >> > >>> in > >> > > > > > > >> > >>> > the > >> > > > > > > >> > >>> > >> > > system > >> > > > > > > >> > >>> > >> > > > in > >> > > > > > > >> > >>> > >> > > > > this case due to the splitting, > >> especially > >> > > when > >> > > > > > there > >> > > > > > > >> are > >> > > > > > > >> > >>> many > >> > > > > > > >> > >>> > >> > messages > >> > > > > > > >> > >>> > >> > > > > need to be split at the same time. > That > >> > could > >> > > > > > > >> potentially > >> > > > > > > >> > >>> be an > >> > > > > > > >> > >>> > >> issue > >> > > > > > > >> > >>> > >> > > for > >> > > > > > > >> > >>> > >> > > > > some users. > >> > > > > > > >> > >>> > >> > > > > > >> > > > > > > >> > >>> > >> > > > > What do you think about this approach? > >> > > > > > > >> > >>> > >> > > > > > >> > > > > > > >> > >>> > >> > > > > Thanks, > >> > > > > > > >> > >>> > >> > > > > > >> > > > > > > >> > >>> > >> > > > > Jiangjie (Becket) Qin > >> > > > > > > >> > >>> > >> > > > > > >> > > > > > > >> > >>> > >> > > > > > >> > > > > > > >> > >>> > >> > > > > > >> > > > > > > >> > >>> > >> > > > > On Thu, Feb 23, 2017 at 1:31 PM, Jay > >> Kreps > >> > < > >> > > > > > > >> > >>> j...@confluent.io> > >> > > > > > > >> > >>> > >> wrote: > >> > > > > > > >> > >>> > >> > > > > > >> > > > > > > >> > >>> > >> > > > > > Hey Becket, > >> > > > > > > >> > >>> > >> > > > > > > >> > > > > > > >> > >>> > >> > > > > > Yeah that makes sense. > >> > > > > > > >> > >>> > >> > > > > > > >> > > > > > > >> > >>> > >> > > > > > I agree that you'd really have to > both > >> > fix > >> > > > the > >> > > > > > > >> > estimation > >> > > > > > > >> > >>> > (i.e. > >> > > > > > > >> > >>> > >> > make > >> > > > > > > >> > >>> > >> > > it > >> > > > > > > >> > >>> > >> > > > > per > >> > > > > > > >> > >>> > >> > > > > > topic or make it better estimate the > >> high > >> > > > > > > >> percentiles) > >> > > > > > > >> > AND > >> > > > > > > >> > >>> > have > >> > > > > > > >> > >>> > >> the > >> > > > > > > >> > >>> > >> > > > > > recovery mechanism. If you are > >> > > > underestimating > >> > > > > > > often > >> > > > > > > >> and > >> > > > > > > >> > >>> then > >> > > > > > > >> > >>> > >> > paying > >> > > > > > > >> > >>> > >> > > a > >> > > > > > > >> > >>> > >> > > > > high > >> > > > > > > >> > >>> > >> > > > > > recovery price that won't fly. > >> > > > > > > >> > >>> > >> > > > > > > >> > > > > > > >> > >>> > >> > > > > > I think you take my main point > though, > >> > > which > >> > > > is > >> > > > > > > just > >> > > > > > > >> > that > >> > > > > > > >> > >>> I > >> > > > > > > >> > >>> > >> hate to > >> > > > > > > >> > >>> > >> > > > > exposes > >> > > > > > > >> > >>> > >> > > > > > these super low level options to > users > >> > > > because > >> > > > > it > >> > > > > > > is > >> > > > > > > >> so > >> > > > > > > >> > >>> hard > >> > > > > > > >> > >>> > to > >> > > > > > > >> > >>> > >> > > explain > >> > > > > > > >> > >>> > >> > > > > to > >> > > > > > > >> > >>> > >> > > > > > people what it means and how they > >> should > >> > > set > >> > > > > it. > >> > > > > > So > >> > > > > > > >> if > >> > > > > > > >> > it > >> > > > > > > >> > >>> is > >> > > > > > > >> > >>> > >> > possible > >> > > > > > > >> > >>> > >> > > > to > >> > > > > > > >> > >>> > >> > > > > > make either some combination of > better > >> > > > > estimation > >> > > > > > > and > >> > > > > > > >> > >>> > splitting > >> > > > > > > >> > >>> > >> or > >> > > > > > > >> > >>> > >> > > > better > >> > > > > > > >> > >>> > >> > > > > > tolerance of overage that would be > >> > > > preferrable. > >> > > > > > > >> > >>> > >> > > > > > > >> > > > > > > >> > >>> > >> > > > > > -Jay > >> > > > > > > >> > >>> > >> > > > > > > >> > > > > > > >> > >>> > >> > > > > > On Thu, Feb 23, 2017 at 11:51 AM, > >> Becket > >> > > Qin > >> > > > < > >> > > > > > > >> > >>> > >> becket....@gmail.com > >> > > > > > > >> > >>> > >> > > > >> > > > > > > >> > >>> > >> > > > > wrote: > >> > > > > > > >> > >>> > >> > > > > > > >> > > > > > > >> > >>> > >> > > > > > > @Dong, > >> > > > > > > >> > >>> > >> > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > Thanks for the comments. The > default > >> > > > behavior > >> > > > > > of > >> > > > > > > >> the > >> > > > > > > >> > >>> > producer > >> > > > > > > >> > >>> > >> > won't > >> > > > > > > >> > >>> > >> > > > > > change. > >> > > > > > > >> > >>> > >> > > > > > > If the users want to use the > >> > uncompressed > >> > > > > > message > >> > > > > > > >> > size, > >> > > > > > > >> > >>> they > >> > > > > > > >> > >>> > >> > > probably > >> > > > > > > >> > >>> > >> > > > > > will > >> > > > > > > >> > >>> > >> > > > > > > also bump up the batch size to > >> > somewhere > >> > > > > close > >> > > > > > to > >> > > > > > > >> the > >> > > > > > > >> > >>> max > >> > > > > > > >> > >>> > >> message > >> > > > > > > >> > >>> > >> > > > size. > >> > > > > > > >> > >>> > >> > > > > > > This would be in the document. BTW > >> the > >> > > > > default > >> > > > > > > >> batch > >> > > > > > > >> > >>> size is > >> > > > > > > >> > >>> > >> 16K > >> > > > > > > >> > >>> > >> > > > which > >> > > > > > > >> > >>> > >> > > > > is > >> > > > > > > >> > >>> > >> > > > > > > pretty small. > >> > > > > > > >> > >>> > >> > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > @Jay, > >> > > > > > > >> > >>> > >> > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > Yeah, we actually had debated > quite > >> a > >> > bit > >> > > > > > > >> internally > >> > > > > > > >> > >>> what is > >> > > > > > > >> > >>> > >> the > >> > > > > > > >> > >>> > >> > > best > >> > > > > > > >> > >>> > >> > > > > > > solution to this. > >> > > > > > > >> > >>> > >> > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > I completely agree it is a bug. In > >> > > practice > >> > > > > we > >> > > > > > > >> usually > >> > > > > > > >> > >>> leave > >> > > > > > > >> > >>> > >> some > >> > > > > > > >> > >>> > >> > > > > > headroom > >> > > > > > > >> > >>> > >> > > > > > > to allow the compressed size to > >> grow a > >> > > > little > >> > > > > > if > >> > > > > > > >> the > >> > > > > > > >> > the > >> > > > > > > >> > >>> > >> original > >> > > > > > > >> > >>> > >> > > > > > messages > >> > > > > > > >> > >>> > >> > > > > > > are not compressible, for example, > >> 1000 > >> > > KB > >> > > > > > > instead > >> > > > > > > >> of > >> > > > > > > >> > >>> > exactly > >> > > > > > > >> > >>> > >> 1 > >> > > > > > > >> > >>> > >> > MB. > >> > > > > > > >> > >>> > >> > > > It > >> > > > > > > >> > >>> > >> > > > > is > >> > > > > > > >> > >>> > >> > > > > > > likely safe enough. > >> > > > > > > >> > >>> > >> > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > The major concern for the rejected > >> > > > > alternative > >> > > > > > is > >> > > > > > > >> > >>> > >> performance. It > >> > > > > > > >> > >>> > >> > > > > largely > >> > > > > > > >> > >>> > >> > > > > > > depends on how frequent we need to > >> > split > >> > > a > >> > > > > > batch, > >> > > > > > > >> i.e. > >> > > > > > > >> > >>> how > >> > > > > > > >> > >>> > >> likely > >> > > > > > > >> > >>> > >> > > the > >> > > > > > > >> > >>> > >> > > > > > > estimation can go off. If we only > >> need > >> > to > >> > > > the > >> > > > > > > split > >> > > > > > > >> > work > >> > > > > > > >> > >>> > >> > > > occasionally, > >> > > > > > > >> > >>> > >> > > > > > the > >> > > > > > > >> > >>> > >> > > > > > > cost would be amortized so we > don't > >> > need > >> > > to > >> > > > > > worry > >> > > > > > > >> > about > >> > > > > > > >> > >>> it > >> > > > > > > >> > >>> > too > >> > > > > > > >> > >>> > >> > > much. > >> > > > > > > >> > >>> > >> > > > > > > However, it looks that for a > >> producer > >> > > with > >> > > > > > shared > >> > > > > > > >> > >>> topics, > >> > > > > > > >> > >>> > the > >> > > > > > > >> > >>> > >> > > > > estimation > >> > > > > > > >> > >>> > >> > > > > > is > >> > > > > > > >> > >>> > >> > > > > > > always off. As an example, > consider > >> two > >> > > > > topics, > >> > > > > > > one > >> > > > > > > >> > with > >> > > > > > > >> > >>> > >> > > compression > >> > > > > > > >> > >>> > >> > > > > > ratio > >> > > > > > > >> > >>> > >> > > > > > > 0.6 the other 0.2, assuming > exactly > >> > same > >> > > > > > traffic, > >> > > > > > > >> the > >> > > > > > > >> > >>> > average > >> > > > > > > >> > >>> > >> > > > > compression > >> > > > > > > >> > >>> > >> > > > > > > ratio would be roughly 0.4, which > is > >> > not > >> > > > > right > >> > > > > > > for > >> > > > > > > >> > >>> either of > >> > > > > > > >> > >>> > >> the > >> > > > > > > >> > >>> > >> > > > > topics. > >> > > > > > > >> > >>> > >> > > > > > So > >> > > > > > > >> > >>> > >> > > > > > > almost half of the batches (of the > >> > topics > >> > > > > with > >> > > > > > > 0.6 > >> > > > > > > >> > >>> > compression > >> > > > > > > >> > >>> > >> > > ratio) > >> > > > > > > >> > >>> > >> > > > > > will > >> > > > > > > >> > >>> > >> > > > > > > end up larger than the configured > >> batch > >> > > > size. > >> > > > > > > When > >> > > > > > > >> it > >> > > > > > > >> > >>> comes > >> > > > > > > >> > >>> > to > >> > > > > > > >> > >>> > >> > more > >> > > > > > > >> > >>> > >> > > > > > topics > >> > > > > > > >> > >>> > >> > > > > > > such as mirror maker, this becomes > >> more > >> > > > > > > >> unpredictable. > >> > > > > > > >> > >>> To > >> > > > > > > >> > >>> > >> avoid > >> > > > > > > >> > >>> > >> > > > > frequent > >> > > > > > > >> > >>> > >> > > > > > > rejection / split of the batches, > we > >> > need > >> > > > to > >> > > > > > > >> > configured > >> > > > > > > >> > >>> the > >> > > > > > > >> > >>> > >> batch > >> > > > > > > >> > >>> > >> > > > size > >> > > > > > > >> > >>> > >> > > > > > > pretty conservatively. This could > >> > > actually > >> > > > > hurt > >> > > > > > > the > >> > > > > > > >> > >>> > >> performance > >> > > > > > > >> > >>> > >> > > > because > >> > > > > > > >> > >>> > >> > > > > > we > >> > > > > > > >> > >>> > >> > > > > > > are shoehorn the messages that are > >> > highly > >> > > > > > > >> compressible > >> > > > > > > >> > >>> to a > >> > > > > > > >> > >>> > >> small > >> > > > > > > >> > >>> > >> > > > batch > >> > > > > > > >> > >>> > >> > > > > > so > >> > > > > > > >> > >>> > >> > > > > > > that the other topics that are not > >> that > >> > > > > > > >> compressible > >> > > > > > > >> > >>> will > >> > > > > > > >> > >>> > not > >> > > > > > > >> > >>> > >> > > become > >> > > > > > > >> > >>> > >> > > > > too > >> > > > > > > >> > >>> > >> > > > > > > large with the same batch size. At > >> > > > LinkedIn, > >> > > > > > our > >> > > > > > > >> batch > >> > > > > > > >> > >>> size > >> > > > > > > >> > >>> > is > >> > > > > > > >> > >>> > >> > > > > configured > >> > > > > > > >> > >>> > >> > > > > > > to 64 KB because of this. I think > we > >> > may > >> > > > > > actually > >> > > > > > > >> have > >> > > > > > > >> > >>> > better > >> > > > > > > >> > >>> > >> > > > batching > >> > > > > > > >> > >>> > >> > > > > if > >> > > > > > > >> > >>> > >> > > > > > > we just use the uncompressed > message > >> > size > >> > > > and > >> > > > > > 800 > >> > > > > > > >> KB > >> > > > > > > >> > >>> batch > >> > > > > > > >> > >>> > >> size. > >> > > > > > > >> > >>> > >> > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > We did not think about loosening > the > >> > > > message > >> > > > > > size > >> > > > > > > >> > >>> > restriction, > >> > > > > > > >> > >>> > >> > but > >> > > > > > > >> > >>> > >> > > > that > >> > > > > > > >> > >>> > >> > > > > > > sounds a viable solution given > that > >> the > >> > > > > > consumer > >> > > > > > > >> now > >> > > > > > > >> > can > >> > > > > > > >> > >>> > fetch > >> > > > > > > >> > >>> > >> > > > > oversized > >> > > > > > > >> > >>> > >> > > > > > > messages. One concern would be > that > >> on > >> > > the > >> > > > > > broker > >> > > > > > > >> side > >> > > > > > > >> > >>> > >> oversized > >> > > > > > > >> > >>> > >> > > > > messages > >> > > > > > > >> > >>> > >> > > > > > > will bring more memory pressure. > >> With > >> > > > KIP-92, > >> > > > > > we > >> > > > > > > >> may > >> > > > > > > >> > >>> > mitigate > >> > > > > > > >> > >>> > >> > that, > >> > > > > > > >> > >>> > >> > > > but > >> > > > > > > >> > >>> > >> > > > > > the > >> > > > > > > >> > >>> > >> > > > > > > memory allocation for large > messages > >> > may > >> > > > not > >> > > > > be > >> > > > > > > >> very > >> > > > > > > >> > GC > >> > > > > > > >> > >>> > >> > friendly. I > >> > > > > > > >> > >>> > >> > > > > need > >> > > > > > > >> > >>> > >> > > > > > to > >> > > > > > > >> > >>> > >> > > > > > > think about this a little more. > >> > > > > > > >> > >>> > >> > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > Thanks, > >> > > > > > > >> > >>> > >> > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > Jiangjie (Becket) Qin > >> > > > > > > >> > >>> > >> > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > On Wed, Feb 22, 2017 at 8:57 PM, > Jay > >> > > Kreps > >> > > > < > >> > > > > > > >> > >>> > j...@confluent.io> > >> > > > > > > >> > >>> > >> > > wrote: > >> > > > > > > >> > >>> > >> > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > > Hey Becket, > >> > > > > > > >> > >>> > >> > > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > > I get the problem we want to > solve > >> > with > >> > > > > this, > >> > > > > > > >> but I > >> > > > > > > >> > >>> don't > >> > > > > > > >> > >>> > >> think > >> > > > > > > >> > >>> > >> > > > this > >> > > > > > > >> > >>> > >> > > > > is > >> > > > > > > >> > >>> > >> > > > > > > > something that makes sense as a > >> user > >> > > > > > controlled > >> > > > > > > >> knob > >> > > > > > > >> > >>> that > >> > > > > > > >> > >>> > >> > > everyone > >> > > > > > > >> > >>> > >> > > > > > > sending > >> > > > > > > >> > >>> > >> > > > > > > > data to kafka has to think > about. > >> It > >> > is > >> > > > > > > >> basically a > >> > > > > > > >> > >>> bug, > >> > > > > > > >> > >>> > >> right? > >> > > > > > > >> > >>> > >> > > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > > First, as a technical question > is > >> it > >> > > true > >> > > > > > that > >> > > > > > > >> using > >> > > > > > > >> > >>> the > >> > > > > > > >> > >>> > >> > > > uncompressed > >> > > > > > > >> > >>> > >> > > > > > > size > >> > > > > > > >> > >>> > >> > > > > > > > for batching actually guarantees > >> that > >> > > you > >> > > > > > > observe > >> > > > > > > >> > the > >> > > > > > > >> > >>> > >> limit? I > >> > > > > > > >> > >>> > >> > > > think > >> > > > > > > >> > >>> > >> > > > > > that > >> > > > > > > >> > >>> > >> > > > > > > > implies that compression always > >> makes > >> > > the > >> > > > > > > >> messages > >> > > > > > > >> > >>> > smaller, > >> > > > > > > >> > >>> > >> > > which i > >> > > > > > > >> > >>> > >> > > > > > think > >> > > > > > > >> > >>> > >> > > > > > > > usually true but is not > >> guaranteed, > >> > > > right? > >> > > > > > e.g. > >> > > > > > > >> if > >> > > > > > > >> > >>> someone > >> > > > > > > >> > >>> > >> > > encrypts > >> > > > > > > >> > >>> > >> > > > > > their > >> > > > > > > >> > >>> > >> > > > > > > > data which tends to randomize it > >> and > >> > > then > >> > > > > > > enables > >> > > > > > > >> > >>> > >> > compressesion, > >> > > > > > > >> > >>> > >> > > it > >> > > > > > > >> > >>> > >> > > > > > could > >> > > > > > > >> > >>> > >> > > > > > > > slightly get bigger? > >> > > > > > > >> > >>> > >> > > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > > I also wonder if the rejected > >> > > > alternatives > >> > > > > > you > >> > > > > > > >> > >>> describe > >> > > > > > > >> > >>> > >> > couldn't > >> > > > > > > >> > >>> > >> > > be > >> > > > > > > >> > >>> > >> > > > > > made > >> > > > > > > >> > >>> > >> > > > > > > to > >> > > > > > > >> > >>> > >> > > > > > > > work: basically try to be a bit > >> > better > >> > > at > >> > > > > > > >> estimation > >> > > > > > > >> > >>> and > >> > > > > > > >> > >>> > >> > recover > >> > > > > > > >> > >>> > >> > > > when > >> > > > > > > >> > >>> > >> > > > > > we > >> > > > > > > >> > >>> > >> > > > > > > > guess wrong. I don't think the > >> memory > >> > > > usage > >> > > > > > > >> should > >> > > > > > > >> > be > >> > > > > > > >> > >>> a > >> > > > > > > >> > >>> > >> > problem: > >> > > > > > > >> > >>> > >> > > > > isn't > >> > > > > > > >> > >>> > >> > > > > > it > >> > > > > > > >> > >>> > >> > > > > > > > the same memory usage the > >> consumer of > >> > > > that > >> > > > > > > topic > >> > > > > > > >> > would > >> > > > > > > >> > >>> > need? > >> > > > > > > >> > >>> > >> > And > >> > > > > > > >> > >>> > >> > > > > can't > >> > > > > > > >> > >>> > >> > > > > > > you > >> > > > > > > >> > >>> > >> > > > > > > > do the splitting and > recompression > >> > in a > >> > > > > > > streaming > >> > > > > > > >> > >>> fashion? > >> > > > > > > >> > >>> > >> If > >> > > > > > > >> > >>> > >> > we > >> > > > > > > >> > >>> > >> > > an > >> > > > > > > >> > >>> > >> > > > > > make > >> > > > > > > >> > >>> > >> > > > > > > > the estimation rate low and the > >> > > recovery > >> > > > > cost > >> > > > > > > is > >> > > > > > > >> > just > >> > > > > > > >> > >>> ~2x > >> > > > > > > >> > >>> > >> the > >> > > > > > > >> > >>> > >> > > > normal > >> > > > > > > >> > >>> > >> > > > > > cost > >> > > > > > > >> > >>> > >> > > > > > > > for that batch that should be > >> totally > >> > > > fine, > >> > > > > > > >> right? > >> > > > > > > >> > >>> (It's > >> > > > > > > >> > >>> > >> > > > technically > >> > > > > > > >> > >>> > >> > > > > > true > >> > > > > > > >> > >>> > >> > > > > > > > you might have to split more > than > >> > once, > >> > > > but > >> > > > > > > since > >> > > > > > > >> > you > >> > > > > > > >> > >>> > halve > >> > > > > > > >> > >>> > >> it > >> > > > > > > >> > >>> > >> > > each > >> > > > > > > >> > >>> > >> > > > > > time > >> > > > > > > >> > >>> > >> > > > > > > I > >> > > > > > > >> > >>> > >> > > > > > > > think should you get a number of > >> > > halvings > >> > > > > > that > >> > > > > > > is > >> > > > > > > >> > >>> > >> logarithmic > >> > > > > > > >> > >>> > >> > in > >> > > > > > > >> > >>> > >> > > > the > >> > > > > > > >> > >>> > >> > > > > > miss > >> > > > > > > >> > >>> > >> > > > > > > > size, which, with better > >> estimation > >> > > you'd > >> > > > > > hope > >> > > > > > > >> would > >> > > > > > > >> > >>> be > >> > > > > > > >> > >>> > >> super > >> > > > > > > >> > >>> > >> > > duper > >> > > > > > > >> > >>> > >> > > > > > > small). > >> > > > > > > >> > >>> > >> > > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > > Alternatively maybe we could > work > >> on > >> > > the > >> > > > > > other > >> > > > > > > >> side > >> > > > > > > >> > >>> of the > >> > > > > > > >> > >>> > >> > > problem > >> > > > > > > >> > >>> > >> > > > > and > >> > > > > > > >> > >>> > >> > > > > > > try > >> > > > > > > >> > >>> > >> > > > > > > > to make it so that a small miss > on > >> > > > message > >> > > > > > size > >> > > > > > > >> > isn't > >> > > > > > > >> > >>> a > >> > > > > > > >> > >>> > big > >> > > > > > > >> > >>> > >> > > > problem. > >> > > > > > > >> > >>> > >> > > > > I > >> > > > > > > >> > >>> > >> > > > > > > > think original issue was that > max > >> > size > >> > > > and > >> > > > > > > fetch > >> > > > > > > >> > size > >> > > > > > > >> > >>> were > >> > > > > > > >> > >>> > >> > > tightly > >> > > > > > > >> > >>> > >> > > > > > > coupled > >> > > > > > > >> > >>> > >> > > > > > > > and the way memory in the > consumer > >> > > worked > >> > > > > you > >> > > > > > > >> really > >> > > > > > > >> > >>> > wanted > >> > > > > > > >> > >>> > >> > fetch > >> > > > > > > >> > >>> > >> > > > > size > >> > > > > > > >> > >>> > >> > > > > > to > >> > > > > > > >> > >>> > >> > > > > > > > be as small as possible because > >> you'd > >> > > use > >> > > > > > that > >> > > > > > > >> much > >> > > > > > > >> > >>> memory > >> > > > > > > >> > >>> > >> per > >> > > > > > > >> > >>> > >> > > > > fetched > >> > > > > > > >> > >>> > >> > > > > > > > partition and the consumer would > >> get > >> > > > stuck > >> > > > > if > >> > > > > > > its > >> > > > > > > >> > >>> fetch > >> > > > > > > >> > >>> > size > >> > > > > > > >> > >>> > >> > > wasn't > >> > > > > > > >> > >>> > >> > > > > big > >> > > > > > > >> > >>> > >> > > > > > > > enough. I think we made some > >> progress > >> > > on > >> > > > > that > >> > > > > > > >> issue > >> > > > > > > >> > >>> and > >> > > > > > > >> > >>> > >> maybe > >> > > > > > > >> > >>> > >> > > more > >> > > > > > > >> > >>> > >> > > > > > could > >> > > > > > > >> > >>> > >> > > > > > > be > >> > > > > > > >> > >>> > >> > > > > > > > done there so that a small bit > of > >> > > > fuzziness > >> > > > > > > >> around > >> > > > > > > >> > the > >> > > > > > > >> > >>> > size > >> > > > > > > >> > >>> > >> > would > >> > > > > > > >> > >>> > >> > > > not > >> > > > > > > >> > >>> > >> > > > > > be > >> > > > > > > >> > >>> > >> > > > > > > an > >> > > > > > > >> > >>> > >> > > > > > > > issue? > >> > > > > > > >> > >>> > >> > > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > > -Jay > >> > > > > > > >> > >>> > >> > > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > > On Tue, Feb 21, 2017 at 12:30 > PM, > >> > > Becket > >> > > > > Qin > >> > > > > > < > >> > > > > > > >> > >>> > >> > > becket....@gmail.com > >> > > > > > > >> > >>> > >> > > > > > >> > > > > > > >> > >>> > >> > > > > > > wrote: > >> > > > > > > >> > >>> > >> > > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > > > Hi folks, > >> > > > > > > >> > >>> > >> > > > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > > > I would like to start the > >> > discussion > >> > > > > thread > >> > > > > > > on > >> > > > > > > >> > >>> KIP-126. > >> > > > > > > >> > >>> > >> The > >> > > > > > > >> > >>> > >> > KIP > >> > > > > > > >> > >>> > >> > > > > > propose > >> > > > > > > >> > >>> > >> > > > > > > > > adding a new configuration to > >> > > > > KafkaProducer > >> > > > > > > to > >> > > > > > > >> > allow > >> > > > > > > >> > >>> > >> batching > >> > > > > > > >> > >>> > >> > > > based > >> > > > > > > >> > >>> > >> > > > > > on > >> > > > > > > >> > >>> > >> > > > > > > > > uncompressed message size. > >> > > > > > > >> > >>> > >> > > > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > > > Comments are welcome. > >> > > > > > > >> > >>> > >> > > > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > > > The KIP wiki is following: > >> > > > > > > >> > >>> > >> > > > > > > > > > https://cwiki.apache.org/confl > >> > > > > > > >> > >>> uence/display/KAFKA/KIP- > >> > > > > > > >> > >>> > >> > > > > > > > > 126+-+Allow+KafkaProducer+to+b > >> > > > > > > >> > >>> > >> atch+based+on+uncompressed+siz > >> > > > > > > >> > >>> > >> > e > >> > > > > > > >> > >>> > >> > > > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > > > Thanks, > >> > > > > > > >> > >>> > >> > > > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > > > Jiangjie (Becket) Qin > >> > > > > > > >> > >>> > >> > > > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > > >> > > > > > > >> > >>> > >> > > > > > > >> > > > > > > >> > >>> > >> > > > > > >> > > > > > > >> > >>> > >> > > > > >> > > > > > > >> > >>> > >> > > > > >> > > > > > > >> > >>> > >> > > > > >> > > > > > > >> > >>> > >> > > > -- > >> > > > > > > >> > >>> > >> > > > -Regards, > >> > > > > > > >> > >>> > >> > > > Mayuresh R. Gharat > >> > > > > > > >> > >>> > >> > > > (862) 250-7125 > >> > > > > > > >> > >>> > >> > > > > >> > > > > > > >> > >>> > >> > > > >> > > > > > > >> > >>> > >> > > >> > > > > > > >> > >>> > >> > > >> > > > > > > >> > >>> > >> > > >> > > > > > > >> > >>> > >> > -- > >> > > > > > > >> > >>> > >> > -Regards, > >> > > > > > > >> > >>> > >> > Mayuresh R. Gharat > >> > > > > > > >> > >>> > >> > (862) 250-7125 > >> > > > > > > >> > >>> > >> > > >> > > > > > > >> > >>> > >> > >> > > > > > > >> > >>> > > > >> > > > > > > >> > >>> > > > >> > > > > > > >> > >>> > > >> > > > > > > >> > >>> > >> > > > > > > >> > >> > >> > > > > > > >> > >> > >> > > > > > > >> > > > >> > > > > > > >> > > >> > > > > > > >> > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > > > >
