Hi Jay, Yeah, I got your point.
I think there might be a solution which do not require adding a new configuration. We can start from a very conservative compression ratio say 1.0 and lower it very slowly according to the actual compression ratio until we hit a point that we have to split a batch. At that point, we exponentially back off on the compression ratio. The idea is somewhat like TCP. This should help avoid frequent split. The upper bound of the batch size is also a little awkward today because we say the batch size is based on compressed size, but users cannot set it to the max message size because that will result in oversized messages. With this change we will be able to allow the users to set the message size to close to max message size. However the downside is that there could be latency spikes in the system in this case due to the splitting, especially when there are many messages need to be split at the same time. That could potentially be an issue for some users. What do you think about this approach? Thanks, Jiangjie (Becket) Qin On Thu, Feb 23, 2017 at 1:31 PM, Jay Kreps <j...@confluent.io> wrote: > Hey Becket, > > Yeah that makes sense. > > I agree that you'd really have to both fix the estimation (i.e. make it per > topic or make it better estimate the high percentiles) AND have the > recovery mechanism. If you are underestimating often and then paying a high > recovery price that won't fly. > > I think you take my main point though, which is just that I hate to exposes > these super low level options to users because it is so hard to explain to > people what it means and how they should set it. So if it is possible to > make either some combination of better estimation and splitting or better > tolerance of overage that would be preferrable. > > -Jay > > On Thu, Feb 23, 2017 at 11:51 AM, Becket Qin <becket....@gmail.com> wrote: > > > @Dong, > > > > Thanks for the comments. The default behavior of the producer won't > change. > > If the users want to use the uncompressed message size, they probably > will > > also bump up the batch size to somewhere close to the max message size. > > This would be in the document. BTW the default batch size is 16K which is > > pretty small. > > > > @Jay, > > > > Yeah, we actually had debated quite a bit internally what is the best > > solution to this. > > > > I completely agree it is a bug. In practice we usually leave some > headroom > > to allow the compressed size to grow a little if the the original > messages > > are not compressible, for example, 1000 KB instead of exactly 1 MB. It is > > likely safe enough. > > > > The major concern for the rejected alternative is performance. It largely > > depends on how frequent we need to split a batch, i.e. how likely the > > estimation can go off. If we only need to the split work occasionally, > the > > cost would be amortized so we don't need to worry about it too much. > > However, it looks that for a producer with shared topics, the estimation > is > > always off. As an example, consider two topics, one with compression > ratio > > 0.6 the other 0.2, assuming exactly same traffic, the average compression > > ratio would be roughly 0.4, which is not right for either of the topics. > So > > almost half of the batches (of the topics with 0.6 compression ratio) > will > > end up larger than the configured batch size. When it comes to more > topics > > such as mirror maker, this becomes more unpredictable. To avoid frequent > > rejection / split of the batches, we need to configured the batch size > > pretty conservatively. This could actually hurt the performance because > we > > are shoehorn the messages that are highly compressible to a small batch > so > > that the other topics that are not that compressible will not become too > > large with the same batch size. At LinkedIn, our batch size is configured > > to 64 KB because of this. I think we may actually have better batching if > > we just use the uncompressed message size and 800 KB batch size. > > > > We did not think about loosening the message size restriction, but that > > sounds a viable solution given that the consumer now can fetch oversized > > messages. One concern would be that on the broker side oversized messages > > will bring more memory pressure. With KIP-92, we may mitigate that, but > the > > memory allocation for large messages may not be very GC friendly. I need > to > > think about this a little more. > > > > Thanks, > > > > Jiangjie (Becket) Qin > > > > > > On Wed, Feb 22, 2017 at 8:57 PM, Jay Kreps <j...@confluent.io> wrote: > > > > > Hey Becket, > > > > > > I get the problem we want to solve with this, but I don't think this is > > > something that makes sense as a user controlled knob that everyone > > sending > > > data to kafka has to think about. It is basically a bug, right? > > > > > > First, as a technical question is it true that using the uncompressed > > size > > > for batching actually guarantees that you observe the limit? I think > that > > > implies that compression always makes the messages smaller, which i > think > > > usually true but is not guaranteed, right? e.g. if someone encrypts > their > > > data which tends to randomize it and then enables compressesion, it > could > > > slightly get bigger? > > > > > > I also wonder if the rejected alternatives you describe couldn't be > made > > to > > > work: basically try to be a bit better at estimation and recover when > we > > > guess wrong. I don't think the memory usage should be a problem: isn't > it > > > the same memory usage the consumer of that topic would need? And can't > > you > > > do the splitting and recompression in a streaming fashion? If we an > make > > > the estimation rate low and the recovery cost is just ~2x the normal > cost > > > for that batch that should be totally fine, right? (It's technically > true > > > you might have to split more than once, but since you halve it each > time > > I > > > think should you get a number of halvings that is logarithmic in the > miss > > > size, which, with better estimation you'd hope would be super duper > > small). > > > > > > Alternatively maybe we could work on the other side of the problem and > > try > > > to make it so that a small miss on message size isn't a big problem. I > > > think original issue was that max size and fetch size were tightly > > coupled > > > and the way memory in the consumer worked you really wanted fetch size > to > > > be as small as possible because you'd use that much memory per fetched > > > partition and the consumer would get stuck if its fetch size wasn't big > > > enough. I think we made some progress on that issue and maybe more > could > > be > > > done there so that a small bit of fuzziness around the size would not > be > > an > > > issue? > > > > > > -Jay > > > > > > > > > > > > On Tue, Feb 21, 2017 at 12:30 PM, Becket Qin <becket....@gmail.com> > > wrote: > > > > > > > Hi folks, > > > > > > > > I would like to start the discussion thread on KIP-126. The KIP > propose > > > > adding a new configuration to KafkaProducer to allow batching based > on > > > > uncompressed message size. > > > > > > > > Comments are welcome. > > > > > > > > The KIP wiki is following: > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > 126+-+Allow+KafkaProducer+to+batch+based+on+uncompressed+size > > > > > > > > Thanks, > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > >
