Hey Becket, Good point! Thanks for the comment.
I have updated the KIP to move the compression to user thread in the common case. Basically user thread can be responsible for compressing and moving messages from per-topic queue to per-partition queue once the metadata is available. Only if IO threads has nothing to do (e.g. all messages in the per-partition queue has been sent), then the IO thread can try to compress and move some messages to the per-partition queue. Can you see if the latest solution in the KIP address the problem? Also, I have added a new section to analyze how the changes proposed in this KIP would change the performance of producer. Thanks, Dong On Fri, Apr 13, 2018 at 11:01 AM, Becket Qin <becket....@gmail.com> wrote: > Thanks for the KIP, Dong. > > In the current threading model, compression is done by the user threads, > therefore the producer sender thread can focus on IO. With the proposed > changes, does that mean the producer sender thread will have to do all the > compression as well? Would this become a performance bottleneck? > > Thanks, > > Jiangjie (Becket) Qin > > On Thu, Apr 12, 2018 at 1:41 AM, Dong Lin <lindon...@gmail.com> wrote: > > > Hey Ted, > > > > Thanks for your comments. With the proposed solution in the KIP, the > memory > > is only allocated once for the given message, which is the same as the > > existing implementation. The serialized message will be moved from > > per-topic queue to per-partition queue without incurring additional > memory > > overhead. Does this address your question? > > > > Thanks, > > Dong > > > > > > On Wed, Apr 11, 2018 at 9:36 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > > > Looks like per-topic queue is introduced. > > > > > > In terms of memory consumption, how does the KIP allocate memory > > > between per-topic > > > queue and per-partition queue ? > > > > > > Thanks > > > > > > On Wed, Apr 11, 2018 at 8:50 PM, Dong Lin <lindon...@gmail.com> wrote: > > > > > > > Hi all, > > > > > > > > I have created KIP-286: producer.send() should not block on metadata > > > > update. See > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > > 286%3A+producer.send%28%29+should+not+block+on+metadata+update > > > > . > > > > > > > > The KIP intends to improve user-experience of producer.send() when > > > metadata > > > > is not available. It is related but different from the previous > > > discussion > > > > in KAFKA-3539 in the sense that user still has the option of letting > > > > producer.send() block on full producer queue. > > > > > > > > Comments are welcome! > > > > > > > > Thanks, > > > > Dong > > > > > > > > > >
