Hey Moses, I like the direction here. My thinking is that a single additional work queue, s.t. send() can enqueue and return, seems like the lightest touch. However, I don't think we can trivially process that queue in an internal thread pool without subtly changing behavior for some users. For example, users will often run send() in multiple threads in order to serialize faster, but that wouldn't work quite the same if there were an internal thread pool.
For this reason I'm thinking we need to make sure any such changes are opt-in. Maybe a new constructor with an additional ThreadFactory parameter. That would at least clearly indicate that work will happen off-thread, and would require opt-in for the new behavior. Under the hood, this ThreadFactory could be used to create the worker thread that process queued sends, which could fan-out to per-partition threads from there. So then you'd have two ways to send: the existing way, where serde and interceptors and whatnot are executed on the calling thread, and the new way, which returns right away and uses an internal Executor. As you point out, the semantics would be identical in either case, and it would be very easy for clients to switch. Ryanne On Thu, May 13, 2021, 9:00 AM Nakamura <nny...@gmail.com> wrote: > Hey Folks, > I just posted a new proposal > < > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181306446 > > > in the wiki. I think we have an opportunity to improve the > KafkaProducer#send user experience. It would certainly make our lives > easier. Please take a look! There are two subproblems that I could use > guidance on, so I would appreciate feedback on both of them. > Best, > Moses >
