Jay, thanks a lot for the quick response.
For #2, I do see some isSendable() check in Sender.java and
NetworkClient.java that is eventually mapped to
InFlightRequests#canSendMore() check. it wasn't immediately clear to me how
is that translated to transmissions.hasSend() check. anyway, I will dig a
little more.
On Mon, Nov 10, 2014 at 2:59 PM, Jay Kreps <jay.kr...@gmail.com> wrote:
> 1. No, the send does a poll/select on all the connections that will block
> for a specified time waiting for data to read or write on any connection.
> 2. The api of the selector only allows you to send a request to a ready
> connection. The definition of ready is that it doesn't have another request
> in the process of being written (it can have other requests outstanding
> that were previously written). So if you hit this case it is a programming
> error, and it should never happen in the producer. The write path for the
> data is it is written to the internal queue/buffer and the sender grabs
> data from that for ready connections and writes to them. This slightly
> complex ready/send api is required to allow back-pressure in the producer
> to work.
>
> -Jay
>
> On Mon, Nov 10, 2014 at 2:44 PM, Steven Wu <stevenz...@gmail.com> wrote:
>
> > I am checking out the source code of 0.8.2 producer code. I have two
> > questions and hope to get some clarifications.
> >
> > 1) Sender thread/Runnable has this run loop. what if the in-memory queue
> is
> > mostly empty (i.e. producer has very few msgs to send out)? will this
> > become a simple tight loop that just wastes cpu?
> >
> > // main loop, runs until close is called
> > while (running) {
> > try {
> > run(time.milliseconds());
> > } catch (Exception e) {
> > log.error("Uncaught error in kafka producer I/O thread:
> ",
> > e);
> > }
> > }
> >
> > 2) Selector#poll() [line #200]
> >
> > if (transmissions.hasSend())
> > throw new IllegalStateException("Attempt to begin a send
> > operation with prior send operation still in progress.");
> >
> > 2.1) since it's aysnc API, back-to-back sends to the same broker seem
> very
> > possible to me. should we throw an exception here?
> > 2.2) if this happens, it seems that we will silently drop the msgs
> without
> > executing callbacks?
> >
> >
> > Thanks,
> > Steven
> >
>
