Hello,

  Playing around with the 0.8.2-beta producer client. One of my test cases
is to ensure producers can deal with Kafka being down when the producer is
created. My tests failed miserably because of the default blocking in the
producer with regard to metadata.fetch.timeout.ms. The first line of new
producer is waitOnMetadata which is blocking.

I can handle this case by loading topic meta on init and setting the
timeout value to very low metadata.fetch.timeout.ms and either throwing
away messages or creating my own internal queue to buffer.

I’m surprised the metasync isn’t done async. If it fails, return that in
the future/callback. This way the API could actually be considered safely
async and the producer buffer could try to hold on to things until
block.on.buffer.full kicks in to either drop messages or block. You’d
probably need a partition callback since numPartitions wouldn’t be
available.

The implication is that people's apps will work fine if first messages are
sent while kafka server is up, however, if kafka is down and they restart
their app, the new producer will block all sends and blow things up if you
haven't written your app to be aware of this edge case.


Thanks,

Paul

Reply via email to