Yes, that's a potential issue. Perhaps we just need to have a lower default value for metadata.fetch.timeout.ms ?
Thanks, Jun On Wed, Dec 17, 2014 at 11:10 PM, Paul Pearcy <paul.pea...@blackboard.com> wrote: > Heya, > Playing around with the 0.8.2-beta producer client. One of my test cases > is to ensure producers can deal with Kafka being down when the producer is > created. My tests failed miserably because of the default blocking in the > producer with regard to metadata.fetch.timeout.ms. The first line of new > producer is waitOnMetadata which is blocking. > > I can handle this case by loading topic meta on init and setting the > timeout value to very low metadata.fetch.timeout.ms and either throwing > away messages or creating my own internal queue to buffer. > > I'm surprised the metasync isn't done async. If it fails, return that in > the future/callback. This way the API could actually be considered safely > async and the producer buffer could try to hold on to things until > block.on.buffer.full kicks in. You'd probably need a partition callback > since numPartitions wouldn't be available. > > The implication is that people's apps will work fine if first messages are > sent while kafka server is up, however, if kafka is down and they restart > their app, the new producer will block all sends and blow things up if you > haven't written your app to be aware of this edge case. > > Thanks, > Paul > This email and any attachments may contain confidential and proprietary > information of Blackboard that is for the sole use of the intended > recipient. If you are not the intended recipient, disclosure, copying, > re-distribution or other use of any of this information is strictly > prohibited. Please immediately notify the sender and delete this > transmission if you received this email in error. >
