Hi John, I should preface this by saying I've never used Storm and KafkaBolt and am not a streaming expert.
However, if you're running out of buffer in the producer (as is what's happening in the other thread you referenced), you can possibly alleviate this by adding more producers, or by tuning the producers. Tuning the brokers or adding more brokers may help as well, but it's hard to say for sure without looking at your monitors and knowing more about the use case and cluster. I suggest setting up monitoring and looking deeply at the JMX metrics that are created to understand where each message spends most of its time (producer, broker, consumer to start, then request queues, io threads, etc). The docs go through each JMX metric relevant here. Then from there you can start understanding how to alleviate the problem. Feel free to share metrics and more information and we can explore them together. Alex On Thu, Feb 18, 2016 at 5:18 AM, John Yost <hokiege...@gmail.com> wrote: > Hi Everyone, > > I am encountering this exception similar to Saurabh's report earlier today > as I try to scale up a Storm -> Kafka output via the KafkaBolt (i.e., add > more KafkaBolt executors). > > Question...does this necessarily indicate back pressure from Kafka where > the Kafka writes cannot keep up with the incoming messages sent over by > Storm? If so, do I add brokers to the cluster, do I add more topics, a > combo thereof or something else? > > As always, any thoughts from people who know more than I do are > appreciated. :) > > Thanks > > --John > -- *Alex Loddengaard | **Solutions Architect | Confluent* *Download Apache Kafka and Confluent Platform: www.confluent.io/download <http://www.confluent.io/download>*
