Hi Senthil, We never did get to the root cause unfortunately. Our kafka cluster is currently owned by a different team and it wasn't a priority for them to figure that out, or at least communicate the problem to us :(
Cheers, Craig On Thu, Jul 19, 2018 at 8:13 AM SenthilKumar K <senthilec...@gmail.com> wrote: > Hi Craig Ching, > > Reg. *We did end up turning on debug logs for the console consumer and > found that one broker seemed to be having problems, it would lead to > timeouts communicating with it. After restarting that broker, things > sorted themselves out.* > > We had similar problem on prod cluster and i'm trying to figure out the > root cause for why broker stopped responding ? Pls check my email subject : > *"**Kafka Broker Not Responding" *where i described problem in detail. > > Curious to know , Were you able to figure out reason for broker failure ?. > Of Course , turning off/on is not the ideal solution. > > --Senthil > > > On Thu, Jun 14, 2018 at 9:48 AM, Craig Ching <craigch...@gmail.com> wrote: > > > Hi Manikumar! > > > > Thanks for responding! Sorry it took me so long to get back! > > > > We did end up turning on debug logs for the console consumer and found > > that one broker seemed to be having problems, it would lead to timeouts > > communicating with it. After restarting that broker, things sorted > > themselves out. However, I always hate the “turn off/ turn on” solution > ;) > > It’s interesting to me that the 1.1.0 consumer, though it reported > timeouts > > in the logs, never had a problem, it seemed able to recover. Whereas the > > 1.0.1 consumer (talking to a 1.1.0 cluster remember) couldn’t recover. > > Does any of this make sense? I’m happy to provide more details and logs > if > > necessary as I’d like to understand the root problem here. > > > > Thanks again! > > > > Cheers, > > Craig > > > > > On Jun 13, 2018, at 12:43 AM, Manikumar <manikumar.re...@gmail.com> > > wrote: > > > > > > Can you post consumer debug logs? > > > You can enable console consumer debug logs here: > > > kafka/config/tools-log4j.properties > > > > > > On Wed, Jun 13, 2018 at 9:55 AM Craig Ching <craigch...@gmail.com> > > wrote: > > > > > >> Hi! > > >> > > >> We’re having a problem with a new kafka cluster at 1.1.0. The problem > > is, > > >> in general, that consumers can’t consume from the different broker > (old > > >> broker was 0.11 I think). The easiest recipe I have for reproducing > the > > >> problem is that downloading kafka 1.0.1 and running console consumer > > can’t > > >> consume from the 1.1.0 cluster while a 1.1.0 console consumer can. > > We’re > > >> invoking console consumer like this: > > >> > > >> bin/kafka-console-consumer.sh \ > > >> —bootstrap-server [kafka server] \ > > >> —topic [our topic] \ > > >> —max-messages 3 > > >> > > >> That works for a 1.1.0 console consumer, but not for 1.0.1. However, > if > > >> we change that to: > > >> > > >> bin/kafka-console-consumer.sh \ > > >> —zookeeper [zookeeper server] \ > > >> —topic [our topic] \ > > >> —max-messages 3 > > >> > > >> Then it works for 1.0.1. > > >> > > >> I was wondering, is the zookeeper schema published for 1.1.0? I have > a > > >> feeling that maybe something is wrong in zookeeper and I know earlier > > >> versions of kafka used to publish the zk schema, could there be a > > problem > > >> in zk? if so, what might I look for? > > >> > > >> Any help is greatly appreciated! > > >> > > >> Cheers, > > >> Craig > > > > >
