Which version of ZK are you using? Thanks,
Jun On Fri, Jun 7, 2013 at 12:20 PM, Evan Chan <e...@ooyala.com> wrote: > [ Sorry if this mail is duplicated, this is my fourth try sending this > message] > > Hey guys, > > I sincerely apologize if this has been covered before, I haven't quite > found a similar situation. > > We are using Kafka 0.7.2 in production, and we are using the ZK high level > Scala consumer. However, we find the ZK consumer very unstable. It would > work for one or two weeks, then suddenly it would complain about ZK nodes > disappearing, and one consumer would die, then another, then another, until > our pipeline is no longer pulling any data. There are multiple > NullPointerExceptions, and other problems. We can restart it, but it > does not stay up predictably. > > On the other hand, I have a simple app which I wrote using the simple > consumer to mirror select partitions (will blog about this later) and it > just works flawlessly. > > So we are faced with a dilemma to get back on track: > 1) Use SimpleConsumer, and write our own balancing code (but honestly our > boxes almost never go down, compared to the rate of ZK mishaps) > 2) Upgrade to Kafka 0.8 and hope that that resolves the issue. > > There seem to be so many improvements in 0.8 that that seems to be the > biggest win long-term, so I am wondering if people can comment on: > - has anyone tried using 0.8 in production? Is it stable yet? > - How much more stable is the ZK consumer in 0.8? > - will it be possible to change the offset in the 0.8 consumer? That was > the other reason why we wanted to move to SimpleConsumer. > > thanks, > Evan >
