Re: producer exceptions when broker dies

2013-10-27 Thread Jason Rosenberg
Perhaps, it's a matter of semantics. But, I think I'm not talking only about failure, but normal operation. It's normal to take a cluster down for maintenance, or code update. And this should be done a rolling restart manner (1 server at a time). The reason for replication, is to increase relia

How to commit offset every time, automatically

2013-10-27 Thread Cheolwoo Choi
Hi, all I'm using kafka-0.7.2 I'd like to commit offset every time reading a message block. "autocommit.interval.ms" property is not enough. To do that, should I use SimpleConsumer, and write a commit logic manually? (i.e. zkUtils.commitOffset(groupId, messageAndOffset.offset() ) Is there no way

Re: How to commit offset every time, automatically

2013-10-27 Thread Philip O'Toole
You have two choices. -- Do what you say, and write your own consumer, based on the SimpleConsumer. Handle all commits, ZK accesses, and balancing yourself. -- Use a ConsumerConnector for every partition, and call commitOffsets() explicitly when you have processed a message. This does a commit fo

Re: How to commit offset every time, automatically

2013-10-27 Thread Cheolwoo Choi
Thank you for your reply. The 2nd one is what I want. I has tested with it, and it works fine. Thanks. I hope it supports 'consumerConnector.gracefulShutdown()' to finalize a connector gracefully (with some user properties). :-) On Mon, Oct 28, 2013 at 10:16 AM, Philip O'Toole wrote: > You

Re: Kafka 0.8 Java API

2013-10-27 Thread Jun Rao
You can find the api doc in http://kafka.apache.org/documentation.html#api The 0.8 beta1 release is available from http://kafka.apache.org/downloads.html Thanks, Jun On Sat, Oct 26, 2013 at 1:57 PM, Smith, Kieran R < kieran.r.sm...@jpmorgan.com> wrote: > Hi, > > Is there an eta on the 0.8 Kaf