Re: Simple Consumer and offsets

2015-02-23 Thread Joel Koshy
> Assuming one can store the offset in an external location (redis/db > etc), along with the rest of the state that a program requires, > wouldn't it be possible to manage things such that, you use the High > Level API with auto commit turned off and do your custom offset > management followed by t

Re: Simple Consumer and offsets

2015-02-23 Thread Jun Rao
We are developing a new java consumer that addresses the offset reset issue better. Thanks, Jun On Thu, Feb 19, 2015 at 11:53 PM, Arunkumar Srambikkal (asrambik) < asram...@cisco.com> wrote: > If I may use the same thread to discuss the exact same issue > > Assuming one can store the offse

RE: Simple Consumer and offsets

2015-02-19 Thread Arunkumar Srambikkal (asrambik)
If I may use the same thread to discuss the exact same issue Assuming one can store the offset in an external location (redis/db etc), along with the rest of the state that a program requires, wouldn't it be possible to manage things such that, you use the High Level API with auto commit t

Re: Simple Consumer and offsets

2015-02-19 Thread Joel Koshy
Yeah that is a good point - will do the update as part of the doc changes in KAFKA-1729 On Thu, Feb 19, 2015 at 09:26:30PM -0500, Evan Huus wrote: > On Thu, Feb 19, 2015 at 8:43 PM, Joel Koshy wrote: > > > If you are using v0 of OffsetCommit/FetchRequest then you can issue > > that to any broker

Re: Simple Consumer and offsets

2015-02-19 Thread Evan Huus
On Thu, Feb 19, 2015 at 8:43 PM, Joel Koshy wrote: > If you are using v0 of OffsetCommit/FetchRequest then you can issue > that to any broker. For version > 0 you will need to issue it to the > coordinator. You can discover the coordinator by sending a > ConsumerMetadataRequest to any broker. >

Re: Simple Consumer and offsets

2015-02-19 Thread Joel Koshy
If you are using v0 of OffsetCommit/FetchRequest then you can issue that to any broker. For version > 0 you will need to issue it to the coordinator. You can discover the coordinator by sending a ConsumerMetadataRequest to any broker. On Thu, Feb 19, 2015 at 07:55:16PM +, Suren wrote: > Joel/A

Re: Simple Consumer and offsets

2015-02-19 Thread Suren
Joel/All, The SimpleConsumer constructor requires a specific host and port. Can this be any broker? If it needs to be a specific broker, for 0.8.2, should this be the offset coordinator? For 0.8.1, does it matter? -Suren On Thursday, February 19, 2015 10:43 AM, Joel Koshy wrote: I

Re: Simple Consumer and offsets

2015-02-19 Thread Joel Koshy
I see - yes, you can use the SimpleConsumer for that. However, your high-level consumers need to be shutdown while you do that (otherwise they may auto-commit while you are resetting offsets). Thanks, Joel On Thu, Feb 19, 2015 at 03:29:19PM +, Suren wrote: > We are using the High Level Consu

Re: Simple Consumer and offsets

2015-02-19 Thread Suren
We are using the High Level Consumer API to interact with Kafka for our normal use cases. However, on consumer restart in the case of consumer failures, we want to be able to manually reset offsets in certain situations. And ideally we'd like to use the same api in 0.8.1 and 0.8.2. :-) It looked

Re: Simple Consumer and offsets

2015-02-19 Thread Joel Koshy
Not sure what you mean by using the SimpleConsumer on failure recovery. Can you elaborate on this? On Thu, Feb 19, 2015 at 03:04:47PM +, Suren wrote: > Haven't used either one now. Sounds like 0.8.2.1 will help. > We are using the High Level Consumer generally but are thinking to use the > Si

Re: Simple Consumer and offsets

2015-02-19 Thread Suren
Haven't used either one now. Sounds like 0.8.2.1 will help. We are using the High Level Consumer generally but are thinking to use the SimpleConsumer on failure recovery to set the offsets. Is that the recommended approach for this use case? Thanks. -Suren On Thursday, February 19, 2015 9:

Re: Simple Consumer and offsets

2015-02-19 Thread Joel Koshy
Are you using it from Java or Scala? i.e., are you using the javaapi.SimpleConsumer or kafka.consumer.SimpleConsumer In 0.8.2 javaapi we explicitly set version 0 of the OffsetCommitRequest/OffsetFetchRequest which means it will commit/fetch to/from ZooKeeper only. If you use the scala API you can

Re: Simple Consumer and offsets

2015-02-19 Thread Suren
Joel, Looking at SimpleConsumer in the 0.8.2 code, it is using OffsetCommitRequest and sending that over to a broker. Is the broker storing that in ZK? -Suren On Tuesday, February 17, 2015 12:22 PM, Joel Koshy wrote: Hi Chris, In 0.8.2, the simple consumer Java API supports commit

Re: Simple Consumer and offsets

2015-02-17 Thread Joel Koshy
Hi Chris, In 0.8.2, the simple consumer Java API supports committing/fetching offsets that are stored in ZooKeeper. You don't need to issue any ConsumerMetadataRequest for this. Unfortunately, the API currently does not support fetching offsets that are stored in Kafka. Thanks, Joel On Mon, Feb