Hi,
thanks for sharing this and getting feedback. Sorry I am probably missing
something basic, but I'm not sure how a multi-threaded consumer would
work. I can imagine its either:
a) I just have one thread poll kafka. If I want to process msgs in
multiple threads, than I deal w/ that after pol
You can use ConsumerOffsetChecker
https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/tools/ConsumerOffsetChecker.scala
(thats a link to the 0.8 code, but its also in 0.7)
it just prints current offsets to screen, but you can look at the code
to see what you'd do to just the offsets
trying to make
> consumer apps a bit more proactive in that case, if not using the default
> autocommit.
>
> Interesting stuff
>
> Jason
>
>
> On Sat, Nov 23, 2013 at 12:55 AM, Imran Rashid wrote:
>
>> MessageAndMetadata includes the partition and offset along
hread is
> currently 'owning', and therefore, you don't know which partitions you can
> commit to.
>
>
> On Fri, Nov 22, 2013 at 5:28 PM, Imran Rashid wrote:
>
>> I don't think I need control over which partitions are processed by a
>> thread --
gt; meaning per partition, doesn't really scale with this design.
>
> Jason
>
>
> On Fri, Nov 22, 2013 at 9:06 AM, Imran Rashid wrote:
>
>> hmm, ok, after some more thought I see what you mean.I was thinking
>> async because of a more complicated use case -- my
in, a single offset
> commit wouldn't make much sense.
>
> Jason
>
>
> On Fri, Nov 22, 2013 at 1:30 AM, Imran Rashid wrote:
>
>> Hi Jason,
>>
>> thank you so much!
>>
>> I was missing the consumer.timeout.ms property. In fact I had
>>
> I have a configuration like this. I use a default commit cycle of 60
> seconds, and use 100ms timeout on the consumer iterator.
>
> The auto-commit mode built in has the fatal flaw, as you have pointed out,
> of possibly dropping a single message on a restart.
>
> Jason
&g
t; why not just disable autocommit and only call commit offsets() after you've
> processed a batch? it isn't obvious to me how doing so would allow a
> message to be processed zero times.
> On Nov 21, 2013 5:52 PM, "Imran Rashid" wrote:
>
>> Hi Edward,
>>
&
r the
>>
>> b = iter.next()
>>
>> is called, at-least-once is guaranteed.
>>
>> Does that make sense?
>>
>> Guozhang
>>
>>
>> On Thu, Nov 21, 2013 at 2:14 PM, Imran Rashid
>> wrote:
>>
>> > sorry to keep bugging the
-once is guaranteed.
>
> Does that make sense?
>
> Guozhang
>
>
> On Thu, Nov 21, 2013 at 2:14 PM, Imran Rashid wrote:
>
>> sorry to keep bugging the list, but I feel like I am either missing
>> something important, or I'm finding something wrong w/ the s
sorry to keep bugging the list, but I feel like I am either missing
something important, or I'm finding something wrong w/ the standard
consumer api, (or maybe just the docs need some clarification).
I started to think that I should probably just accept at least once
semantics ... but I eventually
before the offset is
committed to zookeeper). I guess you probably wouldn't want read
exactly once w/out batch processing, or at least very low rate of
messages, to avoid a ton of zookeeper updates. Hopefully this helps
explain the use case more.
thanks
On Thu, Nov 21, 2013 at 12:19 PM, Imra
roughput, since workers can
keep going during batch processing).
Would such a change be reasonable? I could probably submit a patch for it.
On Wed, Nov 20, 2013 at 9:05 AM, Imran Rashid wrote:
> perfect, thank you!
>
> On Wed, Nov 20, 2013 at 8:44 AM, Neha Narkhede
> wrote:
>>
; Neha
> On Nov 20, 2013 6:39 AM, "Imran Rashid" wrote:
>
>> Hi,
>>
>> I have an application which reads messages from a kafka queue, builds
>> up a batch of messages, and then performs some action on that batch.
>> So far I have just used the ConsumerGroup
Hi,
I have an application which reads messages from a kafka queue, builds
up a batch of messages, and then performs some action on that batch.
So far I have just used the ConsumerGroup api. However, I realized
there is a potential problem -- my app may die sometime in the middle
of the batch, and
15 matches
Mail list logo
