Feb 24, 2014 at 8:18 AM, Withers, Robert
mailto:robert.with...@dish.com>>wrote:
Jun,
Are you saying it is possible to get events from the high-level consumer
regarding various state machine changes? For instance, can we get a
notification when a rebalance starts and ends, when a partition
can just call
those methods based on the events you get.
Also, we already have an api to get the first and the last offset of a
partition (getOffsetBefore).
Thanks,
Jun
On Sat, Feb 22, 2014 at 11:29 AM, Withers, Robert
wrote:
> This is a good idea, too. I would modify it to include str
We use kafka as a durable buffer for 3rd party event traffic. It acts as the
event source in a lambda architecture. We want it to be exactly once and we
are close, though we can lose messages aggregating for Hadoop. To really tie
this all together, I think there should be an Apache project to
nk through.
The usage would be something like
long end = consumer.lastOffset(tp);
while(consumer.position < end)
process(consumer.poll());
-Jay
On Sat, Feb 22, 2014 at 1:52 AM, Withers, Robert
mailto:robert.with...@dish.com>>wrote:
Jun,
I was originally thinking a non-blockin
inguish btw FetchingInProgressException
and NoMessagePendingException? The nextMsgs() method that you want is
exactly what poll() does.
Thanks,
Jun
On Wed, Feb 19, 2014 at 8:45 AM, Withers, Robert
mailto:robert.with...@dish.com>>wrote:
I am not clear on why the consumer stream should be positionab
I am not clear on why the consumer stream should be positionable, especially if
it is limited to the in-memory fetched messages. Could someone explain to me,
please? I really like the idea of committing the offset specifically on those
partitions with changed read offsets, only.
2 items I w
The core problem is our consumers stop consuming and lag increases. We found
this blog:
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped,why?.
This lists 3 possibilities.
The blog also talks earlier about spurious rebalances, due to improper GC
setting
We are creating a consumer with properties and I did not see a property that
screamed that it was to start at the beginning of a topic. Is there such a
property?
Thanks,
rob
[cid:image001.png@01CE891F.54E75000]
Rob Withers
Staff Analyst/Developer
o: (720) 514-8963
c: (571) 262-1873
l be great if you are interested in finishing it :)
Tim
On Tue, Jun 18, 2013 at 8:23 AM, Withers, Robert wrote:
> I see an old C# client, which is 2 years old. Does anyone have a C#
> client that works with the kafka 0.8 producer?
>
> Thanks,
> rob
>
>
I see an old C# client, which is 2 years old. Does anyone have a C# client
that works with the kafka 0.8 producer?
Thanks,
rob
Thanks, Jun. We have considered doing message filtering in the consumer.
However, the thrust of my question below is not filtering, but dispatching. If
we take Chris' recommendation and pump a small set of msg types, belonging to
the same "class" of messages, such as Account History, through
Thanks for the info. Are you saying that even with a single connector, with
say 3 topics and 3 threads per topic and 3 brokers with 3 partitions for all 3
topics on all 3 brokers, that a consumer box would have 9 sockets open? What
if there are 6 partitions per topic, would that be 18 open soc
Ahh, yes, Andrea - it was you who suggested to me to use the sbt plugin
approach. I finally got around to it and this worked perfectly the first time,
other than the scala unit tests. Thanks very much for explaining it to me.
Thanks,
rob
-Original Message-
From: Andrea Gazzarini [m
yed intact.
I got a new feature, which allows auto-purging of logs. This keeps OPS
maintenance to a minimum."
thanks,
rob
-Original Message-----
From: Withers, Robert [mailto:robert.with...@dish.com]
Sent: Friday, May 17, 2013 7:38 AM
To: users@kafka.apache.org
Subject: RE: are
,
Rob Withers
Staff Analyst/Developer
o: (720) 514-8963
c: (571) 262-1873
-Original Message-
From: Withers, Robert [mailto:robert.with...@dish.com]
Sent: Friday, May 17, 2013 11:49 AM
To: users@kafka.apache.org
Subject: RE: could an Encoder/Decoder be stateful?
I shall. Thanks
gt; > it's instantiated through java reflection. Decoder instance can
> > actually be
> shared
> > by different consumers since it's passed in directly through the
> > consumer
> api.
> >
> > Thanks,
> >
> > Jun
> >
> &g
ao [mailto:jun...@gmail.com]
> > Sent: Thursday, May 16, 2013 10:00 PM
> > To: users@kafka.apache.org
> > Subject: Re: API to to query messages amount under one topic
> >
> > What do you plan to use this information for?
> >
> > Thanks,
> >
> > Jun
>
le to shutdown a consumerConnector without flushing the
offset
If you turn off auto.commit.enable, that will ensure that messages are
replayed whenever a consumer starts up and rebalances.
Thanks,
Neha
On May 17, 2013 6:35 AM, "Withers, Robert" wrote:
> Certainly I will try. Our und
> > Can you describe your requirements in a little more detail?
> >
> > Thanks,
> > Neha
> > On May 16, 2013 6:11 AM, "Withers, Robert"
> > wrote:
> >
> > > is it technically feasible to use an only-once simple consumer within
> > > a consumer group?
> > >
> > > thanks,
> > > rob
>
>
,
Neha
On May 17, 2013 6:31 AM, "Withers, Robert" wrote:
> Awesome! Thanks for the clarification. I would like to offer my strong
> vote that this get tackled before a beta, to get it firmly into 0.8.
> Stabilize everything else to the existing use, but make offset updates
&g
more details about what you mean by measuring replay when
you kill a consumer?
Thanks,
Neha
On May 17, 2013 6:26 AM, "Withers, Robert" wrote:
> Would it be possible for someone to provide me with a 0.8 jar that
> implements a ConsumerConnector.hardShutdown, which would interrupt all
Awesome! Thanks for the clarification. I would like to offer my strong vote
that this get tackled before a beta, to get it firmly into 0.8. Stabilize
everything else to the existing use, but make offset updates batched.
thanks,
rob
From: Neha Narkhede
Would it be possible for someone to provide me with a 0.8 jar that implements a
ConsumerConnector.hardShutdown, which would interrupt all threads yet not do a
final offset flush. We want to measure replay so we want to simulate a kill
-9, but we want to keep running the process to flush stats a
is it technically feasible to use an only-once simple consumer within a
consumer group?
thanks,
rob
Excellent news. Thanks, Neha.
From: Neha Narkhede [neha.narkh...@gmail.com]
Sent: Wednesday, May 15, 2013 10:25 PM
To: users@kafka.apache.org
Subject: Re: when do you think 0.8 could get promoted from beta to a release?
We are currently working on fixing a
Is there a way to share an encoder instance, or inject shared state, across
producers/consumers? Guice?
From: Jun Rao [jun...@gmail.com]
Sent: Wednesday, May 15, 2013 11:02 PM
To: users@kafka.apache.org
Subject: Re: could an Encoder/Decoder be stateful?
E
Any possibility for a queuedMessageCount(topic, partitionNumber) protocol?
thanks,
rob
From: Jun Rao [jun...@gmail.com]
Sent: Wednesday, May 15, 2013 10:59 PM
To: users@kafka.apache.org
Subject: Re: API to to query messages amount under one topic
In 0.8, y
This LeaderNotAvailableException only occurs once now, running kafka 8 from a
dos shell. We are following KAFKA-876 closely which we also experience.
thanks!
rob
From: Withers, Robert
Sent: Friday, April 19, 2013 6:38 AM
To: users@kafka.apache.org
ot; on the checkout that builds your binary. If you can't
figure this out, could you try the latest code in the 0.8 branch and see if
the problem still exists?
Thanks,
Jun
On Sat, Apr 6, 2013 at 10:36 AM, Withers, Robert wrote:
> I am not entirely sure as it has been about a year since I used
I am having difficulty specifying a subset of stats to watch with JmxTool, in
kafka 0.7.2. Here is the command I am attempting to use, but I do not know how
to specify the specific stat within SocketServerStats to watch. I keep getting
an error when trying any of the following:
--object-name
ly(MapLike.scala:134)
at scala.collection.immutable.Map$Map1.apply(Map.scala:93)
at kafka.cluster.Partition.getOrCreateReplica(Partition.scala:81)
at kafka.cluster.Partition$$anonfun$2.apply(Partition.scala:145)
at kafka.cluster.Partition$$anonfun$2.apply(Partitio
Hi,
I have an Eclipse java project, with the 2.9.1 scala jar
(scala-library-2.9.1.RC4.jar) in the build path, as well as the
kafka_2.8.0-0.8-SNAPSHOT.jar. I wrote the simple Producer example and try to
run it and it fails with a NoClassDefFoundError runtime exception. Here is my
code and the
13 10:02 AM
To: users@kafka.apache.org
Subject: Re: LeaderNotAvailableException: reventing the running of
kafka-console-producer.bat
If this happens on every message, it indicates a problem. Could you wipe out
all ZK and Kafka data and try it again? If it still doesn't work, could you
, Apr 4, 2013 at 1:37 PM, Withers, Robert wrote:
> I am brand new to exploration into kafka, for work. Any help is much
> appreciated.
>
> A co-worker built and updated a package for us to use on windows. I have
> no idea what version it is. We cannot update through our firewal
I am brand new to exploration into kafka, for work. Any help is much
appreciated.
A co-worker built and updated a package for us to use on windows. I have no
idea what version it is. We cannot update through our firewall. :( I am
able to start the zookeeper and a kafka server, but when I
I am struggling to load kafka into eclipse to get started. I have tried to
follow the instructions here:
https://cwiki.apache.org/KAFKA/developer-setup.html, but I cannot connect to
the SVN repo to check-out. A co-worked pulled from github, but I seem to be
missing a lot of jars. This post m
36 matches
Mail list logo
