I am using the kafka java api async client (the one that wraps the Scala
client). It's dropping quite a bit of data due to the queue being full, and
the process ends up taking a lot of cpu too.
I am posting to a topic with 1024 partitions (across 3 brokers) - maybe the
high number of brokers is on
The offset request does not consider HW, and hence should not be affected
if it is not advanced yet.
As Jun said, the getOffsetsBefore API just send an OffsetRequest to Kafka
and will get an OffsetResponse in return, but this may not be exposed to
the API though.
Guozhang
On Tue, Dec 9, 2014 at
Hi, Jun
The getOffsetBefore API only return the offset array, i can't find anywhere
to get the error code.
Thanks.
On Sat, Dec 13, 2014 at 1:02 AM, Jun Rao wrote:
>
> Yes, for regular consumers, we always return the high watermark as the
> latest offset. Did you check the error code in the getO
Ah, make sense. It seems that this is already fixed in 0.8.2 and trunk.
Thanks,
Jun
On Fri, Dec 12, 2014 at 5:34 PM, Helin Xiang wrote:
> Hi, Jun
>
> What you said is right, but in the code of simpleconsumer ( where the
> BlockingChannel.disconnect()
> will be called ), it firstly checked if t
Thanks, Jun.
Finally we found we just need to remove our partition key, and it won't
send data to the already dismissed broker. although it will cause the data
not so even among the partitions.
On Sat, Dec 13, 2014 at 1:50 AM, Jun Rao wrote:
>
> To get the same behavior of 0.7, you just need to
See ZookeeperConsumerConnectorTest.
Thanks,
Jun
On Fri, Dec 12, 2014 at 3:59 PM, Shannon Lloyd wrote:
> Hi Jun,
>
> Thanks for that - can you point me to one of your tests that does this?
> i.e. a test which shuts down the consumer and then re-initialises it (or
> rather creates a new consumer
Hi, Jun
What you said is right, but in the code of simpleconsumer ( where the
BlockingChannel.disconnect()
will be called ), it firstly checked if the channel is connected, that's
the real problem.
And we reproduced the problem in our testing environment.
first we use iptables to drop packet and
Hi Jun,
Thanks for that - can you point me to one of your tests that does this?
i.e. a test which shuts down the consumer and then re-initialises it (or
rather creates a new consumer with the same config) and then uses it? I'd
be keen to see what settings (properties) you're using. Maybe I've just
If I understand KAFKA-1476 it is only a command line tool that gives access
by using ZKUtils not an API to Kafka. We're looking for a Kafka API so I
guess that this functionality is missing.
thanks for the pointer
Svante Karlsson
2014-12-12 19:03 GMT+01:00 Jiangjie Qin :
>
> KAFKA-1476 is add
>
> Is there any solution to avoid this duplicated entry in target cluster? I
> am using Kafka
That is the expected behavior when the mirror maker process is killed. It
is expected that in most cases, it be shutdown cleanly. If it is killed, it
doesn't get a chance to checkpoint it's offset, whic
Basically, don't want to use ZK, for the reasons driving the new client
offset features in 0.8.2.
So we are looking for a library we could use that provides an alternative
client offset implementation (broker-side, redis, local file, anything
else). We could roll our own and put it behind the clie
KAFKA-1476 is addressing this need, but it¹s not checked in yet. Currently
maybe you can use zookeeper client to check the zookeeper path.
‹Jiangjie (Becket) Qin
On 12/12/14, 2:45 AM, "svante karlsson" wrote:
>Disregard the creation question - we must have done something wrong
>because
>now our
Currently, no.
Thanks,
Jun
On Thu, Dec 11, 2014 at 3:39 PM, hsy...@gmail.com wrote:
> Hi Guys,
>
> If I know the brokers. Is there a way to know the zookeeper host from
> broker list?
>
> Thanks!
> Siyuan
>
To get the same behavior of 0.7, you just need to create multiple
partitions for a topic in 0.8 with a replication factor of 1. When one of
the partitions is not available, the producer will route the data to other
partitions.
Thanks,
Jun
On Wed, Dec 10, 2014 at 5:58 PM, Helin Xiang wrote:
> Y
Yes, that seems to be the implementation in metrics-core. It always maps
the value of a Gauge to an Object, instead of the generic type. So, you
will have to do the casting. If you just want to get the value, you can
probably just call toString() on the object.
Thanks,
Jun
On Wed, Dec 10, 2014 a
Suren,
Like out of the box storage or roll your own?
-Steve
On Fri, Dec 12, 2014 at 6:33 AM, Surendranauth Hiraman <
suren.hira...@velos.io> wrote:
> My team is using Kafka 0.8.1 and we may not be able to upgrade to 0.8.2 to
> take advantage of the broker-side commit of client offsets.
>
> Is a
Any error in the controller and the state-change log? In particular, did
broker 5 receive a LeaderAndIsr request for those affected partitions?
Thanks,
Jun
On Tue, Dec 9, 2014 at 8:43 AM, Neil Harkins wrote:
> Hi. We've suffered a single node HW failure (broker_id 4)
> with at least 2 replicas
Hmm, but if we hit an exception in BlockingChannel.connect(), we will
call BlockingChannel.disconnect(), which will close the socket channel.
Thanks,
Jun
On Tue, Dec 9, 2014 at 7:09 PM, Helin Xiang wrote:
> Hi, Jun
>
> We experienced a network device problem. and cause all brokers crashed.
> A
Yes, for regular consumers, we always return the high watermark as the
latest offset. Did you check the error code in the getOffsetBefore request?
Thanks,
Jun
On Tue, Dec 9, 2014 at 12:31 AM, 向河林 wrote:
> Thanks Guozhang，
>
> I checked the server log, and I am 90% sure that no leader movement
Hmm, that's what we do in our unit tests and it seems that we can cleanly
shutdown the consumer.
Thanks,
Jun
On Mon, Dec 8, 2014 at 9:17 PM, Shannon Lloyd wrote:
> Not explicitly. Some additional background might help. I'm running an
> integration test using an embedded Kafka cluster and ZK qu
My team is using Kafka 0.8.1 and we may not be able to upgrade to 0.8.2 to
take advantage of the broker-side commit of client offsets.
Is anyone aware of a Java/Scala library for client offset storage outside
of ZK?
-- Suren
SUREN HIRAMAN, VP TECHNOLOGY
Velos
Accelerating Machine Learning
440
Disregard the creation question - we must have done something wrong because
now our code is working without obvious changes (on another set of
brokers).
However it turns out to be difficult to know the existing consumer group
strings. Is the message format in __consumer_offsets "public"/stable in
22 matches
Mail list logo
