Re: Issue with high level consumer in 8.1.1 after restart

you can use one more auto.offset.reset=smallest/largest. I ALSO face the same issue and it work fine for me . Might be my key name is not correct, please check the key in kafka doxumentation ?.. On 4/6/15, Madhukar Bharti wrote: > Hi Mayuresh, > > We are having only one consumer in the group and

RE: How to reduce produce-localtimems 999th percentile?

We're using requireAck=0, we have tried sync/async producer, got same issue, any hint for more investigation? Thanks, Honghai Chen -Original Message- From: Honghai Chen [mailto:honghai.c...@microsoft.com] Sent: Thursday, April 23, 2015 1:01 PM To: users@kafka.apache.org Subject: How to

How to reduce produce-localtimems 999th percentile?

We got 300ms for produce-localtimems .999th percentile for some machines of one cluster, actually the value for other machines of the cluster at most around 100ms. The QPS only 20~80 per second. The strange things is that only several machines has this kind of issue, all other machines has no

MirrorMaker producer partitioning

Hello, We're running 0.8.1.1, and having an issue with the MirrorMaker not evenly distributing production over all topic partitions ( http://i.imgur.com/03u7bA9.png). Each producer seems to be using the default behavior of sending to one partition for a period of time before switching. Over time,

Re: How to set console consumer group ID

Yes, you pass any consumer property including group.id by having them in property file and passing path to it using --consumer.config of consumer consumer. Thanks, Raja. On Wed, Apr 22, 2015 at 1:45 AM, Lukáš Havrlant wrote: > Hi, > is it possible to set group ID for console consumer on command

How to set console consumer group ID

Hi, is it possible to set group ID for console consumer on command line? Something like $ bin/kafka-console-consumer.sh --groupid myGroupId Lukáš

Re: Issue with high level consumer in 8.1.1 after restart

Can you resend the logs? Also I did not get what you meant by : "Sometimes after restart consumers picks the messages sometimes it starts but not receives any messages. " Thanks, Mayuresh On Wed, Apr 22, 2015 at 5:20 AM, Madhukar Bharti wrote: > Hi All, > > Any update on this? We are facing s

Using different IPs for clients and kafka inter-node communication

I have a 3-node setup where each kafka/zookeeper node has two network interfaces - one globally routeable (10.x.x.x) and one internal (192.168.x.x). I would like the kafka nodes to use the internal interface to talk to its neighboring brokers (faster and more efficient) and use the globally routeab

Re: [DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

1. *Support for running potentially long-running delegated tasks outside  the network thread*: It is recommended that delegated tasks indicated by  a handshake status of NEED_TASK are run on a separate thread since they may  block (  http://docs.oracle.com/javase/7/docs/api/javax/net/ssl/SSLEngine.

Re: [DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

Hi Rajini,        Thanks for the details. I did go through your code . There was a discussion before about not having selector related code into the channel or extending the selector it self.  1. *Support for running potentially long-running delegated tasks outside  the network thread*: It is re

Re: Issue with high level consumer in 8.1.1 after restart

Hi All, Any update on this? We are facing same issue all the time when ever we re-start consumers. On Mon, Apr 6, 2015 at 11:19 PM, Madhukar Bharti wrote: > Hi Mayuresh, > > We are having only one consumer in the group and only one partition for > that topic. > We have set auto.commit.enable fa

Re: [DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

When we were working on the client-side SSL implementation for Kafka, we found that returning selection interest from handshake() method wasn't sufficient to handle some of the SSL sequences. We resorted to managing the selection key and interest state within SSLChannel to avoid SSL-specific knowle

Manually controlling the start offset in the high level API

Hi, I'm a committer at the Apache Flink project. I'm working on adding support for exactly-once semantics for Flink's stream processing component. Therefore, we want to keep track of the read offset from the KafkaSource and restart the consumption from the last known offset (tracked within Flink).

Re: Fetch API Offset

I have found out what was the issue -- it was bug in the library. Thanks everyone for help. On Tue, Apr 21, 2015 at 9:02 PM, Andrei wrote: > @Piotr I can confirm that in native (Java/Scala) library returned message > offsets start with the one specified in request, not the next one. Note > sure a