Re: Kafka server - conflicted ephemeral node

Thanks for your response gharatmayuresh1, but I don't know what you mean exactly. I have restart my server and I want to find out the cause in case it happen again. 2015-04-21 11:36 GMT+08:00 : > Try bouncing > 10.144.38.185 > > This should resolve the issue. > > Thanks, > > Mayuresh > Sent from

Re: Kafka server - conflicted ephemeral node

Try bouncing 10.144.38.185 This should resolve the issue. Thanks, Mayuresh Sent from my iPhone > On Apr 20, 2015, at 8:22 PM, 小宇 wrote: > > 10.144.38.185

Kafka server - conflicted ephemeral node

Hi, all Our kafka server encounter an unexpected shut down last night, I checked the logs, it says there is a conflicted ephemeral node, but can't find what issue it is, can anyone help check this, thanks! [2015-04-21 03:00:00,562] INFO conflict in /brokers/ids/0 data: {"jmx_port":-1,"timestamp

Re: Kafka Monitoring using JMX

You can use this https://github.com/Stackdriver/jmxtrans-config-stackdriver/blob/master/jmxtrans/stackdriver/json-specify-instance/kafka.json as an example of how Mbean are named and how " is being escaped with \", and just use different output writer for any thing prior to 0.8.2.1 version. After

Re: SimpleConsumer.getOffsetsBefore

1. getOffsetsBefore is per partition, not per consumer group. The SimpleConsumer is completely unaware of consumer groups. 2. "offsets closest to specified timestamp per topic-consumerGroup" <- I'm not sure I understand what you mean. Each consumer group persists one offset per partition, thats the

Re: what response does broker send when there is not message on the partition

I believe it doesn't take consumers into account at all. Just the offset available on the partition. Why would you need it to? On Mon, Apr 20, 2015 at 3:46 AM, Alexey Borschenko wrote: >> You can also send a FetchOffsetRequest and check for the last >> available offset (log end offset) - this way

offset tracking problems -?

I have to perform frequent re- deployments, and i run into the offset problem: Unable to Receive Message:kafka server: The requested offset is outside the range of offsets maintained by the server for the given topic/partition I tried to re-create topics and reset topic metadata in zookeeper from s

Re: ReplicaFetcherThread Error, Massive Logging, and Leader Flapping

Hi Jiangjie, There's is nothing of note in the controller log. I've attached that log along with the state change log in the following gist: https://gist.github.com/banker/78b56a3a5246b25ace4c This represents a 2-hour period on April 15th. Since I've disabled the broker on question (on April 15t

Re: Topic existance Java API

Mingtao,           I think you are looking at scala version 2.8.2 . Check your kafka version by kafka_2.8.2-${version} . In 0.8.2  we’ve AdminUtils.topicExists(zkClient, topic) . Thanks, Harsha On April 20, 2015 at 2:20:27 PM, Mingtao Zhang (mail2ming...@gmail.com) wrote: Hi, I am on Kafka

Topic existance Java API

Hi, I am on Kafka 2.8.2. Looking for a way to check whether topic existed or not in Java. I saw there is a method called listTopics - TopicCommand.listTopics, but the return type is void. Is there an API to check topic existance? (preferable argument: (zkconnect, topic)) Thanks in advance! Bes

Re: Kafka Monitoring using JMX

Hi Naidu You'll need to escape the " with a \ in the Mbean names. I've run across this too and it was al pain. It can get a bit tricky if you're doing it in code because you need to account for double escapes and so forth. This is a bug in the version of Metrics that Kafka is using. There is a JIR

Re: [DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

Hi Jun,            I am using the underlying protocol GSS-API that sasl also uses. I can add details about LDAP/AD . For AD , this is in general the integration of AD to KERBEROS. I.e   kerberos can talk to AD to get the kinit login credentials ( more of a setup details between kerberos and AD)

Re: [DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

Hi, Harsha, For SASL, a common use case is the integration with LDAP/AD. For completeness, could you describe (or provide a link) how such integration can be done? Also, what about the SSL support, do you plan to describe it in same same KIP or a separate one? Thanks, Jun On Mon, Apr 20, 2015

Kafka Monitoring using JMX

We are planning to monitor health of our Kafka environment using JMX. I have looked at below links to find what is available via https://cwiki.apache.org/confluence/display/KAFKA/Available+Metrics  https://cwiki.apache.org/confluence/display/KAFKA/JMX+Reporters why some of the kafka objects ("ka

Re: Task not serializable

I meant to reach spark users... sorry about the noise. On Mon, Apr 20, 2015 at 8:44 AM, Jean-Pascal Billaud wrote: > Hi, > > I am getting this serialization exception and I am not too sure what > "Graph is unexpectedly null when DStream is being serialized" means? > > 15/04/20 06:12:38 INFO yarn

Re: [DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

Hi,      I updated the KIP-12 with more details. Please take a look   https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=51809888 Thanks, Harsha On February 11, 2015 at 10:02:43 AM, Harsha (ka...@harsha.io) wrote: Thanks Joe. It will be part of KafkaServer and will run on its own

Re: Kafka Zookeeper queries

Producers usually do not query zookeeper at all. Consumers usually query zookeeper at beginning or rebalance. It is supposed to be in frequent if you don¹t have consumers come and go all the time. One exception is that if you are using zookeeper based consumer offset commit, it will commit offset t

offset issues fix

I am using kafka 0.8.1 /zookeeper combination...Can someone advise what the proper procedure would be to fix the issue with producer / consumers offsets going out of range when consumer has to be re- started? Do I need to delete meta- data pertaining to topics? What would be the proper way of doing

Task not serializable

Hi, I am getting this serialization exception and I am not too sure what "Graph is unexpectedly null when DStream is being serialized" means? 15/04/20 06:12:38 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: Task not serializable) Exceptio

Re: Kafka Zookeeper queries

Hi, Could anyone help with this? Thanks. On Sun, Apr 19, 2015 at 12:58 AM, Achanta Vamsi Subhash < achanta.va...@flipkart.com> wrote: > Hi, > > How often does Kafka query zookeeper while producing and consuming? > > Ex: > If there is a single partition to which we produce and a HighLevel > cons

SimpleConsumer.getOffsetsBefore

Does *SimpleConsumer.getOffsetsBefore* takes in account consumer group specific offsets? I need to be able to get offsets closest to specified timestamp per topic-consumerGroup. Is there any way to achieve this using Kafka API ? Thanx!

Re: what response does broker send when there is not message on the partition

> You can also send a FetchOffsetRequest and check for the last > available offset (log end offset) - this way you won't have to send a > fetch request that is likely to fail. Does this takes in account specific consumer offsets stored in Zookeeper? On Fri, Apr 17, 2015 at 5:57 PM, Gwen Shapira

Re: Rebalance not happening even after increasing max retries causing conflict in ZK

Mike, The endless rebalance errors occur due the error that Mayuresh just pasted. The rebalance attempts fail because of the conflict in the zkNode. Below is the exact trace. *2014-12-09 13:22:11 k.u.ZkUtils$ [INFO] I wrote this conflicted ephemeral node [{"version":1,"subscription":{"__":5}

Re: running kafka/zookeeper running locally

Warren I am looking into "https://gist.github.com/asmaier/6465468";, its not compiling in my box, i am using kafak-0.8.2.1, and i am using testng java code for writing my test cases. On Mon, Apr 20, 2015 at 11:33 AM, Warren Henning wrote: > Also consider with whether you could get away with moc