Re: Dropping messages ?

Can you verify if the missing messages exist in the broker or not by ways Jun have said, you can just create another consumer group which does no processing but just count messages. Guozhang On Mon, Feb 10, 2014 at 10:23 PM, Kat Walker wrote: > Hi Jun/Guozhang > > We might have to retry our Q

Re: kafka consumer not consuming messages

Arjun, Are you using the same group name for the console consumer and the java consumer? Guozhang On Mon, Feb 10, 2014 at 11:38 PM, Arjun wrote: > Hi Jun, > > No its not that problem. I am not getting what the problem is can you > please help. > > thanks > Arjun Narasimha Kota > > > On Monday

Re: kafka consumer not consuming messages

nope i will try that. thanks for suggesting On Tuesday 11 February 2014 01:59 PM, Guozhang Wang wrote: Arjun, Are you using the same group name for the console consumer and the java consumer? Guozhang On Mon, Feb 10, 2014 at 11:38 PM, Arjun wrote: Hi Jun, No its not that problem. I am no

Re: kafka consumer not consuming messages

With the same group id from the console consumer its working fine. On Tuesday 11 February 2014 01:59 PM, Guozhang Wang wrote: Arjun, Are you using the same group name for the console consumer and the java consumer? Guozhang On Mon, Feb 10, 2014 at 11:38 PM, Arjun wrote: Hi Jun, No its no

Re: Mirrormaker clients not balanced

You are totally right. I just tried, and it works. I thought about creating the dirs and "see what happens" but I was afraid I could break anything :D Thank you very, Jun! On Tue, Feb 11, 2014 at 5:31 AM, Jun Rao wrote: > This is probably your issue: > > https://cwiki.apache.org/confluence/di

Producer connection unsuccessful

I am running zookeeper and 2 brokers(port 9093 and 9094) on remote machine. I am able create topic using 'kafka-create-topic.sh' I am also to get topic list using 'kafka-list-topic.sh' But whenever I try to produce msg it gives me following error ERROR Producer connection to ipaddres:9093 unsucces

RE: Dropping messages ?

Hi Guozhang The dropped messages were found in one of the topic partitions as indicated in my first email. It is clear that the messages were persisted in Kafka logs. Can you answer my other questions regarding Kafka's consistency and durability. Thanks Kat > Date: Tue, 11 Feb 2014 00:24:27 -0

Monitoring producer failures on kafka 0.7 (using zookeeper)

Hi, I'm working with a kafka producer in async mode using zookeeper. >From time to time, i observe network problems and i see the following error in >my logs: [2014-02-11 11:21:57,848][INFO ][org.apache.zookeeper.ClientCnxn] Unable to read additional data from server sessionid 0x1441cc4a9c5

Description of jmx exposed metrics?

Hi! Sorry if this question has already been answered, but I've search the archives, the project page and the wiki unsuccessfully. I'd like to know the meaning of the jmx exposed metrics. I can guess based on the name but I'd like to be sure. Plus, some of them seem to be reset when I restart the

Re: Description of jmx exposed metrics?

Here tis! https://kafka.apache.org/documentation.html#monitoring On Feb 11, 2014, at 6:50 AM, Tomas Nunez wrote: > Hi! > > Sorry if this question has already been answered, but I've search the > archives, the project page and the wiki unsuccessfully. > > I'd like to know the meaning of the j

Re: Description of jmx exposed metrics?

Although that is not all of them, just recommended ones to pay attention to :/ On Feb 11, 2014, at 6:50 AM, Tomas Nunez wrote: > Hi! > > Sorry if this question has already been answered, but I've search the > archives, the project page and the wiki unsuccessfully. > > I'd like to know the mea

RE: How to compile with a newer version of zookeeper

When I telnet to the zookeeper and type "status", this is what I got: Zookeeper version: 3.3.3-1203054, built on 11/17/2011 05:47 GMT Is that 3.3.4? So 0.8 final also uses 3.3.4, is that right? Thanks. Regards, Libo -Original Message- From: Neha Narkhede [mailto:neha.narkh...@gmail.com

RE: How to compile with a newer version of zookeeper

Yes. That is 3.3.4. Thanks, Neha On Feb 11, 2014 6:59 AM, "Yu, Libo" wrote: > When I telnet to the zookeeper and type "status", this is what I got: > > Zookeeper version: 3.3.3-1203054, built on 11/17/2011 05:47 GMT > > Is that 3.3.4? So 0.8 final also uses 3.3.4, is that right? Thanks. > > Rega

Re: 0.72 Consumer: message is invalid, compression codec: NoCompressionCodec

We do catch the exception. However, we don't know what to do with it. Retrying may not fix the problem. So, we just log it and let the thread die. Thanks, Jun On Mon, Feb 10, 2014 at 8:42 PM, Philip O'Toole wrote: > Yes, there might be - we experience link resets every so often, and > definit

Re: kafka consumer not consuming messages

What's the fetch.wait.max.ms and fetch.min.bytes you used? Thanks, Jun On Tue, Feb 11, 2014 at 12:54 AM, Arjun wrote: > With the same group id from the console consumer its working fine. > > > On Tuesday 11 February 2014 01:59 PM, Guozhang Wang wrote: > >> Arjun, >> >> Are you using the same

Re: Producer connection unsuccessful

Can you telnet to that broker ipaddress and port from the producer host? Seems like a network connection issue. Thanks, Jun On Tue, Feb 11, 2014 at 1:56 AM, Harshal Bhutkar < harshal_bhut...@persistent.co.in> wrote: > I am running zookeeper and 2 brokers(port 9093 and 9094) on remote > machine

Re: Dropping messages ?

Are you committing offsets manually? How do you realize that some messages are lost? Do you log every message returned to Kafka consumer client? Is it possible that a message is returned to the consumer, but is lost in the application logic? Thanks, Jun On Mon, Feb 10, 2014 at 10:23 PM, Kat Wal

Re: Monitoring producer failures on kafka 0.7 (using zookeeper)

These are ZK warnings. It seems to be related to the SASL setting. Thanks, Jun On Tue, Feb 11, 2014 at 2:46 AM, Laurent Thoulon < laurent.thou...@ldmobile.net> wrote: > > Hi, > > I'm working with a kafka producer in async mode using zookeeper. > From time to time, i observe network problems an

Re: New Consumer API discussion

Hey Pradeep, That wiki is fairly old and it predated more flexible subscription mechanisms. In the high-level consumer you currently have wildcard subscription and in the new proposed interface you can actually subscribe based on any logic you want to create a "union" of streams. Personally I thin

Re: Description of jmx exposed metrics?

These are the 0.8 jmx. The 0.7 one can be found in https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring Thanks, Jun On Tue, Feb 11, 2014 at 6:51 AM, Andrew Otto wrote: > Here tis! > > https://kafka.apache.org/documentation.html#monitoring > > > On Feb 11, 2014, a

Re: Description of jmx exposed metrics?

Yup... I read both of them, but I saw much more data in the jmx and I was trying to understant it to make the most of them :) But I'll settle knowing that is the useful data. Thank you both! On Tue, Feb 11, 2014 at 5:05 PM, Jun Rao wrote: > These are the 0.8 jmx. The 0.7 one can be found in >

Re: kafka consumer not consuming messages

fetch.wait.max.ms=1 fetch.min.bytes=128 My message size is much more than that. On Feb 11, 2014 9:21 PM, "Jun Rao" wrote: > What's the fetch.wait.max.ms and fetch.min.bytes you used? > > Thanks, > > Jun > > > On Tue, Feb 11, 2014 at 12:54 AM, Arjun wrote: > > > With the same group id from t

Re: Description of jmx exposed metrics?

Then I guess mi next question is: The jmx metric "Kafka -> kafka.BrokerTopicStat. -> MessagesIn" counts how many messages went in that topic since kafka was started, and it will reset to 0 when I restart the service, right? Same thing for Kafka -> kafka.BrokerAllTopicStat -> MessagesIn, right? Th

Re: Description of jmx exposed metrics?

Yes. All jmx beans start from 0 on broker startup. Thanks, Neha On Tue, Feb 11, 2014 at 9:42 AM, Tomas Nunez wrote: > Then I guess mi next question is: > The jmx metric "Kafka -> kafka.BrokerTopicStat. -> MessagesIn" > counts how many messages went in that topic since kafka was started, and it

Re: Description of jmx exposed metrics?

Yes, but this counts how many messages went in that topic, right? This should always be increasing and never decreasing, if I understand correctly. I'm asking that because in my kafka configuration I have 18 topics, and in all of them but one this metric is increasing. There is a topic where this

Re: New Consumer API discussion

Hi Jay, I apologize for derailing the conversation about the consumer API. We should start a new discussion about hierarchical topics, if we want to keep talking about it. My final thought on the matter is that, hierarchical topics is still an important feature to have in Kafka, because it gives u

Use cases for new java client

Hi - Recently Mr. Kreps was kind enough to answer a bunch of questions from some of us here at Jive software. One subject that particularly caught our interest was the new java client, which sounds like it will super useful. We have some client code which sits on top of SimpleConsumer which we u

Re: New Consumer API discussion

Pradeep, To be clear, we want to get feedback on the APIs from the javadocsince the wiki will be slightly behind on the APIs. 1. Regarding consistency, do you have specific feedback on

Re: Use cases for new java client

Thanks for sharing your use case! In the new consumer APIs, we intend to support custom offset management as well as custom partition assignment. We have a thread discussing the new consumer API javadoc to collect feedback from users. It will greatly help to have your feedback as well - http://grok

Re: 0.72 Consumer: message is invalid, compression codec: NoCompressionCodec

In that case, is there a way to detect that a consumer instance is no longer usable, so that we can recreate the instance on the fly again to have it reconnect? Without having to restart our app? Thanks, -Vinh On Feb 11, 2014, at 7:45 AM, Jun Rao wrote: > We do catch the exception. However,

Re: New Consumer API discussion

Updated thoughts. 1. subscribe(String topic, int... paritions) and unsubscribe(String topic, int... partitions) should be subscribe(TopicPartition... topicPartitions)and unsubscribe(TopicPartition... topicPartitons) 2. Does it make sense to provide a convenience method to subs

some brokers cannot register themselves with zookeeper

Hi team, This is an issue that has frustrated me for quit some time. One of our clusters has three hosts. In my startup script, three zookeeper processes are brought up first followed by three kafka processes. The problem we have is that after three kafka processes are up, only one broker has b

Re: New Consumer API discussion

Hi, thanks for sharing this and getting feedback. Sorry I am probably missing something basic, but I'm not sure how a multi-threaded consumer would work. I can imagine its either: a) I just have one thread poll kafka. If I want to process msgs in multiple threads, than I deal w/ that after pol

Re: 0.72 Consumer: message is invalid, compression codec: NoCompressionCodec

Actually, looking at the code, the consumer client code can also catch this exception while iterating for messages. The fetcher thread inserts a special message before dying, which triggers an exception while client calls next message https://github.com/apache/kafka/blob/0.7.2/core/src/main/sca

Re: New Consumer API discussion

Hi Imran, 1. I think choosing between a) and b) is really dependent on the consuming traffic. We decided to make the consumer client single-threaded and let users to decide using one or multiple clients based on traffic mainly because with a multi-thread client, the fetcher thread could die silent

Re: some brokers cannot register themselves with zookeeper

Hello Libo, Which Kafka version are you using? Pre-0.8.1 there is a bug that can cause a registration path to be deleted: https://issues.apache.org/jira/browse/KAFKA-992 And this has been fixed in 0.8.1 Guozhang On Tue, Feb 11, 2014 at 1:16 PM, Yu, Libo wrote: > Hi team, > > This is an issu

Re: New Consumer API discussion

Comments inline: On Mon, Feb 10, 2014 at 2:31 PM, Guozhang Wang wrote: > Hello Jay, > > Thanks for the detailed comments. > > 1. Yeah we could discuss a bit more on that. > > 2. Since subscribe() is incremental, adding one topic-partition is OK, and > personally I think it is cleaner than subsc

Re: 0.72 Consumer: message is invalid, compression codec: NoCompressionCodec

Works for me. It's the least disruptive option. > On Feb 11, 2014, at 3:11 PM, Suyog Rao wrote: > > Actually, looking at the code, the consumer client code can also catch this > exception while iterating for messages. The fetcher thread inserts a special > message before dying, which triggers

Re: kafka consumer not consuming messages

Could you double check that you used the correct topic name? If so, could you run ConsumerOffsetChecker as described in https://cwiki.apache.org/confluence/display/KAFKA/FAQ and see if there is any lag? Thanks, Jun On Tue, Feb 11, 2014 at 8:45 AM, Arjun Kota wrote: > fetch.wait.max.ms=1 >

RE: Dropping messages ?

> Are you committing offsets manually? We are not committing offsets manually. As indicated in my first email, we are using the high level consumer API. >How do you realize that some messages >are lost? Do you log every message returned to Kafka consumer client? Yes we log every message recei

Re: kafka consumer not consuming messages

The topic name is correct, the o/p of the ConsumerOffserChecker is arjunn@arjunn-lt:~/Downloads/Kafka0.8/new/kafka_2.8.0-0.8.0$ bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group1 --zkconnect 127.0.0.1:2181,127.0.0.1:2182,127.0.0.1:2183 --topic taf.referral.emails.service G

Re: Dropping messages ?

I haven't seen an issue like this before: a consumer doesn't see some messages existing in a broker. So, some more debugging is needed. If you start a new consumer consuming from the beginning, do you see those missing messages? Thanks, Jun On Tue, Feb 11, 2014 at 9:43 PM, Kat Walker wrote: >

Publishing/marking multiple event messages as single transaction?

Hi, And again, maybe this is more archtectural question, but I would like to hear how people tackle this problem. Anyway, as far as I can tell, a producer can publish multiple events as part of one transaction on its side. Examples are: - Multiple entities are created as part of one