Re: kafka consumer not consuming messages

Hi Jun, No its not that problem. I am not getting what the problem is can you please help. thanks Arjun Narasimha Kota On Monday 10 February 2014 09:10 PM, Jun Rao wrote: Does https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whydoesmyconsumernevergetanydata? apply? Thanks, Jun O

RE: Dropping messages ?

Hi Jun/Guozhang We might have to retry our QA tests in its entirety. We simply cannot reset consumer offset as there is a lot of processing involved after consuming those messages. This might take almost a week. The Kafka message also contains `volatile` data which is fetched from a database an

Re: 0.72 Consumer: message is invalid, compression codec: NoCompressionCodec

Yes, there might be - we experience link resets every so often, and definitely did today. Assume it is this, are you surprised the thread went down? Perhaps we need to catch this? Philip > On Feb 10, 2014, at 8:38 PM, Jun Rao wrote: > > This indicates that message checksum validation failed

Re: code + sbt tips

In trunk, we have the gradle support now. It seems to build the intellij project cleaner. Thanks, Jun On Mon, Feb 10, 2014 at 7:14 PM, S Ahmed wrote: > Few quick questions that I hope people can help me with: > > > 1. most of you guys use intellij, do you always build using sbt? i.e. you > l

Re: 0.72 Consumer: message is invalid, compression codec: NoCompressionCodec

This indicates that message checksum validation failed. Is there any issue with the network? Thanks, Jun On Mon, Feb 10, 2014 at 5:00 PM, Philip O'Toole wrote: > Saw this thrown today, which brought down a Consumer thread -- we're using > Consumers built on the High-level consumer framework.

Re: Mirrormaker clients not balanced

This is probably your issue: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-IamusingtheZK-basedproducerin0.7andIseedataonlyproducedonsomeofthebrokers,butnotall,why ? Thanks, Jun On Mon, Feb 10, 2014 at 4:04 PM, Tomas Nunez wrote: > I don't see anything in kafabe03. In fact, I just

Re: How to compile with a newer version of zookeeper

0.8-beta already depends on zookeeper 3.3.4. Also, Kafka 0.8 final is better and more stable compared to 0.8-beta Thanks, Neha On Mon, Feb 10, 2014 at 6:19 PM, Libo Yu wrote: > Hi team, > > We are using Kafka 0.8-beta1. The zookeeper in it is 3.3.3 (although the > version in the license file i

How to compile with a newer version of zookeeper

Hi team, We are using Kafka 0.8-beta1. The zookeeper in it is 3.3.3 (although the version in the license file is 3.3.4). I want to upgrade to a newer version. Any idea what I need to do in order to compile broker with a newer version of zookeeper? Thanks. Libo

code + sbt tips

Few quick questions that I hope people can help me with: 1. most of you guys use intellij, do you always build using sbt? i.e. you lose out on the bulid with IDE features like clicking on an error that jumps to that part of the code etc. 2. do you just build using the default scala version 2.8

Re: 0.72 Consumer: message is invalid, compression codec: NoCompressionCodec

I should we *think* this exception brought down the Consumer thread. The problematic partition on our system was 2-29, so this is definitely the related thread. Philip On Mon, Feb 10, 2014 at 5:00 PM, Philip O'Toole wrote: > Saw this thrown today, which brought down a Consumer thread -- we're

0.72 Consumer: message is invalid, compression codec: NoCompressionCodec

Saw this thrown today, which brought down a Consumer thread -- we're using Consumers built on the High-level consumer framework. What may have happened here? We are using a custom C++ Producer which does not do compression, and which hasn't changed in months, but this error is relatively new to us,

Re: Config for new clients (and server)

Yeah I am aware of how zookeeper behaves, I think it is kind of gross. I think logging it at DEBUG gets you what you want--by default we don't pollute logs, but anyone who wants to log this can enable DEBUG logging on org.apache.kafka.clients.producer.ProducerConfig. If we want this on by default

Re: Mirrormaker clients not balanced

I don't see anything in kafabe03. In fact, I just restarted it, and I saw in the logs a lot of other topics registering. First, lines like this: (...) [2014-02-10 20:30:07,195] INFO Loading log 'topic2-0' (kafka.log.LogManager) [2014-02-10 20:30:07,201] INFO Loading the last segment /var/kafka/topi

Re: New Consumer API discussion

WRT to hierarchical topics, I'm referring to KAFKA-1175. I would just like to think through the implications for the Consumer API if and when we do implement hierarchical topics. For example, in the proposal

Re: Config for new clients (and server)

+1 Jun. On Mon, Feb 10, 2014 at 2:17 PM, Sriram Subramanian < srsubraman...@linkedin.com> wrote: > +1 on Jun's suggestion. > > On 2/10/14 2:01 PM, "Jun Rao" wrote: > > >I actually prefer to see those at INFO level. The reason is that the > >config > >system in an application can be complex. Som

Re: New Consumer API discussion

Thanks for the feedback. Mattijs - - Constructors link to http://kafka.apache.org/documentation.html#consumerconfigs for valid configurations, which lists zookeeper.connect rather than metadata.broker.list, the value for BROKER_LIST_CONFIG in ConsumerConfig. Fixed it to just point to ConsumerConf

Re: New Consumer API discussion

Hi Mattijs: 2. As Neha said, one design of the new consumer is to have non-blocking consuming API instead of blocking API. Do you have a strong reason in mind to still keep the blocking API instead of just using "while(no-data) poll(timeout)"? 3. No we have not thought about hierarchical topics.

Re: New Consumer API discussion

Hello Jay, Thanks for the detailed comments. 1. Yeah we could discuss a bit more on that. 2. Since subscribe() is incremental, adding one topic-partition is OK, and personally I think it is cleaner than subscribe(String topic, int...partition)? 3. Originally I was thinking about two interfaces:

Re: Config for new clients (and server)

+1 on Jun's suggestion. On 2/10/14 2:01 PM, "Jun Rao" wrote: >I actually prefer to see those at INFO level. The reason is that the >config >system in an application can be complex. Some configs can be overridden in >different layers and it may not be easy to determine what the final >binding >va

Re: New Consumer API discussion

Hi Mattijs: We have not updated the wiki pages for config yet, and it will not be updated until we release 0.9 with these changes. Currently consumers do have a commitOffsets function that can be called by the users, but for most use cases auto.commit is turned on and this function gets called by

Re: Config for new clients (and server)

I actually prefer to see those at INFO level. The reason is that the config system in an application can be complex. Some configs can be overridden in different layers and it may not be easy to determine what the final binding value is. The logging in Kafka will serve as the source of truth. For r

RE: Building a producer/consumer supporting exactly-once messaging

Thanks Jay for the info, and Neha for adding it to the FAQ! On the producer side I've been going down Jay's second route, i.e. adding metadata to the messages as they are published. Though in my case I don't just want to avoid duplicates on a per-message basis but be able to quickly identify a

Re: New Consumer API discussion

Couple of very quick thoughts. 1. +1 about renaming commit(...) and commitAsync(...) 2. I'd also like to extend the above for the poll() method as well. poll() and pollWithTimeout(long, TimeUnit)? 3. Have you guys given any thought around how this API would be used with hierarchical topics? 4. Wo

Re: New Consumer API discussion

A few items: 1. ConsumerRebalanceCallback a. onPartitionsRevoked would be a better name. b. We should discuss the possibility of splitting this into two interfaces. The motivation would be that in Java 8 single method interfaces can directly take methods which might be more intuitive. c. I

Re: New Consumer API discussion

Hey Neha, This looks really promising, I particularly like the ability to commit offsets for topic/partition tuples over just commit(). Some remarks: - Constructors link to http://kafka.apache.org/documentation.html#consumerconfigs for valid configurations, which lists zookeeper.connect rath

New Consumer API discussion

As mentioned in previous emails, we are also working on a re-implementation of the consumer. I would like to use this email thread to discuss the details of the public API. I would also like us to be picky about this public api now so it is as good as possible and we don't need to break it in the f

Re: Building a producer/consumer supporting exactly-once messaging

Ack, nice, should have thought of doing that... -Jay On Mon, Feb 10, 2014 at 10:12 AM, Neha Narkhede wrote: > Added this to our FAQ - > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIgetexactlyonemessagingfromKafka > ? > > > > On Mon, Feb 10, 2014 at 9:46 AM, Jay Kreps wrote

Re: Config for new clients (and server)

Hey Jun, I think that is reasonable but would object to having it be debug logging? I think logging out a bunch of noise during normal operation in a client library is pretty ugly. Also, is there value in exposing the final configs programmatically? -Jay On Sun, Feb 9, 2014 at 9:23 PM, Jun Rao

Re: Building a producer/consumer supporting exactly-once messaging

Added this to our FAQ - https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIgetexactlyonemessagingfromKafka ? On Mon, Feb 10, 2014 at 9:46 AM, Jay Kreps wrote: > The out-of-the-box support for this in Kafka isn't great right now. > > Exactly once semantics has two parts: avoiding d

Re: Building a producer/consumer supporting exactly-once messaging

The out-of-the-box support for this in Kafka isn't great right now. Exactly once semantics has two parts: avoiding duplication during data production and avoiding duplicates during data consumption. There are two approaches to getting exactly once semantics during data production. 1. Use a singl

Re: Building a producer/consumer supporting exactly-once messaging

Have you read this part of the documentation? http://kafka.apache.org/documentation.html#semantics Just wondering if that solves your use case. On Mon, Feb 10, 2014 at 9:11 AM, Garry Turkington < g.turking...@improvedigital.com> wrote: > Hi, > > I've been doing some prototyping on Kafka for a f

Re: Dropping messages ?

Are these messages sent to a newly created topic? Guozhang On Mon, Feb 10, 2014 at 7:38 AM, Jun Rao wrote: > If you reset the consumer offset and try to consume those messages again, > do you see the same drop? > > Thanks, > > Jun > > > On Mon, Feb 10, 2014 at 1:21 AM, A A wrote: > > > Hi > >

Building a producer/consumer supporting exactly-once messaging

Hi, I've been doing some prototyping on Kafka for a few months now and like what I see. It's a good fit for some of my use cases in the areas of data distribution but also for processing - liking a lot of what I see in Samza. I'm now working through some of the operational issues and have a que

Re: Mirrormaker clients not balanced

One of the be brokers (3) is not registered in ZK. Do you see ZK session expiration (potentially due to GC) in that broker? Thanks, Jun On Mon, Feb 10, 2014 at 7:28 AM, Tomas Nunez wrote: > Hi > > I'm new around here and I'm dealing with a problem, and reading the > documentation I don't know

Mirrormaker clients not balanced

Hi I'm new around here and I'm dealing with a problem, and reading the documentation I don't know where else to look. I have a cross-dc mirrormaker setup: Mirrormaker is consuming from 5 frontend servers in each DC (10 in total) and 5 backend servers are consuming from mirrormaker. That's working

Re: Querry regarding setting up Kafka server on offline linux machine.

This seems like a maven issue. Not sure the reason though. You could also just try the binary release for 0.8.0. Thanks, Jun On Mon, Feb 10, 2014 at 1:55 AM, Saurabh Gupta A < saurabh.a.gu...@ericsson.com> wrote: > Hello, > I am new to Kafka, facing below problem while setting up Kafka. > I do

Re: kafka consumer not consuming messages

Does https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whydoesmyconsumernevergetanydata? apply? Thanks, Jun On Sun, Feb 9, 2014 at 10:27 PM, Arjun wrote: > Hi, > > I started using kafka some time back. I was experimenting with 0.8. My > problem is the kafka is unable to consume the me

Re: Dropping messages ?

If you reset the consumer offset and try to consume those messages again, do you see the same drop? Thanks, Jun On Mon, Feb 10, 2014 at 1:21 AM, A A wrote: > Hi > > We have been using Kafka(0.8) for the past few months with the following > setup > Kafka Broker - 1Zookeepers Ensemble - 3Partit

Re: Pattern for using kafka producer API

If you are only worried about throughput, you can use one producer in async mode. You can tune the batch size and time for better performance. Thanks, Jun On Sun, Feb 9, 2014 at 11:42 PM, pushkar priyadarshi < priyadarshi.push...@gmail.com> wrote: > What is the most appropriate design for usin

Querry regarding setting up Kafka server on offline linux machine.

Hello, I am new to Kafka, facing below problem while setting up Kafka. I downloaded "kafka-0.8.0-src.gz". Then I followed below steps: > tar xzf kafka-.tgz > cd kafka- > ./sbt update Below is the error I am facing: Please note that I am using linux machine which is offline and I have to install

Re: kafka consumer not consuming messages

On extension to the same problem i am seeing this "INFO Closing socket connection to /127.0.0.1. (kafka.network.Processor)" in my log continuously. I searched the web and found this code in an exception block " https://apache.googlesource.com/kafka/+/40a80fa7b7ae3d49e32c40fbaad1a4e402b2ac71/cor

Dropping messages ?

Hi We have been using Kafka(0.8) for the past few months with the following setup Kafka Broker - 1Zookeepers Ensemble - 3Partitions per topic - 3 Yesterday, while running Stress tests in one of the QA machines , we observed that a few messages which were produced within a couple of milliseconds