Re: lost messages -?

2015-03-26 Thread Victor L
Where's this tool (DumpLogSegments) in Kafka distro? Is it Java class in kafka jar, or is it third party binary? Thank you, On Wed, Mar 25, 2015 at 1:11 PM, Mayuresh Gharat wrote: > DumpLogSegments will give you output something like this : > > offset: 780613873770 isvalid: true payloadsize: 805

Re: Kafka server relocation

2015-03-26 Thread nitin sharma
thanks Mayuresh and Jiangjie for your response. I have actually not understood Mirror maker clearly and hence bit skeptical if i will be able to execute it effectively. Online i hv seen the following command to be execute, but not understood what is consumer-1 & -2.properties here? do i need to

Re: lost messages -?

2015-03-26 Thread Harsha
Victor,         Its under kaka.tools.DumpLogSegments you can use kafka-run-class to execute it. --  Harsha On March 26, 2015 at 5:29:32 AM, Victor L (vlyamt...@gmail.com) wrote: Where's this tool (DumpLogSegments) in Kafka distro? Is it Java class in kafka jar, or is it third party binary?

Re: Kafka server relocation

2015-03-26 Thread tao xiao
Both consumer-1 and consumer-2 are properties of source clusters mirror maker transfers data from. Mirror maker is designed to be able to consume data from N sources (N >= 1) and transfer data to one destination cluster. You are free to supply as many consumer properties as you want to instruct mir

GC problem on broker startup under minimal load

2015-03-26 Thread Timo Ahokas
Hi, We are seeing really weird GC behaviour on our Kafka cluster. After a clean stop/start of the whole cluster, the servers go into a loop of increasingly slow GC cycles for an hour and then seem to recover to a normal state without any external changes to the servers or traffic. The load to the

Re: Rebalance not happening even after increasing max retries causing conflict in ZK

2015-03-26 Thread Mike Axiak
Hi guys, At HubSpot we think the issue is related to slow consumers. During a rebalance, one of the first things the consumer does is signal a shutdown to the fetcher [1] [2], in order to relinquish ownership of the partitions. This waits for the shutdown of all shutdown fetcher threads, which ca

Re: Rebalance not happening even after increasing max retries causing conflict in ZK

2015-03-26 Thread Mayuresh Gharat
Did you see something like this in any of the consumer logs : "Conflict in ….. data : ……. stored data :……” ? Thanks, Mayuresh On Thu, Mar 26, 2015 at 1:50 PM, Mike Axiak wrote: > Hi guys, > > At HubSpot we think the issue is related to slow consumers. During a > rebalance, one of the first t

Re: Rebalance not happening even after increasing max retries causing conflict in ZK

2015-03-26 Thread Mike Axiak
No, we don't normally see conflicts. We'll see endless attempts to rebalance. -Mike On Thu, Mar 26, 2015 at 5:15 PM, Mayuresh Gharat wrote: > Did you see something like this in any of the consumer logs : > > "Conflict in ….. data : ……. stored data :……” ? > > Thanks, > > Mayuresh > > On Thu, Ma

Re: Database Replication Question

2015-03-26 Thread Guozhang Wang
Hello Josh, We do not have a ticket open for idempotent producer as it is still in the discussion process, but here is the wiki: https://cwiki.apache.org/confluence/display/KAFKA/Idempotent+Producer As for transactional messaging, we have a prototype implementation at LinkedIn which is not yet i

Re: Kafka 0.9 consumer API

2015-03-26 Thread Guozhang Wang
Rajiv, Those are good points. As for implementation we have developed a class in producer that can be probably re-used for the consumer as well. org.apache.kafka.clients.producer.internals.BufferPool Please feel free to add more comments on KAFKA-2045. Guozhang On Tue, Mar 24, 2015 at 12:21 P

Re: Producer Behavior When one or more Brokers' Disk is Full.

2015-03-26 Thread Guozhang Wang
Hi Bhavesh, 1. Server disk-full is treated the same as other error, that an error code will be returned (in this case I think it is "Unknown" error though, as disk IO exception is not specifically captured). 2. Upon receiving the error from the brokers, producer will retry based on its configs. H

Re: Producer Behavior When one or more Brokers' Disk is Full.

2015-03-26 Thread svante karlsson
>4. As for recovering broker from disk full, if replication is enabled one >can just bring it down (the leader of the partition will then migrate to >other brokers), clear the disk space, and bring it up again; if replication >is not enabled then you can first move the partitions away from this bro

Re: New Offset Management API Question

2015-03-26 Thread Joel Koshy
> 1) Does Offset Commit/Fetch API works with Simple Consumer ? Yes - in 0.8.2.1. There is an example given at https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka > 2) With MM, can you selectively MM offset topic. > > Use Case: > Let's suppose you

Re: Kafka 0.9 consumer API

2015-03-26 Thread Rajiv Kurian
Thanks Guozhang, I am currently working on a project at my current company where I process data from Kafka. The data is all tiny Kafka messages (25 -35 bytes) and so far we were bottle necked on our processing speed. Recently we have made significant improvements and our processing speed has gone

Re: Producer Behavior When one or more Brokers' Disk is Full.

2015-03-26 Thread Guozhang Wang
Hmm, I think Svante is correct, writes on disk-full would probably cause the underlying file system to get in a bad state, and in that sense the broker needs to be brought down for maintenance. Guozhang On Thu, Mar 26, 2015 at 3:49 PM, svante karlsson wrote: > >4. As for recovering broker from