Re: How To Use kafka-topics.sh To Move Replica To Different Broker

You need to use kafka-reassign-partitions.sh to do this. See http://kafka.apache.org/documentation.html#basic_ops_cluster_expansion Thanks, Jun On Wed, May 7, 2014 at 12:44 PM, Lung, Paul wrote: > Hi Guys, > > I’m using Kafka 0.8.1, and I’m having difficulty trying to move a replica > from on

Re: How To Use kafka-topics.sh To Move Replica To Different Broker

You cannot use the kafka-topics tool to move replicas to other brokers. To learn how to move replicas, please see the "Custom partition assignment and migration" section here . Hope that helps! If not, please let us know,

Re: how to know kafka producer api status

Yes, in sync mode, if send() fails, an exception will be thrown. Thanks, Jun On Fri, May 9, 2014 at 1:00 AM, Yonghui Zhao wrote: > If l use java producer api in sync mode. > > public void send(kafka.producer.KeyedMessage message) { /* compiled > code */ } > > How to know whether a send proc

Re: kafka performance question

A couple of thoughts spring to mind, are you sending the whole file as 1 message? and is your producer code using sync or async mode? Cheers Rob. On 14 May 2014 15:49, Jun Rao wrote: > How many brokers and partitions do you have? You may try increasing > batch.num.messages. > > Thanks, > >

NFS and/or local filesystem consumer?

Hi all, We are experimenting as using Kafka as a midpoint between microscopes and a Spark cluster for data analysis. Our microscopes almost universally use Windows machines for acquisition (as do most scientific instruments), and our compute cluster (which runs Spark among many other things) r

Re: how to know kafka producer api status

It typically throws a exception in the end of the sync producer cannot deliver your message. In the case where there is a IOException or similiar exceptions that the Broker cannot deal with, I believe it will try to return UnknownError response which will then throw in the producer. In cases wher

Re: New consumer APIs

Also going to add that I know a per stream commit is a strong requirement for folks I know using Kafka, and seen custom code done just to do so. Tim > On May 9, 2014, at 1:19 PM, Eric Sammer wrote: > > All: > > I've been going over the new consumer APIs and it seems like we're > squishing a l

Re: log.retention.size

Agreed…a global knob is a bit tricky for exactly the reason you've identified. Perhaps the problem could be simplified though by considering the context and purpose of Kafka. I would use a persistent message queue because I want to guarantee that data/messages don't get lost. But, since Kafka

Re: how to know kafka producer api status

Hi Yonghui, If you set producer.type = sync, then the send() call will not return until it have received the ack from the broker, and if the response contains any error code it will re-try send until all retries exhausted, and then will throw an exception. Guozhang On Fri, May 9, 2014 at 4:18 A

NFS and/or local filesystem consumer?

Hi all, Sorry for the possible repost--hadn't seen this in the list after 18 hours and figured I'd try again We are experimenting as using Kafka as a midpoint between microscopes and a Spark cluster for data analysis. Our microscopes almost universally use Windows machines for acquisition

topic.log.retention.hours doesn't work

HI All, My Kafka version is: kafka_2.9.2-0.8.1.1 I used topic.log.retention.hours to define the topic specific retention hours, but it haven't trigger the deletion of outdated logs. Below are my settings: log.retention.hours=12 topic.log.retention.hours=ssnzEvent:1 the general retention hours i

Re: Loss of Leader in Kafka

Hi Neha/Jun, A kafa topic deletion involves - 1. Zookeeper clean up (Nodes in zookeeper clusters to be removed - one under brokers/ and other where the offset info is stored) 2. kafka data logs to cleared on each server in cluster for the topic to be deleted 3. partition and replication data to be

Controller to broker connection timing out through load balancer

Greetings, I'm looking for some feedback with using advertised.host.name and advertised.port on kafka 0.8.1.1 through a load balancer. The brokers are fronted with haproxy to support our cluster mirroring configuration. The setup has been working as expected, where producers, consumers, and brok

Re: New consumer APIs

>> (p.s. I *really* want per-stream commit baked into the API.) Assuming that you mean being able to control commit() per partition, then yes. This is included. You can see some code examples

Re: New consumer APIs

Tim, I'm going to ask you the same question :-) By "per stream commit", do you mean a per partition commit like this API - public OffsetMetadata commit(Map offsets); This API allows the consumer to commit the specified offsets only for selected partitions. Thanks, Neha On Thu, May 15, 2014 a

Max Message Size issue

Hi Kafka Dev Group, We are using Kafka version 0.8 and I am getting following exception: WARN warn, Produce request with correlation id 1617 failed due to [rawlog,19]: kafka.common.MessageSizeTooLargeException WARN warn, Produce request with correlation id 1819 failed due to [rawlog,24]: kaf

Re: owner info in zk is not correct

Hi Yonghui, Could you check if consumer2's fetcher thread is still alive? Also we have an entry in FAQ wiki page about "consumer stopped consuming", current Apache has some issues with the wiki page, but you may want to check it out once Apache page resumes. Guozhang On Wed, May 14, 2014 at 8:5

Re: how to know kafka producer api status

Seems email archive doesn't work now in http://mail-archives.apache.org/mod_mbox/kafka-users/201405.mbox/browser And also I sent email to users-subscr...@kafka.apache.orgto subscribe this group, it doesn't work either. 2014-05-09 16:00 GMT+08:00 Yonghui Zhao : > > If l use java producer ap

Re: Leader for almost all topic, partitions on the same broker

Try using the kafka-topics.sh tool with the --describe option to see if the 1st replica is well distributed across all topics and partitions. The preferred replica election tool just moves the leader to the 1st replica in the replica list for a partition. Also, we fixed a few bugs related to this

Re: Leader for almost all topic, partitions on the same broker

Does that error occur continuously? Preferred leaders are the first replica in the assigned replica list. Could you list the topics and see the distribution of the first replica in all partitions? Thanks, Jun On Wed, May 14, 2014 at 11:45 AM, Jon Bender wrote: > Hello, > > I have a 3-node clus

Re: who is using kafka to stare large messages?

Hi. I´m using Kafka, Can you help me please? :) I´m have the problem to read the next message available when is larger than the maximum fetch size you have specified, i do not manage several size´s for the message becase the size is very diferente of one message to other. Do you now the more eff

Re: Leader for almost all topic, partitions on the same broker

Hello Jon, Did you see all 3 brokers registered in ZK? From the error logs it seems some of the brokers did not successfully startup and hence cannot take any partitions. Guozhang On Wed, May 14, 2014 at 11:45 AM, Jon Bender wrote: > Hello, > > I have a 3-node cluster that has had a couple iss

Re: How To Use kafka-topics.sh To Move Replica To Different Broker

Hi Paul, kafka-topics.sh is not used for moving replicas to different brokers, what you are looking for is the partition-reassignment tool in bin/kafka-reassign-partitions.sh Guozhang On Wed, May 7, 2014 at 12:44 PM, Lung, Paul wrote: > Hi Guys, > > I’m using Kafka 0.8.1, and I’m having dif

Re: Loss of Leader in Kafka

>> Am I right with the above? That is the hacky way of deleting a topic and has not been well tested and hence is not certified to work. There are also several race conditions in the steps above where the controller can keep overwriting the data deleted in zookeeper. So again, delete topic require

Re: Max Message Size issue

>From the documentation, the correct is message.max.bytes ​Regards​ -- Lucas Zago 48 9617 6763

Re: performance testing data to share

Hi Bert, Thanks for sharing the perf number. Just wondering if you can share your hardware setup and broker and producer conf? are you running producer at the kafka node? we don't have an optimal setup yet, such as multiple disk drives. We are not seeing same perf number in our setup. Would li

Re: New consumer APIs

Eric, Thanks for the feedback. It seems that due to Apache mail server issue, your email 6 days ago just got delivered. A quick answer to your question is that for simplicity, the proposed API implicitly assumes there is only 1 stream returned in each Consumer and therefore all seek/commit/poll a

Re: Controller to broker connection timing out through load balancer

The only errors logged are in the controller log: [2014-05-16 20:42:31,846] DEBUG [TopicChangeListener on Controller 1]: Topic change listener fired for path /brokers/topics with children Test2,Test1,Test3,Test4 (kafka.controller.PartitionStateMachine$TopicChangeListener) [2014-05-16 20:42:31,897]

Re: JAVA HEAP settings for KAFKA in production

Just about all of our systems are actually running 64GB of memory now, and yes, we want plenty of space for cache with all the disk and network IO going on. -Todd On 5/7/14, 10:31 AM, "John Penner" wrote: >Just curious if these JAVA_HEAP tunings are still for machines listed in >6.4 >Hardware a

Re: New consumer APIs

Neha: Here's the basic pseudo code of the process acting as the Kafka consumer: executor = Executors.newFixedThreadPool(numberOfThreads) consumer = // get a handle to the broker. mytopicStreams = consumer.getStreams({ "mytopic" => numberOfThreads }).get("mytopic") for (stream : mytopicStreams) {

Re: Compression in Kafka: GZIP or Snappy

I created a ticket for the patch https://issues.apache.org/jira/browse/KAFKA-1456 /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop ***

Re: Leader for almost all topic, partitions on the same broker

Thanks for the responses! >Does that error occur continuously? Preferred leaders are the first replica >in the assigned replica list. Could you list the topics and see the >distribution of the first replica in all partitions? The state we were in showed all topics led by a single broker. The pref

running kafka on netapp

Hi Does anyone have any stats or experience to share with running kafka on netapp based filer / storage. Has anybody faced any issues with it or it did not matter whether the storage is local to the host or a network storage. Thanks

Re: NFS and/or local filesystem consumer?

Correction, the http post may or may not be faster than writing directly to SMB, but hopefully we can improve that speed in a more scalable manner than SMB. --Ken On May 16, 2014, at 11:17 AM, Carlile, Ken wrote: > Hi all, > > Sorry for the possible repost--hadn't seen this in the list aft

Re: Max Message Size issue

You need to change max.message.size on the brokers. Thanks, Jun On Fri, May 16, 2014 at 11:02 AM, Bhavesh Mistry wrote: > Hi Kafka Dev Group, > > We are using Kafka version 0.8 and I am getting following exception: > > > WARN warn, Produce request with correlation id 1617 failed due to > [r

Re: New consumer APIs

Hi Neha, Yes a way that allows each partition to be committed seperately. Couldn't remember if the new consumer allows it, but looks like it does! Tim On Fri, May 16, 2014 at 9:37 AM, Neha Narkhede wrote: > Tim, > > I'm going to ask you the same question :-) > > By "per stream commit", do you

Re: Compression in Kafka: GZIP or Snappy

On May 7, 2014, at 7:16 AM, Maung Than wrote: > Hi All, > > I have read this posting from linkedIn Team member; > http://geekmantra.wordpress.com/2013/03/28/compression-in-kafka-gzip-or-snappy/ > ; Thanks. > > I have few questions and thoughts: > > 4) Has any one else done Snappy Vs. GZ

Re: running kafka on netapp

Kafka prefers local storage since it's more scalable and Kafka handles replication itself across brokers. Thanks, Jun On Thu, May 15, 2014 at 1:56 PM, vipul jhawar wrote: > Hi > > Does anyone have any stats or experience to share with running kafka on > netapp based filer / storage. Has anybod

Re: kafka broker failed to recovery from ZK failure

The problem is indicated by the following log in broker 1's controller log. Were you deleting any topic at that time? [2014-05-12 21:24:37,930] ERROR [BrokerChangeListener on Controller 1]: Error while handling broker changes (kafka.controller.ReplicaStateMachine$BrokerChangeListener) java.util.No

Re: topic.log.retention.hours doesn't work

>From 0.8.1 onwards, per topic config settings cannot be changed through the properties file and you need to use the kafka-topics tool to do that. Please see the documentation here . Thanks, Neha On Thu, May 15, 2014 at 1:17

ISR not updating

Hi - We are running kafka_2.8.0-0.8.0-beta1 (we are a little behind in upgrading). >From what I can tell, connectivity to ZK was lost for a brief period. The >cluster seemed to recover OK except that we now have 2 (out of 125) partitions >where the ISR appears to be out of date. In other words,

Re: NFS and/or local filesystem consumer?

Hi Jun, I was wondering if there was something out there already. GPFS appears to the OS as local filesystem, so if there was a consumer that dumped to local filesystem, we'd be gold. Thanks, --Ken On May 16, 2014, at 7:04 PM, Jun Rao wrote: > You probably would have to write a consumer ap

Re: NFS and/or local filesystem consumer?

You probably would have to write a consumer app to dump data in binary form to GPFS or NFS, since the HDFS api is very special. Thanks, Jun On Fri, May 16, 2014 at 8:17 AM, Carlile, Ken wrote: > Hi all, > > Sorry for the possible repost--hadn't seen this in the list after 18 hours > and figure

[0.7.2] see ZkEventThread triggered when these is no broker change

see these errors in producer's log: ＝＝＝ [2014-05-17 08:38:59,983] ERROR Error handling event ZkEvent[Children of /brokers/topics/test_device_updates changed sent to kafka .producer.ZKBrokerPartitionInfo$BrokerTopicsListener@1b0ec7cc] (org.I0Itec.zkclient.ZkEventThread) java.lang.NullPointerExcept

Re: New consumer APIs

Hey Eric, Yeah this is more similar to what we currently have but with a richer api then a simple Iterator. I think the question is how the poll() on the various streams translates into the ultimate poll that we need to do against the individual socket connections. Some of the things that make t